Log Parsing

Most logs are line-oriented and shape-stable

That's exactly the sweet spot for regex. Each line follows the same template; named capture groups extract the fields.

Apache/nginx Combined Log Format

192.168.1.1 - - [04/May/2026:14:32:11 +0000] "GET /api/data HTTP/1.1" 200 1234

Pattern:

(?P<ip>[\d.]+) - - \[(?P<date>[^\]]+)\] "(?P<method>\w+) (?P<path>[^ ]+) [^"]+" (?P<status>\d+) (?P<bytes>\d+)

App-specific structured logs

Lines like 2026-05-04T14:32:11Z [INFO] Server started on port 8000:

(?P<timestamp>\S+)\s+\[(?P<level>\w+)\]\s+(?P<message>.+)

The verbose-mode payoff

Real log patterns get long fast. VERBOSE mode + named groups makes them maintainable:

LOG = re.compile(r'''
    (?P<timestamp>\d{4}-\d{2}-\d{2}T[\d:.]+Z?)
    \s+\[(?P<level>\w+)\]
    \s+(?P<module>[\w.]+)
    \s+(?P<message>.+)
''', re.VERBOSE)

The streaming pattern

For large log files, stream line-by-line and apply the regex to each:

with open('big.log') as f:
    for line in f:
        m = LOG.match(line)
        if m:
            handle(m.groupdict())

Combined with re.compile, this is fast even on multi-gigabyte logs.

Code

Log parsing patterns·python

import re

# Apache/nginx Combined Log Format
APACHE = re.compile(r'''
    (?P<ip>[\d.]+)
    \s+-\s+-\s+
    \[(?P<date>[^\]]+)\]
    \s+"(?P<method>\w+)\s+(?P<path>[^ ]+)\s+[^"]+"
    \s+(?P<status>\d+)
    \s+(?P<bytes>\d+)
''', re.VERBOSE)

line = '192.168.1.1 - - [04/May/2026:14:32:11 +0000] "GET /api/data HTTP/1.1" 200 1234'
m = APACHE.match(line)
print(m.groupdict())

# App log
APP = re.compile(r'''
    (?P<ts>\d{4}-\d{2}-\d{2}T[\d:.]+Z?)
    \s+\[(?P<level>\w+)\]
    \s+(?P<msg>.+)
''', re.VERBOSE)

m = APP.match('2026-05-04T14:32:11Z [INFO] Server started on port 8000')
print(m.groupdict())
# {'ts': '2026-05-04T14:32:11Z', 'level': 'INFO', 'msg': 'Server started on port 8000'}

# Stream-process a large log
def tail_errors(path):
    with open(path) as f:
        for line in f:
            m = APP.match(line)
            if m and m.group('level') == 'ERROR':
                print(m.group('msg'))

Most logs are line-oriented and shape-stable

Apache/nginx Combined Log Format

App-specific structured logs

The verbose-mode payoff

The streaming pattern

Code

External links

Exercise

Progress

Comments 0