Optimization Techniques

The optimization hierarchy

Before optimizing, measure. Most regex performance "problems" are actually correctness problems wearing a costume. With that caveat, the techniques in order of impact:

1. Compile patterns you reuse

Compilation parses the pattern once. Runtime uses the compiled state machine. re.compile at module level for any pattern used in a loop or function called repeatedly.

2. Anchor patterns

^pattern stops the engine from re-trying at every position. Massive speedup for validation patterns.

3. Replace alternation with character classes

[abc] is dramatically faster than (a|b|c). Engines have specialized character class machinery.

4. Avoid backtracking with atomic groups / negated classes

[^X]+ for "everything until X" beats .*?X in both speed and correctness. Atomic groups for runs you know shouldn't backtrack.

5. Make patterns specific from the left

The engine reads left to right. A pattern starting with \d rejects non-digit text immediately. A pattern starting with .* tries every position before giving up.

6. Use the right engine

For bulk text processing on huge files, ripgrep (RE2) outperforms anything backtracking-based. For one-off tasks, re is fine.

7. Test on realistic input

Patterns that run fast on test data can crawl on production data. Profile with a representative sample.

Code

Optimization examples·python

import re
import time

# 1. Compile reused patterns
DATE = re.compile(r'\d{4}-\d{2}-\d{2}')  # do once
# Use DATE.findall(text) repeatedly

# 2. Anchor when possible
bool(re.fullmatch(r'^\d+$', '12345'))  # very fast — anchored

# 3. Character class beats alternation
# Slow
re.findall(r'(a|b|c|d|e)', 'abcdef' * 1000)
# Fast
re.findall(r'[abcde]', 'abcdef' * 1000)

# 4. Negated class beats lazy
text = '"hello" world ' * 1000
start = time.time()
re.findall(r'".*?"', text)
print(f'lazy: {time.time() - start:.3f}s')

start = time.time()
re.findall(r'"[^"]*"', text)
print(f'negated: {time.time() - start:.3f}s')

# 5. Specific prefix
# Slow
re.findall(r'.*=value', 'key=value' * 100)  # tries every position
# Fast
re.findall(r'\b\w+=value', 'key=value' * 100)  # specific anchor

# 6. Use ripgrep on big files
# rg 'pattern' bigfile.log    — much faster than Python re on >100MB