URL Matching

Two questions, two patterns

"Match a URL" is actually two different questions:

Is this whole string a URL? (validation)
Find URLs in this text. (extraction)

Different patterns, different anchoring.

Validation pattern

^https?://[\w-]+(\.[\w-]+)+(/[\w./?%&=+#-]*)?$

Accepts http or https, requires a domain with at least one dot, optionally a path with allowed URL characters.

Extraction pattern

For finding URLs in free-form text, you want to be more permissive but stop at obvious word boundaries:

https?://[^\s)]+

Matches http(s):// followed by anything that's not whitespace or a closing paren. The closing-paren stop prevents URLs in markdown like (https://example.com) from including the trailing ).

The trailing punctuation problem

URLs at the end of sentences pick up trailing periods, commas, parens. Solution: use a negated class that excludes those, OR strip trailing punctuation in code after the match.

Don't try to fully validate URLs with regex

Real URL validation is messy: scheme variants, IP literals, port numbers, internationalized domains, fragment encoding. For "does this URL parse and resolve?", use your language's URL library: Python urllib.parse.urlparse, JavaScript new URL(...), Go net/url.

Code

URL patterns·python

import re

# Validation (whole-string)
URL_VALID = re.compile(r'^https?://[\w-]+(\.[\w-]+)+(/[\w./?%&=+#-]*)?$')

bool(URL_VALID.fullmatch('https://example.com'))                # True
bool(URL_VALID.fullmatch('http://sub.domain.com/path?q=1'))     # True
bool(URL_VALID.fullmatch('not a url'))                          # False
bool(URL_VALID.fullmatch('ftp://example.com'))                  # False (only http(s))

# Extraction (find in text)
URL_FIND = re.compile(r'https?://[^\s)]+')
re.findall(URL_FIND, 'see https://pippa.dev or https://cwk.io for more (https://other.org)')
# ['https://pippa.dev', 'https://cwk.io', 'https://other.org']

# Strip trailing punctuation in post-processing
import string
def clean_url(url):
    return url.rstrip(string.punctuation)

# For real validation — use the URL library
from urllib.parse import urlparse
parsed = urlparse('https://sub.example.com:8080/path?q=1')
print(parsed.scheme)   # 'https'
print(parsed.netloc)   # 'sub.example.com:8080'
print(parsed.path)     # '/path'

Two questions, two patterns

Validation pattern

Extraction pattern

The trailing punctuation problem

Don't try to fully validate URLs with regex

Code

External links

Exercise

Progress

Comments 0