Encoding — UTF-8 as the recommendation, plus edge cases · Python Quest

UTF-8 is correct 95% of the time

Modern files are UTF-8. Modern web protocols are UTF-8. Modern source code is UTF-8. Unless you have a specific reason to use something else (legacy data, system constraints), UTF-8 is the right default. The remaining 5% — Windows legacy files, Korean Windows tools that emit CP949, old Excel exports — are real and you'll meet them, but they're the exception.

The BOM — and why it exists

A byte-order mark (BOM) is a special character at the start of a file that signals the encoding. UTF-8 doesn't really need one (the encoding is unambiguous), but Windows software often writes it anyway. utf-8-sig as the encoding when reading skips the BOM if present; utf-8 would leave it as a literal character at the start of your data.

What goes wrong — and how to read the symptoms

UnicodeDecodeError means you're trying to decode bytes that aren't valid in the encoding you specified. Common causes: a file is actually UTF-16 or CP1252 and you tried UTF-8; the file is binary and you tried text mode; a multi-byte character was cut in half. Read the error: it tells you the byte position and what byte was problematic.

chardet — when you really don't know

pip install chardet gives you a library that guesses encodings from a byte sample. Useful for one-time imports of mystery files. Don't put it in production paths — the guess is a probability, not a guarantee.

Principle: Always specify encoding when opening text files. The system default is platform-specific and will bite cross-platform code. encoding="utf-8" if you control both ends. Match the actual encoding if you don't.

Needing to guess an encoding means the boundary lacks a contract. Treat detector output as a candidate, preserve irreplaceable original bytes, and verify representative text after conversion.

errors='ignore' is silent loss, not recovery. Even when replacement characters are acceptable, record which file and byte position failed so the converted copy can be checked against the original.

Code

UTF-8 — the modern default·python

# Korean text is multi-byte in UTF-8
text = "피파, 안녕"
b = text.encode("utf-8")
print(b)                     # b'\xed\x94\xbc\xed\x8c\x8c, \xec\x95\x88\xeb\x85\x95'
print(len(text))             # 6 characters
print(len(b))                # 17 bytes — Korean glyph is 3 bytes each

# Round-trip
back = b.decode("utf-8")
print(back == text)          # True

BOM — utf-8-sig vs utf-8·python

import io

# A file with BOM (often from Windows Notepad as UTF-8)
with_bom = b"\xef\xbb\xbfhello"

# utf-8 — BOM stays as the first character
print(with_bom.decode("utf-8"))         # '\ufeffhello'

# utf-8-sig — BOM is consumed
print(with_bom.decode("utf-8-sig"))     # 'hello'

# When reading a file that may have a BOM, prefer utf-8-sig
# It tolerates both presence and absence of the BOM

UnicodeDecodeError — reading the symptoms·python

# CP949 is a Korean Windows encoding
text = "안녕"
cp949_bytes = text.encode("cp949")
print(cp949_bytes)            # b'\xbe\xc8\xb3\xe7'

# Try to decode as UTF-8 — fails
try:
    cp949_bytes.decode("utf-8")
except UnicodeDecodeError as e:
    print("position:", e.start)
    print("reason:", e.reason)
    # The error tells you exactly where decoding failed

# Decode with the correct encoding — works
print(cp949_bytes.decode("cp949"))     # '안녕'

errors= — handling decode failures gracefully·python

messy = b"hello \xff world"           # invalid UTF-8 byte

# Strict — raises
try:
    messy.decode("utf-8")
except UnicodeDecodeError as e:
    print("strict:", e)

# Replace — invalid bytes become U+FFFD
print(messy.decode("utf-8", errors="replace"))   # 'hello \ufffd world'

# Ignore — invalid bytes dropped
print(messy.decode("utf-8", errors="ignore"))    # 'hello  world'

# Backslash-escape — visible representation
print(messy.decode("utf-8", errors="backslashreplace"))   # 'hello \\xff world'

# Use 'replace' for display, 'ignore' rarely, 'strict' (default) for code

Exercise

Create a string containing Korean ("피파"), English ("hello"), an emoji ("🌸"). Encode the string as UTF-8 and print the byte length and character length. Encode again as UTF-16 and compare lengths. Now corrupt the UTF-8 bytes (replace one mid-character byte with 0xFF) and try to decode — observe the UnicodeDecodeError and decode again with errors='replace' to see the U+FFFD character.

Encoding — UTF-8 as the recommendation, plus edge cases

UTF-8 is correct 95% of the time

The BOM — and why it exists

What goes wrong — and how to read the symptoms

chardet — when you really don't know

Code

External links

Exercise

Progress

Comments 0