Modern files are UTF-8. Modern web protocols are UTF-8. Modern source code is UTF-8. Unless you have a specific reason to use something else (legacy data, system constraints), UTF-8 is the right default. The remaining 5% — Windows legacy files, Korean Windows tools that emit CP949, old Excel exports — are real and you'll meet them, but they're the exception.
The BOM — and why it exists
A byte-order mark (BOM) is a special character at the start of a file that signals the encoding. UTF-8 doesn't really need one (the encoding is unambiguous), but Windows software often writes it anyway. utf-8-sig as the encoding when reading skips the BOM if present; utf-8 would leave it as a literal character at the start of your data.
What goes wrong — and how to read the symptoms
UnicodeDecodeError means you're trying to decode bytes that aren't valid in the encoding you specified. Common causes: a file is actually UTF-16 or CP1252 and you tried UTF-8; the file is binary and you tried text mode; a multi-byte character was cut in half. Read the error: it tells you the byte position and what byte was problematic.
chardet — when you really don't know
pip install chardet gives you a library that guesses encodings from a byte sample. Useful for one-time imports of mystery files. Don't put it in production paths — the guess is a probability, not a guarantee.
Principle: Always specify encoding when opening text files. The system default is platform-specific and will bite cross-platform code. encoding="utf-8" if you control both ends. Match the actual encoding if you don't.
Code
UTF-8 — the modern default·python
# Korean text is multi-byte in UTF-8
text = "피파, 안녕"
b = text.encode("utf-8")
print(b) # b'\xed\x94\xbc\xed\x8c\x8c, \xec\x95\x88\xeb\x85\x95'
print(len(text)) # 6 characters
print(len(b)) # 17 bytes — Korean glyph is 3 bytes each
# Round-trip
back = b.decode("utf-8")
print(back == text) # True
BOM — utf-8-sig vs utf-8·python
import io
# A file with BOM (often from Windows Notepad as UTF-8)
with_bom = b"\xef\xbb\xbfhello"
# utf-8 — BOM stays as the first character
print(with_bom.decode("utf-8")) # '\ufeffhello'
# utf-8-sig — BOM is consumed
print(with_bom.decode("utf-8-sig")) # 'hello'
# When reading a file that may have a BOM, prefer utf-8-sig
# It tolerates both presence and absence of the BOM
UnicodeDecodeError — reading the symptoms·python
# CP949 is a Korean Windows encoding
text = "안녕"
cp949_bytes = text.encode("cp949")
print(cp949_bytes) # b'\xbe\xc8\xb3\xe7'
# Try to decode as UTF-8 — fails
try:
cp949_bytes.decode("utf-8")
except UnicodeDecodeError as e:
print("position:", e.start)
print("reason:", e.reason)
# The error tells you exactly where decoding failed
# Decode with the correct encoding — works
print(cp949_bytes.decode("cp949")) # '안녕'
Create a string containing Korean ("피파"), English ("hello"), an emoji ("🌸"). Encode the string as UTF-8 and print the byte length and character length. Encode again as UTF-16 and compare lengths. Now corrupt the UTF-8 bytes (replace one mid-character byte with 0xFF) and try to decode — observe the UnicodeDecodeError and decode again with errors='replace' to see the U+FFFD character.
Progress
Progress is local-only — sign in to sync across devices.