File Encoding

The bytes vs the characters

Files are bytes. Your tools assume those bytes encode characters. Different encodings (UTF-8, UTF-16, ISO-8859-1) interpret the same bytes as different characters. When the assumption breaks, you get mojibake (ëëë) or hard errors.

Detect the encoding

file --mime-encoding file.txt   # GNU and macOS
iconv -f UTF-8 -t UTF-8 file.txt >/dev/null # errors mean it isn't UTF-8

Convert

iconv -f ISO-8859-1 -t UTF-8 in.txt > out.txt
iconv -f UTF-16 -t UTF-8 win.txt > mac.txt

BOM — the invisible enemy

UTF-8 with a Byte Order Mark prepends 3 bytes (EF BB BF) to your file. Most modern tools handle it; some don't, and you see weird characters at the start. Strip it: sed -i '1s/^\xef\xbb\xbf//' file (linux GNU) or perl -i -pe 's/^\x{FEFF}//' file.

Line endings

• Unix (LF) — \n only.
• Windows (CRLF) — \r\n.
• Old Mac (CR) — \r only. Almost extinct.

Convert with tr -d '\r' < in.txt > out.txt or dos2unix file (brew install dos2unix). Many bugs in cross-OS shared files come down to line endings.

Why this matters

A regex like $ assumes \n as the end-of-line marker. If the file has CRLF, your $ match silently includes \r as the last character of every line. Tests pass on dev machines and fail in CI when files travel through Windows.

Encoding detection combines evidence

file can be heuristic, and ASCII-only text is valid under several encodings. Combine declared metadata, BOM, bytes, and producer documentation instead of treating one detector as certainty.

Conversion has decode and encode stages

iconv -f A -t B first decodes as A and then encodes as B. A wrong source guess can produce corrupted but successful output. Test a sample, preserve the original checksum, and verify meaning.