File Encoding

The bytes vs the characters

Files are bytes. Your tools assume those bytes encode characters. Different encodings (UTF-8, UTF-16, ISO-8859-1) interpret the same bytes as different characters. When the assumption breaks, you get mojibake (ëëë) or hard errors.

Detect the encoding

file --mime-encoding file.txt   # GNU and macOS
iconv -f UTF-8 -t UTF-8 file.txt >/dev/null # errors mean it isn't UTF-8

Convert

iconv -f ISO-8859-1 -t UTF-8 in.txt > out.txt
iconv -f UTF-16 -t UTF-8 win.txt > mac.txt

BOM — the invisible enemy

UTF-8 with a Byte Order Mark prepends 3 bytes (EF BB BF) to your file. Most modern tools handle it; some don't, and you see weird characters at the start. Strip it: sed -i '1s/^\xef\xbb\xbf//' file (linux GNU) or perl -i -pe 's/^\x{FEFF}//' file.

Line endings

• Unix (LF) — \n only.
• Windows (CRLF) — \r\n.
• Old Mac (CR) — \r only. Almost extinct.

Convert with tr -d '\r' < in.txt > out.txt or dos2unix file (brew install dos2unix). Many bugs in cross-OS shared files come down to line endings.

Why this matters

A regex like $ assumes \n as the end-of-line marker. If the file has CRLF, your $ match silently includes \r as the last character of every line. Tests pass on dev machines and fail in CI when files travel through Windows.