Character encoding for game text: what breaks it, and why to verify per file

Character encoding is the layer of a text file that decides how bytes on disk map to the characters you see on screen. Most game localization pipelines settle on UTF-8 for good reason — it can represent every character in every language a project is likely to ship, unlike older encodings limited to one script family. But a pipeline choosing UTF-8 and every file in that pipeline actually being UTF-8 are two different claims, and the gap between them is where a specific, recurring class of bugs lives.

UTF-8 as the default, and why

UTF-8 encodes every Unicode code point as one to four bytes, is backward-compatible with plain ASCII for the first 128 code points, and is the dominant encoding for text on the web and in most modern file formats. Choosing it as a project-wide default removes an entire category of decision-making per file and per language — there is no separate encoding to pick for Japanese versus German versus Korean.

What a BOM is, and how it surprises parsers

A byte order mark (BOM) is an optional sequence of bytes (EF BB BF for UTF-8) that some tools write at the very start of a file to signal its encoding. It's invisible in most text editors but is a real set of bytes in the file. A parser that doesn't expect it will read those bytes as part of the first line of content — a common symptom is a CSV header's first column name coming through with extra invisible characters prefixed to it, which then fails to match the key the code expects.

-- file starts with the byte sequence EF BB BF, then:
key,en,ja
-- a naive parser sees the first header cell as "\uFEFFkey", not "key"

What mojibake actually is

Mojibake is what you see when a file's bytes are correctly written in one encoding but decoded by the reader using a different one. The bytes on disk never changed; only the interpretation did. This is why mojibake often looks like a scramble of unrelated symbols rather than obviously missing text — the decoder is producing valid output for the wrong input, not failing to produce output at all.

How legacy encodings enter modern pipelines

The most common entry point is spreadsheet export. Spreadsheet applications exporting to CSV do not always default to UTF-8 — the default can depend on the operating system's regional settings, and a file exported on a machine set to a Japanese or Chinese locale can come out in a legacy multi-byte encoding instead. Nothing about the export process signals this to the person doing the export; the file opens fine on their own machine and looks broken only once it reaches a pipeline that assumes UTF-8.

Symptoms to recognize

A few visual signatures are worth recognizing on sight, since they point to different problems:

• The replacement character (a black diamond with a question mark, U+FFFD) — the decoder found a byte sequence that is not valid in the encoding it assumed, and substituted a placeholder rather than guessing
• Garbled kana or CJK characters replaced by unrelated symbols — classic mojibake, bytes correctly written in one multi-byte encoding but decoded as another
• An invisible extra character at the very start of a file's first line — a BOM the parser didn't strip
• Text that renders correctly in one editor but not another — a strong sign the file's real encoding differs from what at least one of the tools assumes, rather than a genuine data problem

Why encoding must be verified per file, not assumed

A pipeline that assumes UTF-8 for everything will work correctly for every file that actually is UTF-8, and silently corrupt the ones that aren't — there is no error at the file boundary, only wrong characters downstream, often far from where the wrong file entered. Because the entry point is usually a human export step outside the pipeline's control, encoding needs to be verified per file at import time rather than assumed once for the whole project.