Why sampling stops working once AI is in your translation pipeline
Sampling is a reasonable way to review a file when you cannot afford to read every line. Pick a handful of strings, read them closely, and if they look good, ship the rest. Almost every localization team does some version of this, and for a long time it worked well enough. What sampling actually certifies, though, is not the file — it is the assumption that quality is roughly uniform across the file. Once that assumption stops holding, a clean sample stops meaning what people think it means.
That assumption held reasonably well when a single trained human translator produced the whole file. A translator's skill, attention, and judgment do not usually swing wildly from line to line within one session — a sample taken from the middle of the file is a fair proxy for the lines you did not read. Machine and AI output does not behave the same way.
Why AI output is not uniform the way human output is
A human translator carries context across an entire file: they remember what a character's tone was three lines ago, they know a term was already decided, they notice when a sentence does not make sense in the game. An AI system, depending on how it is used, may process each string with a different amount of surrounding context, may apply a glossary decision inconsistently across the file, and can produce a fluent, confident-sounding sentence that is simply wrong for the specific line it was given.
This is not a claim that machine output is worse on average — it is a claim about variance. A translator having a bad day still translates like a translator. A model asked to translate a short, ambiguous, or context-dependent string can produce something that reads perfectly well in isolation and is still incorrect for the scene it appears in. Reading ten lines at random tells you almost nothing about line eleven.
The consequence: a clean sample certifies less than it used to
If quality can vary sharply from string to string, then sampling only tells you about the strings you sampled. It does not extrapolate the way it did for a single consistent translator. Teams that keep using the same sampling ratio they used for human-translated files are, without realizing it, reviewing a shrinking fraction of the risk. The file looks the same size as before; the thing that changed is how much a sample can tell you about the rest of it.
Not every property needs a human eye
The useful move is to separate what needs judgment from what does not, and stop asking a human to do the second job. A large share of localization defects are mechanical, not linguistic — they do not require understanding the sentence, only reading its shape:
- Placeholders and format specifiers present in the source but missing (or malformed) in the translation
- Tags left unbalanced or in a different order than the source
- Text that is byte-for-byte identical to the source, suggesting it was never translated
- Encoding corruption — replacement characters, mangled multi-byte sequences
- Duplicate or missing keys against the source file
What this frees human review to do
None of those checks require reading for meaning, and none of them get less reliable when applied to every single entry instead of a sample — a script does not get tired on line four thousand the way a reviewer does. Running them on one hundred percent of a file costs almost nothing and removes an entire category of defect from human review before a person even opens the file.
That is the actual point of full-coverage mechanical checking: not that it replaces human judgment, but that it stops wasting human judgment on things a computer can verify with certainty. What is left for a person to review is meaning and tone — whether the line fits the character, whether the joke lands, whether the register matches the scene. That is exactly the part sampling was always weakest at estimating from a handful of lines, and it is the part that still needs a human, sample or no sample.
A practical takeaway
If your translation pipeline now includes AI output anywhere, treat mechanical correctness as a full-coverage problem and treat meaning as the thing your limited human review time should be spent on. Sampling is not wrong as a technique — it is wrong as a substitute for checks that could simply run on everything. The question worth asking about any given check is not how big a sample you need, but whether the check actually requires a human at all, or whether it has just always been done that way.