Tracking localization quality over time, without inventing a score

It is tempting to want one number for localization quality — a score you can put on a dashboard and watch go up. Resist that temptation. Any single score for something as varied as translation, formatting, and UI fit either hides too much behind an average or is arbitrary about how it weighs an untranslated string against an awkward line reading. What actually tells you something is less exciting: counts, categories, and trends over time.

This does not mean giving up on visibility — it means building it out of things you can count honestly, and reading the trend rather than any single number in isolation.

Counting by category and severity per build

Every issue found during a build's review has a category (untranslated text, placeholder mismatch, terminology inconsistency, overflow, tone or naturalness) and a severity (something that breaks the build versus something a native reader would quietly wish were phrased better). Recording a count per category per build — not a total, the breakdown — is the smallest unit of tracking that stays honest.

A total count on its own is close to meaningless: forty issues in a build that added ten thousand new words is a very different signal from forty issues in a build that added two hundred. Normalizing loosely against volume of new or changed content, even informally, keeps the count from looking worse just because the build did more work.

• Category: untranslated / placeholder / terminology / overflow / tone / other
• Severity: blocks release / should fix / cosmetic or preference
• New content volume for the build (rough word or entry count is enough)
• Which check found it: automated format check, spreadsheet review, in-context review, player report

New versus resolved versus recurring

Raw counts per build tell you the state of a single point in time. What tells you about a trend is splitting each build's issues into three buckets against the previous build: new issues that were not present before, resolved issues that were present last time and are now fixed, and recurring issues that were flagged before, supposedly fixed, and have reappeared.

Recurring issues deserve the most attention of the three. A new issue is the cost of doing new work. A resolved issue is the system working as intended. A recurring issue means a fix did not actually reach the file, or reappeared after a later change silently reverted it — which points back at the review workflow's closing-the-loop step, not at translation quality itself.

// per-build issue delta, compared against the prior build
build 42: new=14  resolved=9   recurring=2
build 43: new=6   resolved=14  recurring=0
build 44: new=21  resolved=8   recurring=5  ← recurring spiked, investigate

Watching which categories keep coming back

Over several builds, a category that stays flat or grows while others shrink is a signal worth reading literally: something structural is producing that category of issue, and it is not going to fix itself through more review effort in the same place. A steady stream of terminology inconsistencies points at a missing or unused glossary, not at careless translators. A steady stream of overflow issues points at UI that was never designed with length variance in mind, not at translators writing overly long lines.

The useful move here is treating a persistent category as a process question rather than reassigning blame to whoever touched the string last. Asking 'why does this category keep recurring' usually points at a missing constraint upstream — a glossary that exists but nobody checks against, a UI component with no length budget, an export step that silently drops context a translator needed — rather than at any individual's competence.

Why a baseline is what makes any of this mean anything

None of the counts above mean anything without a consistent point of comparison. If build 44 is compared against build 40 in one report and against build 43 in the next, the numbers cannot be read as a trend — they are just two unrelated snapshots that happen to share a chart. A baseline means picking a comparison rule and keeping it: always the immediately preceding build, or always the last release build, and sticking to it long enough for the trend to be visible.

It also means agreeing on what counts as the same issue across builds — the same underlying problem, in the same string, should be recognized as one recurring issue rather than a fresh count each time it is caught, or the recurring bucket above becomes meaningless. Once that consistency is in place, the trend lines do the work that a single quality score never could: they show whether the localization process is actually improving, staying flat, or quietly getting worse, and they show it honestly enough to act on.