Sign up
QA & troubleshootingこの記事を日本語で読む

The invisible characters breaking your key lookups and diffs

Two strings that look completely identical on screen fail a string-equality check. A translation key that a translator retyped no longer matches the key your code looks up. A diff between two versions of a file shows a change on a line where nothing visibly changed. In all three cases the likely cause is the same: an invisible character that a screen doesn't render distinctly, but that a computer treats as a real, different character.

The characters involved

None of these are corruption in the mojibake sense — every one of them is a legitimate Unicode character doing exactly what it is defined to do. The problem is that several of them render as nothing, or as something visually identical to a character they aren't, which makes them invisible to a human reviewing text on screen while remaining completely distinct to any code comparing byte-for-byte or character-for-character.

  • Zero-width space (U+200B) and other zero-width characters (zero-width joiner U+200D, zero-width non-joiner U+200C) — render as literally nothing, but are still characters in the string.
  • Non-breaking space (U+00A0) — renders identically to a regular space in most fonts, but is a different character, so it does not match a search or comparison expecting an ordinary space.
  • A UTF-8 or UTF-16 BOM appearing partway through a file, not just at the start — commonly the result of concatenating files that each carried their own BOM; it renders as nothing or as a stray box depending on the tool.
  • Directional marks such as the left-to-right mark (U+200E) and right-to-left mark (U+200F), and stronger embedding/override controls — used deliberately in bidirectional text, but sometimes carried over accidentally from a source that used them and left invisible in the target language.
  • Fullwidth space (U+3000) versus a regular halfwidth space (U+0020) — both render as blank space, but are different characters with different widths, common in Japanese text where fullwidth spacing is used for visual alignment.

Why they break exact-match comparison, key lookups, and diffs

Every one of these systems — a translation key lookup, a diff tool, a duplicate-string check, a regression comparison between two analysis runs — works by comparing strings character-for-character or byte-for-byte, not by comparing how the text looks when rendered. A key lookup does not care that two strings look the same when displayed; it cares whether the character sequences are identical.

This is exactly why the failures are so confusing to debug by eye: the person looking at the two strings side by side sees no difference at all, because the difference is specifically the kind that does not render. The only way to see it is to inspect the actual character or byte values, not the rendered text.

// These two look identical on screen but are different strings.
const key1 = "menu.start"; // ordinary characters throughout
const key2 = "menu.start​"; // trailing zero-width space — invisible,
                                  // but key1 !== key2 as far as any lookup
                                  // or Map is concerned

How to detect them

Detection has to happen at the character-inspection level, not the visual level, since the whole problem is that these characters don't show up visually.

  • Search for the specific code points directly (regex or code-point search) across your source files — zero-width space, non-breaking space, mid-file BOM, and directional marks are a short, known list worth checking for explicitly.
  • Compare string lengths where two strings are expected to be identical or where one is a copy of the other; an unexpected length difference with no visible difference is a strong signal.
  • Log or display raw code points for a string when a lookup or comparison fails unexpectedly and the two values look identical — most text editors and consoles can show a string as a list of code points or escape sequences on request.

A normalization policy that prevents recurrence

The fix that holds is not chasing individual instances but applying a consistent normalization step at every point text enters your pipeline — import, translation return, manual entry — so invisible characters never accumulate silently in the first place.

  • Strip zero-width characters and stray mid-file BOMs on import unless a specific string legitimately needs one (rare, and worth a comment when it happens).
  • Normalize whitespace variants deliberately rather than by accident: decide whether non-breaking space and fullwidth space are allowed in specific contexts (they often are, for visual formatting) and strip or convert them everywhere else.
  • Apply Unicode normalization (NFC is the common choice for most localization pipelines) consistently on import, so that characters which can be represented multiple equivalent ways in Unicode are always stored the same way — this closes a related but distinct class of the same underlying problem, where two strings are Unicode-equivalent but not byte-identical.
  • Run key and duplicate-string comparisons after normalization, not on raw input, so a translator's or tool's incidental invisible character never silently breaks a lookup that used to work.

Related articles