Sign up
QA & troubleshootingこの記事を日本語で読む

Mojibake: reading the garbled text to find the fix

You open a localization file and instead of Japanese text you see something like 縺ゅ↑縺溘, or instead of a German umlaut you see ä. Nothing about the data is actually broken. The bytes on disk are almost always exactly the ones that were written; what happened is that some program read them back using the wrong encoding and turned correct bytes into the wrong characters. That distinction matters, because it means the fix is rarely to touch the text at all — it is to figure out which decoding step went wrong and undo it.

Mojibake is a decoding mismatch, not data loss

Every text file on disk is just a sequence of bytes. A program has to be told, or has to guess, which encoding those bytes represent before it can turn them into characters on screen. When the guess is wrong, each byte or byte group still maps to some character — just not the one the writer intended. The result looks broken, but the original byte sequence is still sitting there unchanged, which is why mojibake is usually recoverable.

This is different from a file that was genuinely damaged — truncated mid-character, or overwritten with the wrong content. Mojibake is systematic: the same wrong mapping is applied consistently across the whole file, which is exactly what makes the pattern readable once you know what to look for.

Reading the pattern to identify the wrong decoding

The shape of the garbled text tells you which decoder was used, because each wrong encoding produces a distinctive signature.

  • UTF-8 bytes decoded as Shift_JIS or CP932 (common for Japanese text): the output is a dense run of unrelated-looking kanji and hiragana, often including katakana that would never appear together, such as 縺ゅ↑縺溘 where the original was あなた.
  • UTF-8 bytes decoded as a single-byte Western encoding such as ISO-8859-1 or Windows-1252: each UTF-8 multi-byte character becomes two or three separate accented Latin letters, producing the familiar à pattern — ä for ä, ö for ö.
  • A single-byte or Shift_JIS file decoded as UTF-8: the decoder cannot make sense of many byte sequences at all and substitutes U+FFFD REPLACEMENT CHARACTER, so you see a run of ⯑ or boxes instead of letters.
  • A file with no visible corruption but wrong characters in specific spots only: this is more likely a targeted substitution — for example the CP932-versus-JIS wave dash mapping difference — not general mojibake, and needs a different fix.

How to recover the file

The correct fix is to re-decode the original bytes with the correct encoding, not to search-and-replace the garbled characters. Find-and-replace treats the symptom string by string and will miss combinations nobody thought to list, or silently corrupt text that happened to look similar. Re-decoding fixes every instance at once because it addresses the actual cause.

In practice this means: identify what encoding the file is actually in (often knowable from where it came from — an export from a Japanese-locale spreadsheet is very likely Shift_JIS or CP932; a file from a web form is very likely UTF-8), then read the raw bytes and decode them explicitly with that encoding instead of letting a tool auto-detect or default to something else. If you only have the already-mojibake'd text and not the original bytes, you can sometimes reverse the damage by re-encoding the garbled string back into bytes using the wrong encoding that produced it, then decoding those bytes with the correct one.

// Example: UTF-8 bytes were decoded as CP932/Shift_JIS.
// Reversing it means going back through the same two steps in the
// opposite order — encode with the WRONG encoding used to display it,
// then decode with the RIGHT one that was intended originally.
const garbled = "縺ゅ↑縺溘";
const bytes = encodeWith(garbled, "Shift_JIS"); // recover the original UTF-8 bytes
const fixed = decodeWith(bytes, "UTF-8"); // "あなた"

When the data is genuinely unrecoverable

Reversal only works while the original bytes still exist somewhere in the pipeline. If a file was mojibake'd and then saved again in that broken state — especially if the save step re-encoded the garbled characters as UTF-8 — the original byte sequence is gone. What is on disk now is a new, valid encoding of the wrong characters, and there is no reliable way to compute back to the source text from it.

The same is true whenever U+FFFD REPLACEMENT CHARACTER appears: it means the decoder could not represent a byte sequence at all and threw it away, replacing unknown data with a placeholder. Once that substitution has happened and been saved, the original bytes are permanently gone — U+FFFD does not carry any information about what it replaced.

Stopping it from recurring

Mojibake happens at boundaries — anywhere a file changes hands between tools, systems, or people. The fix that actually holds is to pin the encoding explicitly at every boundary in your localization pipeline: the export step, the import step, and any tool in between, rather than relying on auto-detection. Auto-detection heuristics are good but not perfect, and a wrong guess made once, saved, and shipped is exactly how mojibake becomes unrecoverable.

Related articles