Shift_JIS and CP932 are not the same encoding — the pitfalls that follow

Shift_JIS shows up constantly in Japanese game localization pipelines, and it is treated as one encoding almost everywhere in tooling. In practice, most files that call themselves Shift_JIS were written by Windows software and are really CP932 — Microsoft's own superset of it, also known as Windows-31J. The two agree on almost everything, which is exactly what makes the disagreements dangerous: they surface rarely, on specific characters, usually after the data has already moved through several systems.

What CP932 adds on top of Shift_JIS

Shift_JIS proper encodes the character set defined by JIS X 0208: a mix of single-byte ASCII and half-width katakana, plus double-byte kanji, kana, and symbols. CP932 keeps that base and fills in code points that JIS X 0208 left undefined — additional symbols and kanji contributed by NEC, and a separate block of IBM extended characters, both carved out of unused areas of the double-byte range.

Two commonly seen groups live in those extensions: circled numbers such as the ones used for ranking or numbered lists, and bracketed abbreviations such as the one used for 「株式会社」(a corporation). Neither is part of the original JIS X 0208 repertoire — both are why these are called 機種依存文字, platform-dependent characters. A decoder that only implements standard Shift_JIS will not decode them at all; a decoder that implements CP932 will.

The wave dash problem

The best-known interoperability trap involves a single character: the wave dash used for ranges and approximation in Japanese text. JIS X 0208's own mapping table associates that Shift_JIS code point with U+301C WAVE DASH in Unicode. Microsoft's CP932-to-Unicode mapping table associates the same byte sequence with a different code point, U+FF5E FULLWIDTH TILDE.

This means the identical two bytes in a file can round-trip to two different Unicode characters depending on which conversion table decoded them. A string that looks correct in the tool that wrote it can render as the wrong glyph — or fail a string-equality check — the moment it passes through a decoder using the other table. This is not a bug in either table; it is two vendors making a different, both-defensible choice decades apart.

Why the mislabeling happens

Spreadsheet and text-editing tools on Japanese Windows have shipped a Shift_JIS export option for decades, and that option has almost always meant CP932 under the hood — it is the encoding Windows itself uses internally for the Japanese locale. The exported file's metadata, if it has any at all, typically just says Shift_JIS, because from the tool's perspective that is the accurate, familiar name.

The result is a file that decodes cleanly as long as you also treat it as CP932, and produces mangled or missing characters — or silent substitution of the wrong glyph — the moment a stricter, standards-only Shift_JIS decoder touches it.

A safe strategy

The declared encoding in a filename, a spreadsheet's export dialog, or a project's documentation is a hint, not a guarantee. Three practices keep this from becoming a recurring source of corrupted text:

• Detect at the boundary. When a file enters your pipeline, decode it with a CP932-capable decoder rather than a strict Shift_JIS one — CP932 is close enough to a strict superset that it correctly handles the common case and additionally covers the extended characters.
• Normalize to UTF-8 immediately. Do not carry Shift_JIS or CP932 bytes any further than the import boundary; every downstream stage (parsing, diffing, storage, translation memory) should work in UTF-8 so encoding ambiguity cannot resurface later.
• Verify, don't assume. After decoding, check for the specific failure signatures — replacement characters, unexpected byte counts, or known problem characters like the wave dash — rather than trusting that a file labeled Shift_JIS decoded correctly just because no exception was thrown.

The pattern to remember

None of this is really about Japanese text specifically — it is the general lesson that a declared encoding name describes an intent, not a guarantee, and that supersets are exactly where silent, low-frequency corruption hides. Treat the label as a starting hypothesis, decode with the more permissive real-world variant, and normalize to one internal representation as early as possible.