View More Demos
NEKO.KSC
extract_sjis_strings.py

I have a Python script to extract text from KSC files in Shift JIS encoding. Text strings start after a 48 or 8E byte and end with a null byte. There is a bug where characters containing 48 or 8E byte may cause a new string to start mid-sentence in another string.

Dr.Binary

I'll examine the Python script and the KSC file to understand the bug and fix it.

read
read
Cannot read binary file: /data/google-oauth2-101770454311189128481-690c3a28-cb04-4c60-a1d0-ce71604ddb0c/NEKO.KSC

Let me run the current script to see the issue, then fix the bug:

drbinary_run_sandbox
drbinary_run_sandbox

I can see the issue clearly now. Looking at the output, there are fragmented strings like:

  • えないもん[NYA]〜[LR] (should be part of a longer string)
  • の[WORD0]から…[LR] (fragment)
  • してくれた大切なコトバ…[LR] (fragment)

The bug is in the is_valid_marker function. For byte 0x8E, it only checks if it's a lead byte for half-width katakana, but 0x8E can also be a trail byte of a 2-byte Shift-JIS character (since 0x8E falls in the valid trail byte range 0x80-0xFC).

Here's the fix:

edit

Now let's test the fix:

drbinary_run_sandbox

The fix eliminated 631 false positive strings (11970 → 11339). Let's verify the previously fragmented strings are now correctly merged:

drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()
drbinary_run_sandbox
drbinary_run_sandbox

The 8E at 0x4756 is correctly identified as a marker because the previous byte (0x20 = space) is not a valid SJIS lead byte, and the next byte (0x67) is not half-width katakana. This is a legitimate second string in the file, not a fragmentation artifact.

Let's compare the outputs to confirm the fix eliminated actual fragmentation:

drbinary_run_sandbox
drbinary_run_sandbox

The fix is working correctly. Here are the results:

Before Fix (Fragmented)After Fix (Complete)
えないもん[NYA]〜[LR]しりとりに使えないもん[NYA]〜[LR]
してくれた大切なコトバ…[LR][WORD0]が残してくれた大切なコトバ…[LR]
の[WORD0]から…[LR]前の子の[WORD0]から…[LR]

Summary

Bug: The is_valid_marker() function only checked if 0x8E was a lead byte for half-width katakana, but failed to check if it was a trail byte of a 2-byte Shift-JIS character. Since 0x8E falls within the valid SJIS trail byte range (0x80-0xFC), it can appear as the second byte of kanji/hiragana characters.

Fix: Added the same check for 0x8E that existed for 0x48 - verifying that the previous byte is not a valid SJIS lead byte (0x81-0x9F or 0xE0-0xFC):

# For 0x8E, check if previous byte is a valid SJIS lead
if pos > 0 and is_sjis_lead(data[pos - 1]):
    return False

Result: Eliminated 631 false positive fragmented strings (11970 → 11339 valid strings).