View More Demos
NEKO.KSC

I have a binary file containing Japanese text in Shift-JIS encoding. I want to extract the text, its offset in hexadecimal and the length of each string. Some strings contain special byte codes such as 170000, 1614x0016 and 14x00 which are variables/placeholders.

Strings start after a 48 or 8E byte and end with 00. Special codes should be replaced with non-destructive text and continue searching until the next 00 byte. Do not output text that contains invalid characters.

Dr.Binary
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox

I have successfully extracted 12,007 Shift-JIS encoded Japanese strings from the NEKO.KSC file.

Extraction Results

Special codes handled:

  • 16 14 ... 00 16 → replaced with [VAR] (variable/placeholder)
  • 14 → replaced with [WAIT] (pause/wait)
  • 17 00 00 → replaced with [CMD] (command)

String boundaries:

  • Strings start after byte 48 or 8E
  • Strings end at null byte 00

Two output files created:

  1. neko_strings.csv (1.1 MB) - CSV format with columns: Offset, Length, Text, RawHex

  2. neko_strings.txt (835 KB) - Human-readable format

Sample Output

OffsetLengthText
0x0002F516アイドルで [VAR]
0x00032812歌手で [VAR]
0x00035716おわらいの [VAR]
0x0003BB16スポーツの [VAR]
0x0029B419それは おいといて…
0x0029EA8あのね、
0x003FEC23[VAR]は もう 知ってるの
0x00407C26人間に 近づいたか ニャ〜！

The extracted text appears to be dialogue from a Japanese game (likely a virtual pet or character simulation game), containing conversations about idols, singers, sports, introductions, and daily interactions with phrases like "アイドルで [VAR]" (Idol [VAR]), "歌手で [VAR]" (Singer [VAR]), and casual speech patterns.

The special code is 14...00. Codes should contain the missing hex value, for example [VAR1], [VAR2]. Characters in strings may contain 48 or 8E bytes which should not be treated as the start or end of text.

I made a mistake. 170000 is actually 171700. It should be replaced with [MEOW].

Dr.Binary
drbinary_run_sandbox

I've successfully extracted 12,002 Shift-JIS strings from the NEKO.KSC file with all special codes properly replaced.

Extraction Summary

Special codes replaced:

  • [VARn] = 16 14 n 00 16 → 4,947 occurrences
  • [WAITn] = 14 n 00 → 737 occurrences
  • [MEOW] = 17 17 00 00 → 11 occurrences

Output files created:

  • neko_strings.csv - CSV format (Offset, Length, Text, RawHex)
  • neko_strings.txt - Human-readable format

Sample Output

OffsetLengthText
0x0002F517アイドルで [VAR1]
0x00032813歌手で [VAR1]
0x00035717おわらいの [VAR1]
0x0003BB17スポーツの [VAR1]
0x00129411「[WAIT7]」
0x01AA0933どんな お洋服 着たら イイか[MEOW]
0x02360D23[VAR2]を 食べたの[MEOW]
0x04EF0A50自分が いびきを かいてるか 知る方法が あるの[MEOW]

The extraction correctly handles Shift-JIS encoding, where bytes 0x48 and 0x8E can appear as valid trail bytes in Japanese characters (e.g., 結婚相手), only treating them as string start markers when they actually precede text.

The pattern is two 17 bytes followed by one null byte. Continue scanning for text after patterns. 14x00 is another type of variable, not a wait code.

Dr.Binary
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
Dr.Binary
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox

I've successfully extracted 11,788 Shift-JIS strings from the NEKO.KSC file with all special codes properly replaced.

Extraction Summary

Special codes replaced:

  • [VARn] = 16 14 n 00 16 → 5,490 occurrences
  • [XX] = 14 XX 00 (hex variables) → 778 occurrences
  • [MEOW] = 17 17 00 → 3,170 occurrences

Output files created:

  • neko_strings.csv - CSV format (Offset, Length, Text, RawHex)
  • neko_strings.txt - Human-readable format

Sample Output

OffsetLengthText
0x0002F517アイドルで [VAR1]
0x00032813歌手で [VAR1]
0x0012948「[07]」
0x0029274[01]
0x003E4A30何を おしえて くれるの[MEOW]？
0x003EAE32何を おしえて くれるのか[MEOW]〜
0x003F2216気になる[MEOW]〜

Hex Variable Distribution

CodeCount
[01]209
[00]189
[02]116
[03]87
[04]76
[05]35
[06]34
[07]17

The extraction correctly handles:

  • Strings starting after 0x48 or 0x8E bytes
  • Shift-JIS encoding (including where 0x48/0x8E appear as valid trail bytes)
  • All three special code patterns with proper continuation after each code
  • Null byte termination (0x00)

Revised after guardrail review

drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox

I have successfully extracted 11,763 Shift-JIS strings from the NEKO.KSC file with all corrections applied.

Key Fixes Applied

  1. MEOW code pattern: Corrected to 17 17 00 (two 17 bytes + one null)

  2. 14 XX 00 handling: Now properly checks if preceded by 16 to avoid conflict with the 16 14 XX 00 16 pattern. The 16 14 XX 00 16 pattern is detected first to ensure correct parsing.

  3. Standalone markers filtered: Strings that contain ONLY markers (like [01], [VAR1]) without actual Japanese/ASCII content are now excluded. Only strings with meaningful content are included.

Extraction Summary

Total strings: 11,763

Special codes replaced:

  • [VARn] = 16 14 n 00 16 → 4,805 occurrences
  • [XX] = 14 XX 00 → 728 occurrences (properly distinguished from VAR pattern)
  • [MEOW] = 17 17 00 → 3,167 occurrences

Output Files

  • neko_strings.csv (1.2 MB) - CSV format with Offset, Length, Text, RawHex
  • neko_strings.txt (865 KB) - Human-readable format

Sample Output

OffsetLengthText
0x0002F517アイドルで [VAR1]
0x000C2D12[VAR1]の名前
0x0012948「[07]」
0x003E4A30何を おしえて くれるの[MEOW]？
0x00430410[00]さん、
0x00440B37[00]って [VAR1]と 比べたら どんな人？

Can I have the Python script please?

Dr.Binary
write

The Python script has been saved to /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/extract_sjis_strings.py.

Usage

python3 extract_sjis_strings.py NEKO.KSC
# or with custom output prefix:
python3 extract_sjis_strings.py NEKO.KSC my_output

Features

  • Special code handling with correct priority:

    • [VARn] = 16 14 n 00 16 (detected first to avoid conflicts)
    • [XX] = 14 XX 00 (only when NOT part of VAR pattern)
    • [MEOW] = 17 17 00

  • Proper Shift-JIS support: Bytes 0x48 and 0x8E can appear as trail bytes in Japanese characters

  • String boundaries: Starts after 0x48 or 0x8E, ends at null byte 0x00

  • Smart filtering: Excludes standalone markers that contain only brackets and hex digits without actual text content

  • Dual output:

    • neko_strings.csv - CSV format (Offset, Length, Text, RawHex)
    • neko_strings.txt - Human-readable format

The script is portable and can be used with any similar Shift-JIS encoded binary file.

Line returns or 0A bytes should be visible as [LR] characters. Some strings end with line return which should be visible in the output.

Dr.Binary
read
edit
edit
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
edit
drbinary_run_sandbox
drbinary_run_sandbox
read

The Python script has been updated to support line return (0x0A) bytes, which are now displayed as [LR] in the output.

Updated Script

The script now handles:

  • [VARn] = 16 14 n 00 16 - Variable placeholders
  • [XX] = 14 XX 00 - Hex variables (when not part of VAR pattern)
  • [MEOW] = 17 17 00 - Special command
  • [LR] = 0A - Line return characters

Key Features

  1. Line returns are visible: Strings containing 0x0A bytes show [LR] in the text
  2. Strings ending with [LR]: Many strings now properly end with [LR] showing the line terminator
  3. Standalone [LR] filtered: Strings that contain only [LR] (no other content) are filtered out
  4. Multi-line strings preserved: Strings with embedded [LR] characters remain intact

Sample Output

0x0029B4 |   23 | Text: それは おいといて…[LR]
0x0029EA |   24 | Text: あのね、[LR]あのね…[LR]
0x003E4A |   34 | Text: 何を おしえて くれるの[MEOW]？[LR]
0x003E73 |   55 | Text: なぁに？ なぁに？[LR]何を おしえて くれるの[MEOW]〜[LR]

The script extracted 11,901 strings total, including those with line returns properly marked.

Usage:

python3 extract_sjis_strings.py NEKO.KSC output_prefix

Some text starts incorrectly after a line return or starts with an incorrect byte after 8E

The character "？" contains a 48 byte and is followed by a line return (0A). The correct string in this case would start at 0x3e73 and should be output as "なぁに？ なぁに？[LR]何を おしえて くれるの[MEOW]～[LR]" The code 8540 is converted to a heart symbol "♥".

Dr.Binary

I understand the issue now. The bytes 0x48 and 0x8E can appear as part of valid Shift-JIS sequences, so I need to check the context before treating them as string start markers.

drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox

Revised after guardrail review

drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox

The script has been fixed and is now working correctly. Here's a summary of the changes:

Key Fix

The script now properly validates that 0x48 and 0x8E are actual string start markers and not part of valid Shift-JIS sequences:

  • 0x48: Only treated as a marker if the previous byte is NOT a valid SJIS lead byte (0x81-0x9F, 0xE0-0xFC)
  • 0x8E: Only treated as a marker if the next byte is NOT in the half-width katakana range (0xA1-0xDF)

This correctly handles the "？" character which is encoded as 0x81 0x48 - the 0x48 is now recognized as a trail byte, not a start marker.

Verified Example

The specific string you mentioned is now correctly extracted:

Offset: 0x003E73 | Length:   55 | Text: なぁに？ なぁに？[LR]何を おしえて くれるの[MEOW]〜[LR]

Output Statistics

  • Total strings: 11,989
  • Strings with [VARn]: 5,149
  • Strings with [XX]: 780
  • Strings with [MEOW]: 3,575
  • Strings with [HEART]: 499

Files

  • Python script: /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/extract_sjis_strings.py
  • CSV output: /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/neko_strings.csv
  • Text output: /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/neko_strings.txt

The string at 0xaeaf is actually part of the string 0xaeac. It started after a 8E byte that was part of the 思 character in Shift-JIS encoding.

思 is 8E76 in Shift-JIS encoding. The 8E was the head byte

Dr.Binary
drbinary_run_sandbox

I see the issue now. The is_valid_marker function for 0x8E was only checking if the next byte is half-width katakana (0xA1-0xDF), but it should also check if the next byte is a valid SJIS trail byte. If 0x8E is followed by any valid trail byte (0x40-0x7E, 0x80-0xFC), it forms a valid SJIS character like 思 (0x8E 0x76).

I've updated the script to:

  • Check if 0x8E is followed by a valid SJIS trail byte (not just half-width katakana)
  • If it is, treat it as part of the SJIS character, not a start marker

Let me run the updated script and verify:

drbinary_run_sandbox

Revised after guardrail review

read
write
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox
drbinary_run_sandbox

I see the issue now. The problem is that 0x8E at 0xAEAB should be treated as a marker because it's right after a null terminator. The logic should be: if 0x8E appears right after a null terminator (end of previous string), it's a marker regardless of what follows.

Let me fix this:

drbinary_run_sandbox
drbinary_run_sandbox

The fix is now working correctly!

Verification

Before the fix:

  • 0xAEAC had "片"
  • 0xAEAF incorrectly started a new string with "いの人..."

After the fix:

  • 0xAEAC correctly has "片思いの人" (complete string including the 思 character)
  • No string at 0xAEAF (no split occurred)

The Fix

The is_valid_marker() function now correctly handles 0x8E:

elif b == 0x8E:
    # 0x8E is a marker if:
    # 1. Previous byte is null (end of previous string), OR
    # 2. Next byte is NOT a valid SJIS trail byte
    if pos > 0 and data[pos - 1] == 0x00:
        return True
    if pos + 1 < len(data) and is_sjis_trail(data[pos + 1]):
        return False
    return True

This ensures:

  • 0x8E right after a null terminator (like at 0xAEAB) is treated as a marker
  • 0x8E followed by a valid trail byte (like 0x8E 0x76 = 思) is NOT treated as a marker

Final Output

  • Total strings: 10,717
  • String at 0xAEAC: "片思いの人" ✓
  • No split at 0xAEAF: ✓

The script and output files are updated at:

  • /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/extract_sjis_strings.py
  • /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/neko_strings.csv
  • /data/github-113677241-1503164d-6b2f-48ec-adcf-cae782bdb4b2/neko_strings.txt