OpenClaw AI agent mis‑decoded Shift_JIS pages

Description and use

OpenClaw’s link‑understanding step mis‑decoded Shift_JIS pages as UTF‑8, producing replacement characters. Switching to the existing decodeResponseBytes decoder fixed the issue.

Problem

OpenClaw’s link‑understanding step decoded Shift_JIS‑encoded pages as UTF‑8, producing replacement characters and garbled model replies.

Outcome

The bug was caused by using a UTF‑8 decoder on Shift_JIS content. Replacing it with OpenClaw’s robust decodeResponseBytes resolved the issue, restoring correct text extraction and model responses.

Workflow

• 1. Agent fetches page via web_fetch. 2. Original code used new TextDecoder().decode(buffer).trim() which defaults to UTF‑8. 3. Shift_JIS bytes were mis‑decoded, producing replacement characters. 4. Updated link‑understanding to import and call decodeResponseBytes(response, buffer).trim(). 5. Decoder checks BOM, Content‑Type charset, and HTML/XML declarations, falling back to UTF‑8 only when necessary. 6. Regression test serves a Shift_JIS page with header content‑type: text/plain; charset=Shift_JIS and body bytes for "日本語"; test now passes. 7. Real‑world validation on live pages shows 0 replacement characters after the fix.

Evidence

OpenClaw is an open-source AI agent. When a message contains a link, its link-understanding step fetches the page and hands the text to the model, so the agent can answer questions about it.

Source

The Japanese text was gone.

Source

new TextDecoder() with no argument is a UTF-8 decoder. Shift_JIS bytes are not valid UTF-8, so almost every character became �.

Source

It turned out web_fetch, the agent's general web tool, already had a decoder that does this properly.

Source

The regression test serves a page from a local HTTP server with this header: content-type: text/plain; charset=Shift_JIS

Source

On main it fails, with the fix it passes, and nothing else in the suite changes.

Source

Six bytes in a unit test is a start, not proof. I ran the change through the production link-understanding path, with the strict SSRF guard on, against live public pages:

Source

If your agent fetches web pages Check how it turns bytes into text. Both new TextDecoder() and Response.text() decode as UTF-8 no matter what the server says, so an agent built on either one has this bug on legacy-encoded pages.

Source