OpenClaw AI agent mis‑decoded Shift_JIS pages

说明与用途

OpenClaw 的链接理解步骤将 Shift_JIS 页面误解码为 UTF‑8，导致出现替换字符。改用现有的 decodeResponseBytes 解码器解决了此问题。

问题

OpenClaw’s link‑understanding step decoded Shift_JIS‑encoded pages as UTF‑8, producing replacement characters and garbled model replies.

结果

The bug was caused by using a UTF‑8 decoder on Shift_JIS content. Replacing it with OpenClaw’s robust decodeResponseBytes resolved the issue, restoring correct text extraction and model responses.

工作流

• 1. Agent fetches page via web_fetch. 2. Original code used new TextDecoder().decode(buffer).trim() which defaults to UTF‑8. 3. Shift_JIS bytes were mis‑decoded, producing replacement characters. 4. Updated link‑understanding to import and call decodeResponseBytes(response, buffer).trim(). 5. Decoder checks BOM, Content‑Type charset, and HTML/XML declarations, falling back to UTF‑8 only when necessary. 6. Regression test serves a Shift_JIS page with header content‑type: text/plain; charset=Shift_JIS and body bytes for "日本語"; test now passes. 7. Real‑world validation on live pages shows 0 replacement characters after the fix.

证据

OpenClaw is an open-source AI agent. When a message contains a link, its link-understanding step fetches the page and hands the text to the model, so the agent can answer questions about it.

来源

The Japanese text was gone.

来源

new TextDecoder() with no argument is a UTF-8 decoder. Shift_JIS bytes are not valid UTF-8, so almost every character became �.

来源

It turned out web_fetch, the agent's general web tool, already had a decoder that does this properly.

来源

The regression test serves a page from a local HTTP server with this header: content-type: text/plain; charset=Shift_JIS

来源

On main it fails, with the fix it passes, and nothing else in the suite changes.

来源

Six bytes in a unit test is a start, not proof. I ran the change through the production link-understanding path, with the strict SSRF guard on, against live public pages:

来源

If your agent fetches web pages Check how it turns bytes into text. Both new TextDecoder() and Response.text() decode as UTF-8 no matter what the server says, so an agent built on either one has this bug on legacy-encoded pages.

来源