OpenClaw AI agent mis‑decoded Shift_JIS pages

Описание и применение

В OpenClaw ссылка‑понимание декодировало страницы Shift_JIS как UTF‑8, создавая символы замены. Замена декодера на существующий decodeResponseBytes исправила проблему.

Задача

OpenClaw’s link‑understanding step decoded Shift_JIS‑encoded pages as UTF‑8, producing replacement characters and garbled model replies.

Результат

The bug was caused by using a UTF‑8 decoder on Shift_JIS content. Replacing it with OpenClaw’s robust decodeResponseBytes resolved the issue, restoring correct text extraction and model responses.

Рабочий процесс

• 1. Agent fetches page via web_fetch. 2. Original code used new TextDecoder().decode(buffer).trim() which defaults to UTF‑8. 3. Shift_JIS bytes were mis‑decoded, producing replacement characters. 4. Updated link‑understanding to import and call decodeResponseBytes(response, buffer).trim(). 5. Decoder checks BOM, Content‑Type charset, and HTML/XML declarations, falling back to UTF‑8 only when necessary. 6. Regression test serves a Shift_JIS page with header content‑type: text/plain; charset=Shift_JIS and body bytes for "日本語"; test now passes. 7. Real‑world validation on live pages shows 0 replacement characters after the fix.

Доказательства

OpenClaw is an open-source AI agent. When a message contains a link, its link-understanding step fetches the page and hands the text to the model, so the agent can answer questions about it.

Источник

The Japanese text was gone.

Источник

new TextDecoder() with no argument is a UTF-8 decoder. Shift_JIS bytes are not valid UTF-8, so almost every character became �.

Источник

It turned out web_fetch, the agent's general web tool, already had a decoder that does this properly.

Источник

The regression test serves a page from a local HTTP server with this header: content-type: text/plain; charset=Shift_JIS

Источник

On main it fails, with the fix it passes, and nothing else in the suite changes.

Источник

Six bytes in a unit test is a start, not proof. I ran the change through the production link-understanding path, with the strict SSRF guard on, against live public pages:

Источник

If your agent fetches web pages Check how it turns bytes into text. Both new TextDecoder() and Response.text() decode as UTF-8 no matter what the server says, so an agent built on either one has this bug on legacy-encoded pages.

Источник