API Limit 21KB! ...Then 32KB of data caused a mysterious error. The culprit? Confusing bytes with characters.

Description and use

The issue stemmed from counting characters instead of bytes for payload size. Switching to a byte‑based check resolved the API error.

Problem

A bot was rejected by an external API with a "Payload too large" error despite a local size check passing, due to confusion between character count and byte count.

Outcome

Switching from character‑based to byte‑based size validation prevented oversized payloads and resolved the API error.

Workflow

• 1. Observe "Payload too large" error in logs. 2. Verify local size check passes using len() on the string. 3. Realize the API limits bytes, not characters. 4. Test with Japanese text to compare character vs byte counts. 5. Write a function that encodes the string to UTF‑8 and measures byte length. 6. Convert the 21 KB limit to bytes (21*1024). 7. Raise ValueError if byte size exceeds limit before API call. 8. Deploy the updated check and confirm the error disappears.

Evidence

I was checking the "character count" with len(), but the API was looking at the "byte count" calculated by len(str.encode('utf-8')).

Source

When I re-measured the actual error‑causing data by byte count, it easily exceeded 21KB, coming in at nearly 32KB.

Source