A 200 OK is not an article: the bug that made me write my own web layer in Rust

Description and use

Svipall addresses the silent 200 OK bug that returns challenge walls, improving reliability and cost efficiency for AI agents.

Problem

Agent tooling silently accepted a 200 OK response that was actually a vendor challenge wall, leading to incorrect content extraction and misleading summaries.

Outcome

Svipall exposes the hidden failure mode of silent 200 OK responses, reduces token usage by removing boilerplate and hidden text, remembers routes to avoid repeated work, and enforces per‑address visit budgets, improving reliability and cost efficiency for AI agents.

Workflow

• 1. Perform HTTP fetch and capture status, headers, and body.
• 2. Analyze body for known wall patterns (e.g., specific headers, text like "Checking your browser").
• 3. Return a JSON object with fields: status, blocked_reason, wall_kind, wall_vendor, wall_evidence, note.
• 4. If blocked, decide whether to retry with a headless browser or abort based on configuration.
• 5. Store route information per domain (cookies, selectors, captcha strategies) in a persistent profile.
• 6. Apply boilerplate removal and query‑based filtering before passing content to the model.
• 7. Track number of visits per address and enforce a budget with decay.
• 8. Log all actions for audit and debugging.

Evidence

The fetch returns 200 and the body isn’t empty and the extractor finds text in that body and turns the text into clean Markdown. Then the model reads my clean Markdown and writes me a fluent summary that happens to be entirely wrong.

Source

The honest caveat, which lives in the README too: the classification is heuristic, and a clean verdict doesn’t prove that the records you wanted arrived intact.

Source

Boilerplate removal has a number attached to it: median ROUGE‑LSum F1 of 0.920 over the 3,975 gradable pages of the SIGIR‑23 gold standard, against 0.732 with the boilerplate removal switched off.

Source