A 200 OK is not an article: the bug that made me write my own web layer in Rust

Описание и применение

Сервис Svipall решает проблему «200 OK»‑ответов, которые на самом деле являются стенами проверки браузера, улучшая надёжность и экономичность AI‑агентов.

Задача

Agent tooling silently accepted a 200 OK response that was actually a vendor challenge wall, leading to incorrect content extraction and misleading summaries.

Результат

Svipall exposes the hidden failure mode of silent 200 OK responses, reduces token usage by removing boilerplate and hidden text, remembers routes to avoid repeated work, and enforces per‑address visit budgets, improving reliability and cost efficiency for AI agents.

Рабочий процесс

• 1. Perform HTTP fetch and capture status, headers, and body.
• 2. Analyze body for known wall patterns (e.g., specific headers, text like "Checking your browser").
• 3. Return a JSON object with fields: status, blocked_reason, wall_kind, wall_vendor, wall_evidence, note.
• 4. If blocked, decide whether to retry with a headless browser or abort based on configuration.
• 5. Store route information per domain (cookies, selectors, captcha strategies) in a persistent profile.
• 6. Apply boilerplate removal and query‑based filtering before passing content to the model.
• 7. Track number of visits per address and enforce a budget with decay.
• 8. Log all actions for audit and debugging.

Доказательства

The fetch returns 200 and the body isn’t empty and the extractor finds text in that body and turns the text into clean Markdown. Then the model reads my clean Markdown and writes me a fluent summary that happens to be entirely wrong.

Источник

The honest caveat, which lives in the README too: the classification is heuristic, and a clean verdict doesn’t prove that the records you wanted arrived intact.

Источник

Boilerplate removal has a number attached to it: median ROUGE‑LSum F1 of 0.920 over the 3,975 gradable pages of the SIGIR‑23 gold standard, against 0.732 with the boilerplate removal switched off.

Источник