A 200 Status Code Is Not Proof of Life: A Link‑Triage Case Study

Описание и применение

Студент-учитель создал инструмент, который проверяет URL‑адреса, собирая статус, конечный URL, тип контента и тело, чтобы отличить живые ссылки от тех, что находятся за куки‑забором, показывая, что код 200 не всегда означает доступность.

Задача

Determine whether a URL is dead or behind a cookie wall without opening a browser, and identify where HTTP status codes become unreliable.

Результат

The tool accurately distinguished 200 responses that were actually dead or behind a cookie wall, demonstrating that status codes alone are insufficient for link triage.

Рабочий процесс

• 1. Create deterministic fixtures on localhost:8765 to represent various HTTP scenarios (200, 410, 301 without Location, soft 404, cookie wall, JS app, PDF, slow response).
• 2. Run fixtures.py to start the test server.
• 3. Write linktriage.py to fetch each URL with urllib, recording status, final URL, content type, and body bytes.
• 4. Extract text from HTML responses and search for patterns indicating dead pages or cookie walls.
• 5. Optionally send extracted text to a free model endpoint (MonkeyCode) for classification.
• 6. Aggregate results and report which links are live, dead, or walled.

Доказательства

Two days before term started, I pasted 62 links into the course reading list and told myself I'd check them "later."

Источник

That is the moment I stopped trusting status codes.

Источник

CPython 3.12, standard library only, nothing to pip install.

Источник

Run it in one terminal with python fixtures.py. That file is the whole trick.

Источник