A 200 Status Code Is Not Proof of Life: A Link‑Triage Case Study

说明与用途

教师开发了一个轻量级 Python 工具，收集状态码、最终 URL、内容类型和正文，以区分活跃链接与被 Cookie 壁垒阻挡的链接，证明仅凭 200 状态码并不可靠。

问题

Determine whether a URL is dead or behind a cookie wall without opening a browser, and identify where HTTP status codes become unreliable.

结果

The tool accurately distinguished 200 responses that were actually dead or behind a cookie wall, demonstrating that status codes alone are insufficient for link triage.

工作流

• 1. Create deterministic fixtures on localhost:8765 to represent various HTTP scenarios (200, 410, 301 without Location, soft 404, cookie wall, JS app, PDF, slow response).
• 2. Run fixtures.py to start the test server.
• 3. Write linktriage.py to fetch each URL with urllib, recording status, final URL, content type, and body bytes.
• 4. Extract text from HTML responses and search for patterns indicating dead pages or cookie walls.
• 5. Optionally send extracted text to a free model endpoint (MonkeyCode) for classification.
• 6. Aggregate results and report which links are live, dead, or walled.

证据

Two days before term started, I pasted 62 links into the course reading list and told myself I'd check them "later."

来源

That is the moment I stopped trusting status codes.

来源

CPython 3.12, standard library only, nothing to pip install.

来源

Run it in one terminal with python fixtures.py. That file is the whole trick.

来源