Auditing an OSINT Tool Directory for Link Rot and Spam

说明与用途

对OSINT工具目录进行全面审计，发现39个赌博垃圾链接，99个因空格导致的URL错误，并澄清1096个403响应并非死链。采用两步检查方法降低误报率。

问题

Curated OSINT tool directories often contain dead, misdirected, or spammy links that mislead users and degrade trust.

结果

The audit revealed that 39 links now serve casino spam, 99 were broken due to whitespace, and many 403/429 responses were false positives. A two-pass method significantly improved accuracy, and successor URLs for dead tools were identified.

工作流

• 1. Perform a first-pass curl sweep over all URLs, capturing HTTP status and effective URL.
• 2. For any URL that returned non-200 or appeared dead, perform a second pass: try http/https, with/without www, send full browser headers, resolve DNS via multiple resolvers, follow redirects, and check final URL title.
• 3. Identify and correct data-entry errors (trim URLs at first whitespace).
• 4. Flag and document spam content, redirect patterns, and TLS certificate issues.
• 5. Compile findings into a CSV, noting dead tools and their living successors.
• 6. Optionally run a third independent pass to refute false claims.
• 7. Share results with maintainers or publish a cleaned list.

证据

39 links now serve gambling or SEO spam

来源

99 return HTTP 200

来源

Pass 1 — cheap, parallel, over everything

来源