Auditing an OSINT Tool Directory for Link Rot and Spam

Description and use

A comprehensive audit of an OSINT tool directory uncovered 39 gambling spam links, 99 broken URLs due to whitespace, and clarified that 1,096 403 responses are not dead links. A two-pass checking method reduced false positives.

Problem

Curated OSINT tool directories often contain dead, misdirected, or spammy links that mislead users and degrade trust.

Outcome

The audit revealed that 39 links now serve casino spam, 99 were broken due to whitespace, and many 403/429 responses were false positives. A two-pass method significantly improved accuracy, and successor URLs for dead tools were identified.

Workflow

• 1. Perform a first-pass curl sweep over all URLs, capturing HTTP status and effective URL.
• 2. For any URL that returned non-200 or appeared dead, perform a second pass: try http/https, with/without www, send full browser headers, resolve DNS via multiple resolvers, follow redirects, and check final URL title.
• 3. Identify and correct data-entry errors (trim URLs at first whitespace).
• 4. Flag and document spam content, redirect patterns, and TLS certificate issues.
• 5. Compile findings into a CSV, noting dead tools and their living successors.
• 6. Optionally run a third independent pass to refute false claims.
• 7. Share results with maintainers or publish a cleaned list.

Evidence

39 links now serve gambling or SEO spam

Source

99 return HTTP 200

Source

Pass 1 — cheap, parallel, over everything

Source