Auditing an OSINT Tool Directory for Link Rot and Spam

Описание и применение

Проведён полный аудит каталога инструментов OSINT, выявлены 39 ссылок, ведущих на азартные сайты, 99 ошибок из-за пробелов в URL, и уточнено, что 1 096 403‑ответов не означают «мертвые» ссылки. Использован двухпроходный метод проверки, который снизил ложные положительные результаты.

Задача

Curated OSINT tool directories often contain dead, misdirected, or spammy links that mislead users and degrade trust.

Результат

The audit revealed that 39 links now serve casino spam, 99 were broken due to whitespace, and many 403/429 responses were false positives. A two-pass method significantly improved accuracy, and successor URLs for dead tools were identified.

Рабочий процесс

• 1. Perform a first-pass curl sweep over all URLs, capturing HTTP status and effective URL.
• 2. For any URL that returned non-200 or appeared dead, perform a second pass: try http/https, with/without www, send full browser headers, resolve DNS via multiple resolvers, follow redirects, and check final URL title.
• 3. Identify and correct data-entry errors (trim URLs at first whitespace).
• 4. Flag and document spam content, redirect patterns, and TLS certificate issues.
• 5. Compile findings into a CSV, noting dead tools and their living successors.
• 6. Optionally run a third independent pass to refute false claims.
• 7. Share results with maintainers or publish a cleaned list.

Доказательства

39 links now serve gambling or SEO spam

Источник

99 return HTTP 200

Источник

Pass 1 — cheap, parallel, over everything

Источник