Stop Manually Checking Broken Links: A Developer's Guide to Automated 404 Hunting

Description and use

Automated 404 hunting replaces manual link checking after deployments using a curl Bash script, a CORS-proxy browser script, and no-signup tools from Site Tools Box.

Problem

Manual link checking after deployments is slow, error-prone, and leads to undetected 404 errors that harm SEO and user experience.

Outcome

The author replaced manual link clicking with automated 404 hunting using a curl Bash script, a CORS-proxy browser script, and no-signup tools from Site Tools Box, reducing reliance on heavy desktop apps and ad-filled signup sites.

Workflow

• Write a Bash script that reads URLs from urls.txt and uses curl -o /devnull -s -w '%{http_code}' -I to fetch HTTP status codes.
• Run the script and parse output: if status is not 200, print an error; otherwise print success.
• For quick client-side audits, use a public CORS proxy (allorigins.win) to fetch external URLs and extract the HTTP status code via frontend JavaScript.
• For batch checks of up to 20 URLs, use the Bulk HTTP Checker tool from Site Tools Box.
• For single page asset checks (CSS, JS, images), use the Dead Link Checker tool from Site Tools Box.
• Handle results: set up 301 redirects for deleted pages, wait and recheck for temporary 500/502 errors, remove or replace links for permanently dead external sites.

Evidence

If you have terminal access, you can write a quick Bash script to check a list of URLs using curl.

bash
#!/bin/bash
Read from urls.txt
while read -r url; do

-o /dev/null discards the HTML body

-s silences the progress bar

-w writes out the HTTP status code

-I fetches only the headers

status=$(curl -o /dev/null -s -w '%{http_code}' -I "$url")
if [ "$status" -ne 200 ]; then
echo "❌ [$status] $url"
else
echo "✅ [200] $url"
fi
done < urls.txt

To bypass this, you can use a public CORS proxy like allorigins.win to fetch the URL and return the status code via frontend JS:

const proxyUrl = https://api.allorigins.win/get?url=${encodeURIComponent(url)};

I got tired of navigating through ad-ridden sites, so I actually built a couple of lightweight tools for my own workflow and put them on Site Tools Box (a directory of tools with absolutely no sign-up required).

For checking a single page's assets (CSS, JS, images), I use their Dead Link Checker.

For checking a batch of URLs at once (up to 20 URLs), I use their Bulk HTTP Checker.