How to find broken URLs in your sitemap (free methods first)

Description and use

Checking sitemap URLs identifies 404s, server errors, redirect chains, and noindex, saving crawl budget and improving SEO.

Problem

Sitemap entries that return 404, server errors, redirect chains or noindex waste crawl budget and send mixed signals.

Outcome

Identified and addressed broken sitemap URLs, reducing wasted crawl budget and improving SEO signals.

Workflow

• Check a few URLs by hand using curl
• Run a PowerShell script to fetch HTTP status for each sitemap URL
• Use free desktop crawlers with URL limits
• Use the hosted Apify Actor to check all URLs, including sitemap indexes and .xml.gz files
• Analyze results: flag 404s, server errors, redirects, redirect chains, https-to-http downgrades, noindex headers
• Fix or remove problematic sitemap entries
• Point entries to final URLs, not redirects
• Remove noindex URLs from sitemap
• Re‑run checks on a schedule after releases

Evidence

Sitemap lists the URLs you want search engines to index. Entries that return 404, server errors, redirect chains or noindex waste crawl budget and send mixed signals.

Source

The script prints one line per URL: 200 means OK, 3xx means a redirect, and 404 or 5xx means broken. It was tested in Windows PowerShell 5.1 against a public sitemap, a missing page (404) and an HTTP-to-HTTPS redirect (307).

Source

Broken Link & Sitemap Checker is an Apify Actor (lineagersiui/sitemap-url-health-checker) that does the same job on Apify's servers, and covers the gaps in the script above: ... It accepts sitemap indexes and .xml.gz files, and it honours robots.txt.

Source