How to find broken URLs in your sitemap (free methods first)

Описание и применение

Проверка URL‑адресов в карте сайта выявляет 404, ошибки сервера, цепочки редиректов и noindex, что экономит бюджет обхода и улучшает SEO.

Задача

Sitemap entries that return 404, server errors, redirect chains or noindex waste crawl budget and send mixed signals.

Результат

Identified and addressed broken sitemap URLs, reducing wasted crawl budget and improving SEO signals.

Рабочий процесс

• Check a few URLs by hand using curl
• Run a PowerShell script to fetch HTTP status for each sitemap URL
• Use free desktop crawlers with URL limits
• Use the hosted Apify Actor to check all URLs, including sitemap indexes and .xml.gz files
• Analyze results: flag 404s, server errors, redirects, redirect chains, https-to-http downgrades, noindex headers
• Fix or remove problematic sitemap entries
• Point entries to final URLs, not redirects
• Remove noindex URLs from sitemap
• Re‑run checks on a schedule after releases

Доказательства

Sitemap lists the URLs you want search engines to index. Entries that return 404, server errors, redirect chains or noindex waste crawl budget and send mixed signals.

Источник

The script prints one line per URL: 200 means OK, 3xx means a redirect, and 404 or 5xx means broken. It was tested in Windows PowerShell 5.1 against a public sitemap, a missing page (404) and an HTTP-to-HTTPS redirect (307).

Источник

Broken Link & Sitemap Checker is an Apify Actor (lineagersiui/sitemap-url-health-checker) that does the same job on Apify's servers, and covers the gaps in the script above: ... It accepts sitemap indexes and .xml.gz files, and it honours robots.txt.

Источник