Let the sitemap decide which URLs your bot is allowed to announce

Описание и применение

Бот публиковал ссылки, основываясь на журнале, что приводило к 404. Переключившись на sitemap.xml и проверяя статус HTTP, он теперь публикует только живые страницы.

Задача

A bot announced URLs based on a ledger of slugs, leading to 404s when builds failed or deployments were incomplete.

Результат

The bot now reliably announces only published articles, eliminating 404s and ensuring announcements reflect actual live content.

Рабочий процесс

• Take today’s slugs from state.json
• Fetch https://<site>/sitemap.xml and parse <loc> values
• Find <loc> containing the slug; only those become candidates
• If no candidates, report and drop the site
• Optionally fallback to same‑day <lastmod> entries with a 50% ratio guard
• Perform an HTTP request to evaluate liveness (200‑399 ok, 400‑599 definitive broken)

Доказательства

The bot built announcement URLs from my own bookkeeping…

Источник

Only those become announcement candidates.

Источник

The mechanism was behaving exactly as specified.

Источник