I Let AI Bots Crawl My Website for 10 Days

Описание и применение

AI‑боты посещали сайт k1r4.space 500+ раз в день, но после 10 дней ни одна страница не была проиндексирована поисковыми системами. AI‑боты (ClaudeBot, GPTBot, AzureAI) превзошли Googlebot, а IndexNow помогал Bing и Yandex, но не Google.

Задача

Zero pages indexed after 10 days of a new domain despite heavy crawler traffic.

Результат

AI crawlers outnumbered Googlebot, visiting the site quickly (ClaudeBot found a new blog post in 2 minutes). Despite heavy traffic, no pages were indexed after 10 days, which is expected for a new domain. IndexNow helped Bing and Yandex but not Google.

Рабочий процесс

• Set up nginx access log parser to capture bot requests every 30 minutes via cron
• Parse last 5000 log lines, filter by known bot names, count requests per bot
• Submit sitemap (33 URLs) via IndexNow
• Monitor crawler activity and IndexNow participation per search engine

Доказательства

Zero pages indexed after 10 days

Источник

AI crawlers (ClaudeBot + GPTBot + AzureAI) accounted for 39% of all traffic — nearly four times more than Googlebot.

Источник

Google does not participate in IndexNow.

Источник