I Let AI Bots Crawl My Website for 10 Days

Description and use

AI bots crawled k1r4.space over 500 times daily, yet no pages were indexed after 10 days. AI crawlers outpaced Googlebot, and IndexNow worked for Bing and Yandex but not Google.

Problem

Zero pages indexed after 10 days of a new domain despite heavy crawler traffic.

Outcome

AI crawlers outnumbered Googlebot, visiting the site quickly (ClaudeBot found a new blog post in 2 minutes). Despite heavy traffic, no pages were indexed after 10 days, which is expected for a new domain. IndexNow helped Bing and Yandex but not Google.

Workflow

• Set up nginx access log parser to capture bot requests every 30 minutes via cron
• Parse last 5000 log lines, filter by known bot names, count requests per bot
• Submit sitemap (33 URLs) via IndexNow
• Monitor crawler activity and IndexNow participation per search engine

Evidence

Zero pages indexed after 10 days

Source

AI crawlers (ClaudeBot + GPTBot + AzureAI) accounted for 39% of all traffic — nearly four times more than Googlebot.

Source

Google does not participate in IndexNow.

Source