I Let AI Bots Crawl My Website for 10 Days

说明与用途

AI 机器人每天访问 k1r4.space 超过 500 次，但 10 天后仍未被搜索引擎索引。AI 机器人超过 Googlebot，IndexNow 对 Bing 和 Yandex 有效，但对 Google 无效。

问题

Zero pages indexed after 10 days of a new domain despite heavy crawler traffic.

结果

AI crawlers outnumbered Googlebot, visiting the site quickly (ClaudeBot found a new blog post in 2 minutes). Despite heavy traffic, no pages were indexed after 10 days, which is expected for a new domain. IndexNow helped Bing and Yandex but not Google.

工作流

• Set up nginx access log parser to capture bot requests every 30 minutes via cron
• Parse last 5000 log lines, filter by known bot names, count requests per bot
• Submit sitemap (33 URLs) via IndexNow
• Monitor crawler activity and IndexNow participation per search engine

证据

Zero pages indexed after 10 days

来源

AI crawlers (ClaudeBot + GPTBot + AzureAI) accounted for 39% of all traffic — nearly four times more than Googlebot.

来源

Google does not participate in IndexNow.

来源