Implementing and Measuring llms.txt Adoption

说明与用途

作者指出了衡量 llms.txt 采用率的困难。标准分析工具无法跟踪 AI 爬虫对静态文件的请求。提出的解决方案是使用无服务器函数记录 User-Agent 标头，从而提供对爬虫活动的可见性。

问题

Developers can easily implement an llms.txt file to guide AI crawlers, but lack the instrumentation to verify if crawlers (like GPTBot or ClaudeBot) actually fetch and read the file, making it impossible to measure its effectiveness.

结果

The author concludes that while shipping llms.txt is low-cost and beneficial, the industry lacks standard verification methods. The key takeaway is to treat instrumentation as part of the deployment process, using serverless functions to log crawler access rather than relying on standard web analytics.

工作流

• Create and curate an llms.txt file with high-signal links, avoiding sitemap dumps.
• Deploy the file to the root of the website.
• Recognize that standard analytics (GA4, basic hosting logs) do not track static file requests from AI bots.
• Implement a serverless function (e.g., Cloudflare Pages Function) to intercept requests to /llms.txt.
• Log the User-Agent header for each request to verify crawler activity.
• Analyze logs to confirm if and when AI crawlers access the file.

证据

GA4 fires on JavaScript execution in a browser; a crawler doing a raw HTTP fetch of a static Markdown file never runs a line of JS, so it never reports.

来源

The version I actually plan to try next: a Cloudflare Pages Function in front of the /llms.txt route that logs the User-Agent header on every request before serving the file.

来源