Implementing and Measuring llms.txt Adoption

Description and use

The author highlights the difficulty in measuring llms.txt adoption. Standard analytics fail to track AI crawler requests to static files. The proposed solution is to use serverless functions to log User-Agent headers, providing visibility into crawler activity.

Problem

Developers can easily implement an llms.txt file to guide AI crawlers, but lack the instrumentation to verify if crawlers (like GPTBot or ClaudeBot) actually fetch and read the file, making it impossible to measure its effectiveness.

Outcome

The author concludes that while shipping llms.txt is low-cost and beneficial, the industry lacks standard verification methods. The key takeaway is to treat instrumentation as part of the deployment process, using serverless functions to log crawler access rather than relying on standard web analytics.

Workflow

• Create and curate an llms.txt file with high-signal links, avoiding sitemap dumps.
• Deploy the file to the root of the website.
• Recognize that standard analytics (GA4, basic hosting logs) do not track static file requests from AI bots.
• Implement a serverless function (e.g., Cloudflare Pages Function) to intercept requests to /llms.txt.
• Log the User-Agent header for each request to verify crawler activity.
• Analyze logs to confirm if and when AI crawlers access the file.

Evidence

GA4 fires on JavaScript execution in a browser; a crawler doing a raw HTTP fetch of a static Markdown file never runs a line of JS, so it never reports.

Source

The version I actually plan to try next: a Cloudflare Pages Function in front of the /llms.txt route that logs the User-Agent header on every request before serving the file.

Source