Python script vs Firecrawl for llms.txt in 2026: which I use

说明与用途

使用 Python 脚本在 CI 中自动生成 llms.txt 文件，满足规范，包含所有页面、可选更改日志条目，且文件大小低于 4 KB。

问题

Generate a compliant llms.txt file for a 38‑page Markdown docs site, ensuring all pages are linked with concise descriptions, changelog pages are optional, size under 4 KB, and the file is regenerated automatically on every deploy.

结果

The Python script, after a second pass, produced a correct llms.txt file that met all specifications and was cost‑free per deploy. Firecrawl was considered but not used because the source was under the author's control.

工作流

• 1. Write a Python script that walks the docs folder, reads YAML‑ish front matter for titles and descriptions, and builds Markdown sections.
• 2. Handle special cases: lowercase folder names, optional changelog section, and missing descriptions.
• 3. Output the llms.txt file with the required H1, optional blockquote, and H2 sections.
• 4. Configure CI to run the script on every deploy, ensuring the file stays up to date.
• 5. Optionally use Firecrawl’s hosted generator when source control is not available or a live site is needed.
• 6. Review and approve the generated file before merge.

证据

If your docs live in a repo, write a small script and run it in CI. Mine took 47 minutes to get right and now costs nothing per deploy.

来源

The spec at llmstxt.org is short, and you should read it yourself, but here's the shape: ... The file lives at /llms.txt and is plain Markdown.

来源

I ran two approaches against that list: a script I wrote, and Firecrawl's hosted generator.

来源