Python script vs Firecrawl for llms.txt in 2026: which I use

Описание и применение

Автоматическое создание файла llms.txt из Markdown‑документов с помощью Python‑скрипта, интегрированного в CI, обеспечивает соответствие спецификации, включает все страницы, опциональные изменения и размер менее 4 КБ.

Задача

Generate a compliant llms.txt file for a 38‑page Markdown docs site, ensuring all pages are linked with concise descriptions, changelog pages are optional, size under 4 KB, and the file is regenerated automatically on every deploy.

Результат

The Python script, after a second pass, produced a correct llms.txt file that met all specifications and was cost‑free per deploy. Firecrawl was considered but not used because the source was under the author's control.

Рабочий процесс

• 1. Write a Python script that walks the docs folder, reads YAML‑ish front matter for titles and descriptions, and builds Markdown sections.
• 2. Handle special cases: lowercase folder names, optional changelog section, and missing descriptions.
• 3. Output the llms.txt file with the required H1, optional blockquote, and H2 sections.
• 4. Configure CI to run the script on every deploy, ensuring the file stays up to date.
• 5. Optionally use Firecrawl’s hosted generator when source control is not available or a live site is needed.
• 6. Review and approve the generated file before merge.

Доказательства

If your docs live in a repo, write a small script and run it in CI. Mine took 47 minutes to get right and now costs nothing per deploy.

Источник

The spec at llmstxt.org is short, and you should read it yourself, but here's the shape: ... The file lives at /llms.txt and is plain Markdown.

Источник

I ran two approaches against that list: a script I wrote, and Firecrawl's hosted generator.

Источник