Serving Markdown to AI agents from a Cloudflare Worker, without a second copy of the site

Описание и применение

Worker Cloudflare преобразует HTML в Markdown по запросу Accept: text/markdown, избавляя от необходимости хранить отдельную копию Markdown.

Задача

AI agents often download full HTML pages and discard navigation, scripts, and styles, wasting bandwidth and processing time. Some agents support Markdown via the Accept: text/markdown header, but sites typically store separate Markdown copies.

Результат

The Worker now serves Markdown to agents on demand, eliminating the need for a separate Markdown repository and reducing bandwidth and processing overhead for AI agents.

Рабочий процесс

• 1. Intercept the request and fetch the asset.
• 2. Verify the request method is GET, the response is OK, the content‑type is text/html, and the Accept header includes text/markdown with a positive quality value.
• 3. If any check fails, return the original HTML response.
• 4. Otherwise, pipe the response body through an HTMLRewriter instance.
• 5. Register handlers to skip non‑content elements (head, nav, footer, scripts, etc.) and to convert headings, paragraphs, lists, links, images, and formatting tags to Markdown syntax.
• 6. Collect head metadata (title, description, canonical URL, JSON‑LD) and prepend it as front‑matter.
• 7. Stream the resulting Markdown back to the client.
• 8. All other assets (images, CSS, sitemap) pass through unchanged.

Доказательства

AI agents read web pages the expensive way. They download the HTML, throw away the navigation, scripts and styles, and try to recover the text.

Источник

I wanted our landing page to answer that request without keeping a second Markdown copy of the site in sync by hand.

Источник

The rule is narrow on purpose. Convert only a successful GET for an HTML asset, and only when the client actually asked for Markdown.

Источник

Images, CSS, the sitemap and everything else pass through untouched, even when the request prefers Markdown.

Источник

The mapping itself is ordinary. Headings become # prefixes. Paragraphs and sections become blank lines. strong and em become ** and *, and pre becomes a fenced block.

Источник