Serving Markdown to AI agents from a Cloudflare Worker, without a second copy of the site

Description and use

A Cloudflare Worker intercepts requests with Accept: text/markdown and streams a Markdown conversion of the HTML using HTMLRewriter, eliminating the need for a separate Markdown copy.

Problem

AI agents often download full HTML pages and discard navigation, scripts, and styles, wasting bandwidth and processing time. Some agents support Markdown via the Accept: text/markdown header, but sites typically store separate Markdown copies.

Outcome

The Worker now serves Markdown to agents on demand, eliminating the need for a separate Markdown repository and reducing bandwidth and processing overhead for AI agents.

Workflow

• 1. Intercept the request and fetch the asset.
• 2. Verify the request method is GET, the response is OK, the content‑type is text/html, and the Accept header includes text/markdown with a positive quality value.
• 3. If any check fails, return the original HTML response.
• 4. Otherwise, pipe the response body through an HTMLRewriter instance.
• 5. Register handlers to skip non‑content elements (head, nav, footer, scripts, etc.) and to convert headings, paragraphs, lists, links, images, and formatting tags to Markdown syntax.
• 6. Collect head metadata (title, description, canonical URL, JSON‑LD) and prepend it as front‑matter.
• 7. Stream the resulting Markdown back to the client.
• 8. All other assets (images, CSS, sitemap) pass through unchanged.

Evidence

AI agents read web pages the expensive way. They download the HTML, throw away the navigation, scripts and styles, and try to recover the text.

Source

I wanted our landing page to answer that request without keeping a second Markdown copy of the site in sync by hand.

Source

The rule is narrow on purpose. Convert only a successful GET for an HTML asset, and only when the client actually asked for Markdown.

Source

Images, CSS, the sitemap and everything else pass through untouched, even when the request prefers Markdown.

Source

The mapping itself is ordinary. Headings become # prefixes. Paragraphs and sections become blank lines. strong and em become ** and *, and pre becomes a fenced block.

Source