Turning Any Website into Clean LLM-Ready Markdown for AI RAG Pipelines in Python

Description and use

A crawler converts any website into clean Markdown, stripping boilerplate while preserving headings, code blocks, and tables, improving RAG pipeline efficiency.

Problem

Raw HTML wastes up to 78% of the context window in RAG pipelines due to boilerplate like <nav>, <script>, SVG icons, cookie banners, and inline styles.

Outcome

The crawler reduces context waste, preserves semantic structure, and outputs clean Markdown ready for RAG pipelines.

Workflow

• 1. Define NOISE_SELECTORS to remove unwanted tags and classes.
• 2. Fetch the page with requests and parse with BeautifulSoup.
• 3. Decompose all elements matching NOISE_SELECTORS.
• 4. Identify the main content area (main, article, body, or the whole soup).
• 5. Iterate over heading, paragraph, pre, and list elements, converting them to Markdown syntax with appropriate header levels, code fences, and list markers.
• 6. Prepend YAML front‑matter with title and source_url.
• 7. Return the resulting Markdown string (or dict).

Evidence

feeding raw HTML directly into an embedding model wastes up to 78% of your context window on <nav>, <script>, SVG icons, cookie banners, and inline styles.

Source

If you simply call soup.get_text(), your headings, bullet lists, and code blocks flatten into an unstructured wall of text.

Source