Turning Any Website into Clean LLM-Ready Markdown for AI RAG Pipelines in Python

Описание и применение

Краулер преобразует любой веб‑сайт в чистый Markdown, удаляя лишние элементы и сохраняя структуру заголовков, кода и таблиц, что повышает эффективность RAG‑пайплайнов.

Задача

Raw HTML wastes up to 78% of the context window in RAG pipelines due to boilerplate like <nav>, <script>, SVG icons, cookie banners, and inline styles.

Результат

The crawler reduces context waste, preserves semantic structure, and outputs clean Markdown ready for RAG pipelines.

Рабочий процесс

• 1. Define NOISE_SELECTORS to remove unwanted tags and classes.
• 2. Fetch the page with requests and parse with BeautifulSoup.
• 3. Decompose all elements matching NOISE_SELECTORS.
• 4. Identify the main content area (main, article, body, or the whole soup).
• 5. Iterate over heading, paragraph, pre, and list elements, converting them to Markdown syntax with appropriate header levels, code fences, and list markers.
• 6. Prepend YAML front‑matter with title and source_url.
• 7. Return the resulting Markdown string (or dict).

Доказательства

feeding raw HTML directly into an embedding model wastes up to 78% of your context window on <nav>, <script>, SVG icons, cookie banners, and inline styles.

Источник

If you simply call soup.get_text(), your headings, bullet lists, and code blocks flatten into an unstructured wall of text.

Источник