Turning Any Website into Clean LLM-Ready Markdown for AI RAG Pipelines in Python

说明与用途

爬虫将任何网站转换为干净的Markdown，去除冗余元素，同时保留标题、代码块和表格结构，提升RAG管道效率。

问题

Raw HTML wastes up to 78% of the context window in RAG pipelines due to boilerplate like <nav>, <script>, SVG icons, cookie banners, and inline styles.

结果

The crawler reduces context waste, preserves semantic structure, and outputs clean Markdown ready for RAG pipelines.

工作流

• 1. Define NOISE_SELECTORS to remove unwanted tags and classes.
• 2. Fetch the page with requests and parse with BeautifulSoup.
• 3. Decompose all elements matching NOISE_SELECTORS.
• 4. Identify the main content area (main, article, body, or the whole soup).
• 5. Iterate over heading, paragraph, pre, and list elements, converting them to Markdown syntax with appropriate header levels, code fences, and list markers.
• 6. Prepend YAML front‑matter with title and source_url.
• 7. Return the resulting Markdown string (or dict).

证据

feeding raw HTML directly into an embedding model wastes up to 78% of your context window on <nav>, <script>, SVG icons, cookie banners, and inline styles.

来源

If you simply call soup.get_text(), your headings, bullet lists, and code blocks flatten into an unstructured wall of text.

来源