How We Built Perceive: Web Content Extraction for RAG Pipelines

Описание и применение

Perceive преобразует веб‑страницы в чистый Markdown, уменьшая токен‑потерю и улучшая качество эмбеддингов для RAG‑конвейеров.

Задача

Raw HTML sent to RAG pipelines wastes tokens, contaminates embeddings, and breaks chunking, degrading retrieval quality.

Результат

Perceive reduces token waste by 80–90%, eliminates markup contamination in embeddings, and produces semantically coherent Markdown that improves retrieval performance in RAG pipelines.

Рабочий процесс

• 1. Submit URL via API with optional only_main_content flag
• 2. Fetch stage validates HTTP response, handles redirects, rate limits, and robots.txt
• 3. Render stage runs headless browser if JS needed, dismisses cookie banners, scrolls for lazy content, waits for network idle
• 4. Extract stage heuristically identifies content vs chrome using DOM position, text density, and patterns
• 5. Structure stage converts extracted content to Markdown with consistent ATX headings, tables, code blocks, and links
• 6. Return JSON envelope with pre‑signed URL to Markdown and quality metrics

Доказательства

The token waste is real but it is not the worst problem. Three failure modes compound each other.

Источник

The extraction layer uses a combination of element position in the DOM, text-to-link ratio within elements, text density relative to the page body, and known structural patterns to infer content boundaries.

Источник

We enforce consistent Markdown flavour across all output. ATX headings with a space after the hash. Fenced code blocks with language identifiers. GFM-style tables.

Источник