How We Built Perceive: Web Content Extraction for RAG Pipelines

Description and use

Perceive converts web pages into clean Markdown, reducing token waste and improving embedding quality for RAG pipelines.

Problem

Raw HTML sent to RAG pipelines wastes tokens, contaminates embeddings, and breaks chunking, degrading retrieval quality.

Outcome

Perceive reduces token waste by 80–90%, eliminates markup contamination in embeddings, and produces semantically coherent Markdown that improves retrieval performance in RAG pipelines.

Workflow

• 1. Submit URL via API with optional only_main_content flag
• 2. Fetch stage validates HTTP response, handles redirects, rate limits, and robots.txt
• 3. Render stage runs headless browser if JS needed, dismisses cookie banners, scrolls for lazy content, waits for network idle
• 4. Extract stage heuristically identifies content vs chrome using DOM position, text density, and patterns
• 5. Structure stage converts extracted content to Markdown with consistent ATX headings, tables, code blocks, and links
• 6. Return JSON envelope with pre‑signed URL to Markdown and quality metrics

Evidence

The token waste is real but it is not the worst problem. Three failure modes compound each other.

Source

The extraction layer uses a combination of element position in the DOM, text-to-link ratio within elements, text density relative to the page body, and known structural patterns to infer content boundaries.

Source

We enforce consistent Markdown flavour across all output. ATX headings with a space after the hash. Fenced code blocks with language identifiers. GFM-style tables.

Source