PageSift 1.0: the Rust web content engine is feature-complete

Description and use

PageSift 1.0 is a complete Rust-based web content engine that converts URLs, PDFs, and Office files into clean Markdown and structured data for AI pipelines, featuring OCR, selector memory, change monitoring, and transparent error handling.

Problem

Teams needed a self-hosted web content engine that could convert URLs, PDFs, and Office files into clean Markdown and structured data for AI pipelines without per-page pricing or cloud dependencies.

Outcome

PageSift 1.0 delivers fast, feature-complete content extraction with OCR, selector memory, change monitoring, and transparent failure handling, outperforming rivals in benchmarks.

Workflow

• 1. Install PageSift binary or SDK
• 2. Configure API endpoints or SDK usage
• 3. Provide URLs, PDFs, or Office files
• 4. Receive clean Markdown and structured data via HTTP or SDK
• 5. Monitor changes with /v2/monitor
• 6. Handle failures through errors endpoint

Evidence

PageSift went public as 0.20 on October 20; today it ships 1.0.

Source

PageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including /s/), an MCP server, and Python / Node.js / Rust SDKs.

Source

1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is feature completion.

Source

OCR, built from scratch. Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer.

Source

Selector extraction with memory. Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports relocated, ambiguous, or missing instead of silently returning wrong data.

Source

Change monitoring. A /v2/monitor endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.

Source

Browser actions, sessions, and an agent endpoint across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + [[rules]] system, and per-host secret injection.

Source

Performance (re-measured 2026-11-04) ... PDF → Markdown, olmOCR-bench 1,269 pages, single thread: 2.4 ms/page vs pypdf 50.0 ms/page.

Source

In late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it.

Source