PageSift 1.0: the Rust web content engine is feature-complete

Описание и применение

PageSift 1.0 – полностью готовый Rust‑движок для извлечения контента из веб‑страниц, PDF и Office‑файлов, предоставляющий чистый Markdown и структурированные данные для AI‑пайплайнов, с поддержкой OCR, памяти селекторов, мониторинга изменений и прозрачной обработки ошибок.

Задача

Teams needed a self-hosted web content engine that could convert URLs, PDFs, and Office files into clean Markdown and structured data for AI pipelines without per-page pricing or cloud dependencies.

Результат

PageSift 1.0 delivers fast, feature-complete content extraction with OCR, selector memory, change monitoring, and transparent failure handling, outperforming rivals in benchmarks.

Рабочий процесс

• 1. Install PageSift binary or SDK
• 2. Configure API endpoints or SDK usage
• 3. Provide URLs, PDFs, or Office files
• 4. Receive clean Markdown and structured data via HTTP or SDK
• 5. Monitor changes with /v2/monitor
• 6. Handle failures through errors endpoint

Доказательства

PageSift went public as 0.20 on October 20; today it ships 1.0.

Источник

PageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including /s/), an MCP server, and Python / Node.js / Rust SDKs.

Источник

1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is feature completion.

Источник

OCR, built from scratch. Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer.

Источник

Selector extraction with memory. Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports relocated, ambiguous, or missing instead of silently returning wrong data.

Источник

Change monitoring. A /v2/monitor endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.

Источник

Browser actions, sessions, and an agent endpoint across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + [[rules]] system, and per-host secret injection.

Источник

Performance (re-measured 2026-11-04) ... PDF → Markdown, olmOCR-bench 1,269 pages, single thread: 2.4 ms/page vs pypdf 50.0 ms/page.

Источник

In late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it.

Источник