From a Pile of Office Files to a Searchable Local Knowledge Base: I Built an Open-Source Skill

Description and use

The author built a self‑hosted knowledge base that indexes Office/PDF files, enabling keyword and semantic search with source citations, while keeping data on local disk.

Problem

Tens of gigabytes of Office/PDF files were unsearchable; filenames and memory were the only cues, making it hard to answer specific questions about content.

Outcome

The pipeline reduced candidate pool size, cut latency from 96–203 ms to 24–96 ms, and enabled previously unanswerable queries to retrieve correct documents. Retrieval quality was improved by addressing tokenization and document‑size bias.

Workflow

• Full probe inventory and format triage
• SHA‑256 deduplication
• Stratified sample reads with SHA verification
• Auto‑triage into useful/junk/undecided piles
• Batch adjudication with 28 rules
• Entity card generation (5 types, 15 YAML fields)
• Optional presentation layer in Obsidian

Evidence

I had tens of GB of Office/PDF files sitting around: proposals, contracts, notices, monthly reports.

Source

files stay in place, read‑only; the index and vectors live separately; at query time, two paths retrieve and fuse.

Source

Latency 96–203 ms → 24–96 ms (it got faster, because the candidate pool shrank from thousands to hundreds); three groups of previously unanswerable questions all hit the right documents.

Source

The data doesn't leave home in bulk, but the snippets hit at query time — and your question — do go out.

Source