From a Pile of Office Files to a Searchable Local Knowledge Base: I Built an Open-Source Skill

说明与用途

作者构建了一个自托管知识库，使用 FTS5 和向量搜索索引 Office/PDF 文件，实现快速检索和带来源的答案，同时数据保留在本地磁盘。

问题

Tens of gigabytes of Office/PDF files were unsearchable; filenames and memory were the only cues, making it hard to answer specific questions about content.

结果

The pipeline reduced candidate pool size, cut latency from 96–203 ms to 24–96 ms, and enabled previously unanswerable queries to retrieve correct documents. Retrieval quality was improved by addressing tokenization and document‑size bias.

工作流

• Full probe inventory and format triage
• SHA‑256 deduplication
• Stratified sample reads with SHA verification
• Auto‑triage into useful/junk/undecided piles
• Batch adjudication with 28 rules
• Entity card generation (5 types, 15 YAML fields)
• Optional presentation layer in Obsidian

证据

I had tens of GB of Office/PDF files sitting around: proposals, contracts, notices, monthly reports.

来源

files stay in place, read‑only; the index and vectors live separately; at query time, two paths retrieve and fuse.

来源

Latency 96–203 ms → 24–96 ms (it got faster, because the candidate pool shrank from thousands to hundreds); three groups of previously unanswerable questions all hit the right documents.

来源

The data doesn't leave home in bulk, but the snippets hit at query time — and your question — do go out.

来源