From a Pile of Office Files to a Searchable Local Knowledge Base: I Built an Open-Source Skill

Описание и применение

Автор создал локальный поиск по офисным и PDF‑файлам, используя FTS5 и векторный поиск, чтобы быстро находить нужные документы и отвечать на вопросы с указанием источников, при этом данные остаются на локальном диске.

Задача

Tens of gigabytes of Office/PDF files were unsearchable; filenames and memory were the only cues, making it hard to answer specific questions about content.

Результат

The pipeline reduced candidate pool size, cut latency from 96–203 ms to 24–96 ms, and enabled previously unanswerable queries to retrieve correct documents. Retrieval quality was improved by addressing tokenization and document‑size bias.

Рабочий процесс

• Full probe inventory and format triage
• SHA‑256 deduplication
• Stratified sample reads with SHA verification
• Auto‑triage into useful/junk/undecided piles
• Batch adjudication with 28 rules
• Entity card generation (5 types, 15 YAML fields)
• Optional presentation layer in Obsidian

Доказательства

I had tens of GB of Office/PDF files sitting around: proposals, contracts, notices, monthly reports.

Источник

files stay in place, read‑only; the index and vectors live separately; at query time, two paths retrieve and fuse.

Источник

Latency 96–203 ms → 24–96 ms (it got faster, because the candidate pool shrank from thousands to hundreds); three groups of previously unanswerable questions all hit the right documents.

Источник

The data doesn't leave home in bulk, but the snippets hit at query time — and your question — do go out.

Источник