Making three years of a Telegram group chat queryable

Описание и применение

Построена автоматизированная цепочка, преобразующая экспорт Telegram‑чатов в структурированные Markdown‑файлы, которые загружаются в NotebookLM, позволяя быстро искать ответы в многолетнем чате.

Задача

A long Telegram group chat contains valuable knowledge but is unreadable and unsearchable with Telegram’s basic search.

Результат

The pipeline successfully transforms raw Telegram exports into structured, word‑limited Markdown sources that can be ingested by NotebookLM, enabling efficient querying of a multi‑year chat history without duplication.

Рабочий процесс

• 1. Export chat history from Telegram Desktop (JSON) or macOS app (HTML).
• 2. Parse JSON: handle plain strings and array‑of‑runs for formatted text using a recursive join function.
• 3. Parse HTML: split files by regex, handle sticky senders, forwarded messages, and differing date formats, normalising dates to ISO.
• 4. Render each message to Markdown, measuring word count including separators and frontmatter placeholders.
• 5. Pack messages into source files, ensuring each stays below the 400k‑word limit (NotebookLM default) and never splits a message.
• 6. Use a running word sum and memoised per‑record counts to keep packing O(n).
• 7. Generate filenames encoding index, last‑message date, and first‑message ID (e.g., nomad‑001‑2023‑06‑01t16‑05‑52‑1.md).
• 8. On subsequent runs, read existing filenames, invert the pattern to find the highest index and last cursor, and filter new records strictly after that cursor.
• 9. Upload the packed sources to NotebookLM, creating a queryable knowledge base.

Доказательства

The fix is boring in outline: get the history out, turn it into documents, hand them to something that reads — NotebookLM, in my case.

Источник

NotebookLM caps a source at 500,000 words — but the number of sources per notebook is the scarcer cap.

Источник