Making three years of a Telegram group chat queryable

Description and use

An automated pipeline converts Telegram chat exports into structured Markdown sources for NotebookLM, enabling efficient querying of a multi‑year chat history.

Problem

A long Telegram group chat contains valuable knowledge but is unreadable and unsearchable with Telegram’s basic search.

Outcome

The pipeline successfully transforms raw Telegram exports into structured, word‑limited Markdown sources that can be ingested by NotebookLM, enabling efficient querying of a multi‑year chat history without duplication.

Workflow

• 1. Export chat history from Telegram Desktop (JSON) or macOS app (HTML).
• 2. Parse JSON: handle plain strings and array‑of‑runs for formatted text using a recursive join function.
• 3. Parse HTML: split files by regex, handle sticky senders, forwarded messages, and differing date formats, normalising dates to ISO.
• 4. Render each message to Markdown, measuring word count including separators and frontmatter placeholders.
• 5. Pack messages into source files, ensuring each stays below the 400k‑word limit (NotebookLM default) and never splits a message.
• 6. Use a running word sum and memoised per‑record counts to keep packing O(n).
• 7. Generate filenames encoding index, last‑message date, and first‑message ID (e.g., nomad‑001‑2023‑06‑01t16‑05‑52‑1.md).
• 8. On subsequent runs, read existing filenames, invert the pattern to find the highest index and last cursor, and filter new records strictly after that cursor.
• 9. Upload the packed sources to NotebookLM, creating a queryable knowledge base.

Evidence

The fix is boring in outline: get the history out, turn it into documents, hand them to something that reads — NotebookLM, in my case.

Source

NotebookLM caps a source at 500,000 words — but the number of sources per notebook is the scarcer cap.

Source