We put a 62-page Home Office PDF into our RAG index the afternoon it came out

Описание и применение

В течение часа мы добавили 62‑страничный PDF из Home Office в индекс RAG, исправив ошибки nightly‑job, очистив текст, переформатировав таблицы и используя существующие функции chunking и embedding.

Задача

The nightly ingest missed a 62‑page Home Office PDF containing the new ILR rules, so the RAG index was out of date when the decision was published.

Результат

The PDF was indexed within an hour, producing 91 passages of up to 1,500 characters. Retrieval queries returned relevant chapters with high similarity scores, confirming the index worked as intended.

Рабочий процесс

• 1. Fetch the consultation page via the GOV.UK content API to get the PDF URL.
• 2. Download the PDF and extract raw text with pypdf.
• 3. Clean the text: drop page markers, rejoin wrapped lines into paragraphs, skip content before the real "Introduction" heading.
• 4. Handle footnotes that look like headings by ignoring lines that match "Chapter N:" and end with a full stop.
• 5. Reformat tables: keep original text but insert a plain‑text version of the key table with one pathway per line.
• 6. Build document sections (title, url, sections of text).
• 7. Pass sections to the existing chunking, embedding and upsert functions, which use uuid5(url + "#section.chunk") for point ids.
• 8. After successful upsert, delete old points for the same URL to avoid duplication.
• 9. Verify by probing with typical user questions and checking top hits.
• 10. Publish the updated content on the website.

Доказательства

Our ingest reads the GOV.UK search API for one organisation, UK Visas and Immigration, then pulls each page body from the content API. This publication came from the Home Office, not UKVI.

Источник

pypdf gets the text. It does not get you paragraphs. What came out had: a page marker and a bare page number every 40 lines; every line hard‑wrapped at the PDF's column width; a contents list at the front that repeats every heading.

Источник

We kept the original text and put a plain version of the table above it, one pathway per line, with the figures copied exactly.

Источник

Point ids are uuid5(url + "#section.chunk"), so the same passage always lands on the same id.

Источник

under a minute to embed and upload

Источник