We put a 62-page Home Office PDF into our RAG index the afternoon it came out

Description and use

Within an hour we added a 62‑page Home Office PDF to the RAG index, fixing nightly‑job gaps, cleaning text, reformatting tables, and reusing existing chunking and embedding functions.

Problem

The nightly ingest missed a 62‑page Home Office PDF containing the new ILR rules, so the RAG index was out of date when the decision was published.

Outcome

The PDF was indexed within an hour, producing 91 passages of up to 1,500 characters. Retrieval queries returned relevant chapters with high similarity scores, confirming the index worked as intended.

Workflow

• 1. Fetch the consultation page via the GOV.UK content API to get the PDF URL.
• 2. Download the PDF and extract raw text with pypdf.
• 3. Clean the text: drop page markers, rejoin wrapped lines into paragraphs, skip content before the real "Introduction" heading.
• 4. Handle footnotes that look like headings by ignoring lines that match "Chapter N:" and end with a full stop.
• 5. Reformat tables: keep original text but insert a plain‑text version of the key table with one pathway per line.
• 6. Build document sections (title, url, sections of text).
• 7. Pass sections to the existing chunking, embedding and upsert functions, which use uuid5(url + "#section.chunk") for point ids.
• 8. After successful upsert, delete old points for the same URL to avoid duplication.
• 9. Verify by probing with typical user questions and checking top hits.
• 10. Publish the updated content on the website.

Evidence

Our ingest reads the GOV.UK search API for one organisation, UK Visas and Immigration, then pulls each page body from the content API. This publication came from the Home Office, not UKVI.

Source

pypdf gets the text. It does not get you paragraphs. What came out had: a page marker and a bare page number every 40 lines; every line hard‑wrapped at the PDF's column width; a contents list at the front that repeats every heading.

Source

We kept the original text and put a plain version of the table above it, one pathway per line, with the figures copied exactly.

Source

Point ids are uuid5(url + "#section.chunk"), so the same passage always lands on the same id.

Source

under a minute to embed and upload

Source