Using JSON files instead of a database for 1,699 exam paper records

Description and use

Using a JSON file instead of a database for 1,699 exam paper records caused hidden bugs: duplicate slugs, missing homepage data, and file corruption on concurrent writes. Fixes include collision‑aware slug generation, a write queue, and atomic temp‑file writes.

Problem

Managing 1,699 exam paper records with a flat JSON file caused hidden bugs: duplicate slugs, missing homepage data, and file corruption on concurrent writes.

Outcome

The project continued to run with the flat file approach, but with mitigations that prevented duplicate slugs, missing homepage sections, and file corruption. The architecture remained simple and cost‑effective for the read‑heavy, append‑mostly workload.

Workflow

• 1. Read entire JSON file with fs.readFileSync and parse it.
• 2. Generate slugs using a base slug and, if necessary, append the subject code to avoid collisions.
• 3. Build a slug index keyed by file mtime and size to avoid rebuilding on every call.
• 4. Queue write operations with a promise chain to serialize concurrent writes.
• 5. Write to a temporary file and rename to ensure atomicity.
• 6. Restore from a backup if parsing fails.
• 7. Sweep sitemap URLs after any slug change to catch broken links.
• 8. Replace || {} and || [] with explicit error handling for missing keys.

Evidence

95 papers that existed but had no URL

Source

Two requests writing the file at once will corrupt it

Source

The analytics file went through a schema migration a while back

Source