Using JSON files instead of a database for 1,699 exam paper records

Description and use

Using JSON files instead of a database for 1,699 exam paper records caused hidden bugs in slug generation, analytics reading, and concurrent writes. After implementing collision‑aware slug generation, a write queue, and explicit error handling for missing keys, the site now serves all records correctly and protects against file corruption.

Problem

Managing 1,699 exam paper records with flat JSON files caused hidden bugs in slug generation, analytics reading, and concurrent writes, leading to unreachable records, broken homepage sections, and data corruption.

Outcome

The site now serves all 1,699 exam papers correctly, displays accurate download analytics, and prevents file corruption from concurrent writes. The main limitations are process count and the need for manual index rebuilding.

Workflow

• 1. Read entire JSON file with fs.readFileSync and parse it.
• 2. Generate slugs using a base slug and, if necessary, a code‑qualified suffix to avoid collisions.
• 3. Build a slug index keyed by file mtime and size to avoid rebuilding on every call.
• 4. Enqueue write operations via a promise chain to serialize concurrent writes.
• 5. Write to a temporary file and rename to avoid partial writes.
• 6. Restore from a backup if JSON parsing fails.
• 7. Replace silent defaults (|| {}) with explicit error handling for missing keys.

Evidence

95 papers that existed but had no URL

Source

Two requests writing the file at once will corrupt it

Source