Using JSON files instead of a database for 1,699 exam paper records

Description and use

Using JSON files instead of a database for 1,699 exam paper records caused hidden bugs in URL generation, data consistency, and write concurrency. The solution involved collision‑aware slug generation, a write queue, and atomic file writes.

Problem

Managing a large set of exam papers with flat JSON files caused hidden bugs in URL generation, data consistency, and write concurrency.

Outcome

The project remained functional with flat files, but required significant code changes to avoid slug collisions, data drift, and write corruption. The solution improved reliability while keeping the architecture simple.

Workflow

• 1. Read entire JSON file with fs.readFileSync and parse.
• 2. Generate slugs using a base slug and, if necessary, a code‑qualified suffix.
• 3. Build a primary index keyed by base slug to detect collisions.
• 4. Enqueue write operations to serialize file writes and avoid corruption.
• 5. Use temp file + rename for atomic writes and backup on failure.
• 6. Perform a full sitemap sweep after any URL‑related change to catch broken links.
• 7. Replace silent defaults (|| {}) with explicit error handling for missing keys.

Evidence

95 papers that existed but had no URL

Source

Two requests writing the file at once will corrupt it

Source