Using JSON files instead of a database for 1,699 exam paper records

Описание и применение

Использование JSON‑файлов вместо базы данных для 1 699 записей экзаменационных материалов привело к скрытым ошибкам в генерации URL, целостности данных и одновременных записях. Решение включало создание индекса для уникальных slug, очередь записи и атомарные операции с файлом.

Задача

Managing a large set of exam papers with flat JSON files caused hidden bugs in URL generation, data consistency, and write concurrency.

Результат

The project remained functional with flat files, but required significant code changes to avoid slug collisions, data drift, and write corruption. The solution improved reliability while keeping the architecture simple.

Рабочий процесс

• 1. Read entire JSON file with fs.readFileSync and parse.
• 2. Generate slugs using a base slug and, if necessary, a code‑qualified suffix.
• 3. Build a primary index keyed by base slug to detect collisions.
• 4. Enqueue write operations to serialize file writes and avoid corruption.
• 5. Use temp file + rename for atomic writes and backup on failure.
• 6. Perform a full sitemap sweep after any URL‑related change to catch broken links.
• 7. Replace silent defaults (|| {}) with explicit error handling for missing keys.

Доказательства

95 papers that existed but had no URL

Источник

Two requests writing the file at once will corrupt it

Источник