Using JSON files instead of a database for 1,699 exam paper records

Описание и применение

Использование JSON‑файла вместо базы данных для 1699 записей экзаменационных материалов привело к скрытым ошибкам: дублирование slug‑ов, отсутствие данных на главной странице и повреждение файла при одновременных записях. Решения включают проверку уникальности slug‑ов, очередь записей и атомарную запись через временный файл.

Задача

Managing 1,699 exam paper records with a flat JSON file caused hidden bugs: duplicate slugs, missing homepage data, and file corruption on concurrent writes.

Результат

The project continued to run with the flat file approach, but with mitigations that prevented duplicate slugs, missing homepage sections, and file corruption. The architecture remained simple and cost‑effective for the read‑heavy, append‑mostly workload.

Рабочий процесс

• 1. Read entire JSON file with fs.readFileSync and parse it.
• 2. Generate slugs using a base slug and, if necessary, append the subject code to avoid collisions.
• 3. Build a slug index keyed by file mtime and size to avoid rebuilding on every call.
• 4. Queue write operations with a promise chain to serialize concurrent writes.
• 5. Write to a temporary file and rename to ensure atomicity.
• 6. Restore from a backup if parsing fails.
• 7. Sweep sitemap URLs after any slug change to catch broken links.
• 8. Replace || {} and || [] with explicit error handling for missing keys.

Доказательства

95 papers that existed but had no URL

Источник

Two requests writing the file at once will corrupt it

Источник

The analytics file went through a schema migration a while back

Источник