Using JSON files instead of a database for 1,699 exam paper records

说明与用途

将 JSON 文件替代数据库管理 1699 条考试试卷记录导致隐藏错误：slug 重复、主页数据缺失以及并发写入导致文件损坏。解决方案包括冲突检测的 slug 生成、写入队列和原子写入。

问题

Managing 1,699 exam paper records with a flat JSON file caused hidden bugs: duplicate slugs, missing homepage data, and file corruption on concurrent writes.

结果

The project continued to run with the flat file approach, but with mitigations that prevented duplicate slugs, missing homepage sections, and file corruption. The architecture remained simple and cost‑effective for the read‑heavy, append‑mostly workload.

工作流

• 1. Read entire JSON file with fs.readFileSync and parse it.
• 2. Generate slugs using a base slug and, if necessary, append the subject code to avoid collisions.
• 3. Build a slug index keyed by file mtime and size to avoid rebuilding on every call.
• 4. Queue write operations with a promise chain to serialize concurrent writes.
• 5. Write to a temporary file and rename to ensure atomicity.
• 6. Restore from a backup if parsing fails.
• 7. Sweep sitemap URLs after any slug change to catch broken links.
• 8. Replace || {} and || [] with explicit error handling for missing keys.

证据

95 papers that existed but had no URL

来源

Two requests writing the file at once will corrupt it

来源

The analytics file went through a schema migration a while back

来源