My Agent Spawned 400 Workers Overnight. The OOM Killer Killed My Database and Left the Agent Running.

Description and use

Applying OS‑level hard limits (MemoryMax, TasksMax) and protecting the database with OOMScoreAdjust prevented the Pi from becoming unresponsive and preserved data integrity. The agent now fails fast and logs exceptions instead of silently consuming resources.

Problem

A document‑processing agent on a Raspberry Pi spawned 400 worker processes during a bulk import, exhausting memory and causing the OOM killer to kill Postgres repeatedly, leaving the agent alive but the database corrupted and jobs stuck.

Outcome

Applying OS‑level hard limits (MemoryMax, TasksMax) and protecting the database with OOMScoreAdjust prevented the Pi from becoming unresponsive and preserved data integrity. The agent now fails fast and logs exceptions instead of silently consuming resources.

Workflow

• 1. Bulk import dropped 400 documents into the queue.
• 2. Agent looped over pending rows without checking MAX_WORKERS and spawned a subprocess for each.
• 3. 400 headless browsers consumed ~200‑350 MB RSS each, exhausting RAM and triggering swap thrashing.
• 4. OOM killer targeted the largest process (Postgres) based on oom_score, killing it repeatedly.
• 5. Agent remained alive, but database lost recent writes and jobs were stuck.
• 6. After recovery, the author identified three mistakes: wrong layer for concurrency limit, misleading heartbeat monitoring, and insufficient load testing.
• 7. Implemented OS‑level hard limits via systemd drop‑in: MemoryMax=2G, MemoryHigh=1.5G, TasksMax=64, OOMPolicy=stop, Restart=on‑failure.
• 8. Adjusted Postgres OOMScoreAdjust to -800 to protect the database.
• 9. Re‑architected job claiming to use systemd unit templates ([email protected]) to enforce a fixed number of workers.
• 10. Added monitoring on queue depth, per‑worker RSS, and job running time.
• 11. Disabled SD‑card swap or enabled zram and set vm.swappiness=10 to avoid thrashing.
• 12. Updated playbooks and documentation.
• 13. Reflected on cost and lessons learned.

Evidence

I set this MAX_WORKERS=4 ... Every code path that actually started a worker did so from a for row in pending: loop with no check at all, because the loop was written before the constant existed and I never went back.

Source

The kernel spent its time paging, not running, and every process slowed to a crawl, including the ones that were almost done.

Source

The fix that actually holds is one the application can't bypass because it isn't in the application. A systemd drop‑in: [Service] MemoryMax=2G MemoryHigh=1.5G TasksMax=64 OOMPolicy=stop Restart=on‑failure RestartSec=10

Source

Postgres gets a lower OOM score so the kernel reaches for something else first: [Service] OOMScoreAdjust=-800

Source