How I Run a Full Content Pipeline on a $7 VPS With 2GB of RAM (And the 3am Crash That Taught Me Why)

Description and use

A content pipeline on a cheap VPS with 2 GB RAM crashed due to concurrent processing. By serializing tasks, logging peak memory, capping input size, and explicitly invoking garbage collection, peak usage dropped to 29 MB and the pipeline ran reliably.

Problem

A cron job on a 1 vCPU, 2 GB RAM VPS crashed at 3 am, leaving no output and a corrupted swap file.

Outcome

The refactored pipeline runs unattended on the cheap VPS, completing batches without crashes and staying within memory limits.

Workflow

• Process one image at a time
• Log peak RSS per step
• Cap input size and resize before expensive steps
• Call gc.collect() after heavy steps

Evidence

The mistake was concurrency. I processed everything at once, because that is the default when you write a loop.

Source

After the rewrite, generating a full batch of images peaked at 29MB of RSS.

Source