Running a Full Asset Pipeline on 1 vCPU and 2 GB of RAM

Description and use

Building a reliable digital asset pipeline on a 1 vCPU, 2 GB RAM server, with memory management, automated CSV header generation, and an upload ledger.

Problem

Running an asset pipeline on a minimal 1 vCPU, 2 GB RAM server caused out‑of‑memory crashes, incomplete uploads, and duplicate submissions.

Outcome

The refined pipeline processed 39 products on a 1 vCPU, 2 GB RAM server without OOM crashes, ensured accurate upload tracking, and eliminated duplicate submissions.

Workflow

• 1. Run one image job at a time to avoid CPU contention.
• 2. Call gc.collect() after each asset to free Python references.
• 3. Measure peak RSS with resource.getrusage(RUSAGE_SELF).ru_maxrss to detect memory spikes.
• 4. Execute heavy render steps in separate subprocesses so the OS reclaims memory on exit.
• 5. Render images at the required resolution (e.g., 60 dpi for previews).
• 6. Generate CSV headers automatically for each platform (Adobe, Vecteezy, Dreamstime).
• 7. Maintain an upload ledger: check before upload, mark after successful upload to prevent duplicates.
• 8. Package assets into zips with a cover (1280×720) and thumbnail (600×600).
• 9. Convert Markdown guides to styled PDFs using ReportLab in a subprocess.
• 10. Keep dependency list minimal (Pillow, ReportLab) and use only the standard library for other tasks.

Evidence

I killed my pipeline with a single &.

Source

The whole pipeline that produced 39 finished products, 856 MB of output, ran on 1 vCPU and 2 GB of RAM.

Source

One process at a time. Never parallelize image jobs on a single core.

Source

Render at the resolution you need. I used to render everything at 300 dpi just to inspect it.

Source

The fastest way to lose a day is a rejected upload batch. All three major channels validate the CSV header before they look at a single file.

Source

The fix is boring and it works: a ledger. Before each upload, check the ledger. After each upload, mark it done.

Source