Running a Full Asset Pipeline on 1 vCPU and 2 GB of RAM

说明与用途

在1 vCPU、2 GB RAM服务器上构建可靠的数字资产流水线，包括内存管理、自动生成CSV头和上传账本。

问题

Running an asset pipeline on a minimal 1 vCPU, 2 GB RAM server caused out‑of‑memory crashes, incomplete uploads, and duplicate submissions.

结果

The refined pipeline processed 39 products on a 1 vCPU, 2 GB RAM server without OOM crashes, ensured accurate upload tracking, and eliminated duplicate submissions.

工作流

• 1. Run one image job at a time to avoid CPU contention.
• 2. Call gc.collect() after each asset to free Python references.
• 3. Measure peak RSS with resource.getrusage(RUSAGE_SELF).ru_maxrss to detect memory spikes.
• 4. Execute heavy render steps in separate subprocesses so the OS reclaims memory on exit.
• 5. Render images at the required resolution (e.g., 60 dpi for previews).
• 6. Generate CSV headers automatically for each platform (Adobe, Vecteezy, Dreamstime).
• 7. Maintain an upload ledger: check before upload, mark after successful upload to prevent duplicates.
• 8. Package assets into zips with a cover (1280×720) and thumbnail (600×600).
• 9. Convert Markdown guides to styled PDFs using ReportLab in a subprocess.
• 10. Keep dependency list minimal (Pillow, ReportLab) and use only the standard library for other tasks.

证据

I killed my pipeline with a single &.

来源

The whole pipeline that produced 39 finished products, 856 MB of output, ran on 1 vCPU and 2 GB of RAM.

来源

One process at a time. Never parallelize image jobs on a single core.

来源

Render at the resolution you need. I used to render everything at 300 dpi just to inspect it.

来源

The fastest way to lose a day is a rejected upload batch. All three major channels validate the CSV header before they look at a single file.

来源

The fix is boring and it works: a ledger. Before each upload, check the ledger. After each upload, mark it done.

来源