pip install pillow reportlab And Nothing Else: What a 2 GB Server Taught Me About Dependency Bloat

说明与用途

在2 GB RAM、1 vCPU服务器上，OOM错误是由于47个传递性依赖导致的。通过删除不必要的包，仅保留Pillow和reportlab，并采用单进程执行、gc.collect()和子进程隔离，管道现在能够可靠地处理1,700个资产而不出现内存耗尽。

问题

An image rendering pipeline on a 1 vCPU, 2 GB RAM server crashed with OOM due to excessive transitive dependencies in a “simple” automation stack.

结果

The refactored pipeline runs on a 1 vCPU, 2 GB RAM server without OOM errors, processes 1,700 assets, and produces 856 MB of output using only four scripts and two third‑party packages.

工作流

• Measure peak RSS with resource.getrusage(RUSAGE_SELF).ru_maxrss after each batch
• Delete unnecessary dependencies and rewrite scripts to use only stdlib
• Limit image jobs to one process at a time, call gc.collect() between items
• Run heavy render steps in separate subprocesses to return memory to OS
• Inspect previews at 60 dpi instead of 300 dpi
• Create one script per concern: make_metadata.py, upload_tracker.py, pack_product.py, md2pdf.py
• Use a ledger file to track upload status and prevent double‑posting
• Adopt a fixed folder layout (assets/, output/, packs/, research/, scripts/)
• Package the four scripts and layout into a Pipeline Starter Kit

证据

I hit the wall at 2:40 on a Tuesday afternoon. My server, a 1 vCPU box with 2 GB of RAM and no GPU, had just killed an image render with an OOM error.

来源

The task queue alone sat at a few hundred megabytes before it did any work.

来源

The fix was not optimization. It was deletion. My metadata generation and upload tracking now run on the standard library alone.

来源

The scripts that do need help, image covers and PDF rendering, import exactly two packages: Pillow and reportlab.

来源

On a 1-core box, I also stopped fanning out image jobs in parallel. One process at a time, gc.collect() between items, and heavy render steps in their own subprocess so memory returns to the OS.

来源