pip install pillow reportlab And Nothing Else: What a 2 GB Server Taught Me About Dependency Bloat

Описание и применение

На сервере 2 ГБ RAM и 1 vCPU OOM‑ошибка из‑за 47 зависимостей. Удалив лишнее и оставив только Pillow и reportlab, а также применив однопоточность, gc.collect() и subprocess‑изоляцию, удалось стабильно обрабатывать 1 700 активов без переполнения памяти.

Задача

An image rendering pipeline on a 1 vCPU, 2 GB RAM server crashed with OOM due to excessive transitive dependencies in a “simple” automation stack.

Результат

The refactored pipeline runs on a 1 vCPU, 2 GB RAM server without OOM errors, processes 1,700 assets, and produces 856 MB of output using only four scripts and two third‑party packages.

Рабочий процесс

• Measure peak RSS with resource.getrusage(RUSAGE_SELF).ru_maxrss after each batch
• Delete unnecessary dependencies and rewrite scripts to use only stdlib
• Limit image jobs to one process at a time, call gc.collect() between items
• Run heavy render steps in separate subprocesses to return memory to OS
• Inspect previews at 60 dpi instead of 300 dpi
• Create one script per concern: make_metadata.py, upload_tracker.py, pack_product.py, md2pdf.py
• Use a ledger file to track upload status and prevent double‑posting
• Adopt a fixed folder layout (assets/, output/, packs/, research/, scripts/)
• Package the four scripts and layout into a Pipeline Starter Kit

Доказательства

I hit the wall at 2:40 on a Tuesday afternoon. My server, a 1 vCPU box with 2 GB of RAM and no GPU, had just killed an image render with an OOM error.

Источник

The task queue alone sat at a few hundred megabytes before it did any work.

Источник

The fix was not optimization. It was deletion. My metadata generation and upload tracking now run on the standard library alone.

Источник

The scripts that do need help, image covers and PDF rendering, import exactly two packages: Pillow and reportlab.

Источник

On a 1-core box, I also stopped fanning out image jobs in parallel. One process at a time, gc.collect() between items, and heavy render steps in their own subprocess so memory returns to the OS.

Источник