Requirements.txt as Discipline: Stdlib-Only Scripts Need No Install

Описание и применение

Разделение пайплайна на небольшие скрипты, использующие только стандартную библиотеку Python, позволило избежать двойной публикации и запустить весь процесс на сервере с 2 ГБ ОЗУ без установки дополнительных пакетов.

Задача

Running a batch twice caused a double‑post flag on a stock platform, despite no crashes or errors.

Результат

The refactored pipeline produced 39 finished products (856 MB) on a single‑core, 2 GB server without installing any packages, and eliminated double‑post flags.

Рабочий процесс

• Split the original pipeline into four scripts: make_metadata.py, upload_tracker.py, md2pdf.py, pack_product.py.
• Make make_metadata.py and upload_tracker.py depend only on the standard library.
• Use upload_tracker.py to record posted assets and prevent re‑runs from double‑posting.
• Run make_metadata.py with arguments like "python make_metadata.py ./my-assets --platform adobe --out metadata.csv --ext .jpg" to generate platform‑specific CSV headers.
• Use md2pdf.py and pack_product.py for rendering, installing only Pillow and reportlab as needed.
• Follow resource limits: single process, gc.collect() after each asset, measure peak RSS, run heavy renders in separate subprocesses, inspect previews at 60 dpi.
• Maintain a fixed folder layout for assets, output, packs, research, and scripts.
• Keep the research folder permanently for future product creation.
• Deploy the scripts on any machine with Python 3.9+ without creating a virtual environment or installing dependencies.

Доказательства

The second time I ran my upload batch, I got the email I had been dreading. A channel had flagged my account for double-posting.

Источник

I split my pipeline into small standalone scripts, I made two of them depend on nothing but the Python standard library, and I wrote a tracker so a re‑run could never double-post again.

Источник

That split is the whole point. A CSV generator and a ledger are pure text and logic. They have no business pulling in an image library, and when they do not, you can run them anywhere Python exists.

Источник

On a 1 vCPU, 2 GB box with no GPU, the pipeline only works if you respect those limits: One process at a time. Never fan out image jobs in parallel on a single-core box.

Источник