Error Handling Lessons From Batching 39 Asset Products On A 2 GB Server

Описание и применение

Пакет из 39 цифровых продуктов, обработанный на сервере с 2 ГБ ОЗУ, завершился без ошибок после внедрения скриптов для проверки CSV‑заголовков, отслеживания загрузок, измерения пиковой памяти и структурированного хранения файлов.

Задача

Batching 39 asset products on a 2 GB server caused memory errors, double‑posting, metadata CSV failures, and silent packaging errors.

Результат

All 39 products processed successfully, producing ~856 MB output on a 2 GB server without GPU or upgrades.

Рабочий процесс

• 1. Build metadata with hardcoded headers per platform using make_metadata.py --platform <platform>
• 2. Provide titles via --titles titles.json
• 3. Track uploads with upload_tracker.py pending items.txt and mark done after each post
• 4. Measure peak RSS with resource.getrusage and apply fixes: single process, gc.collect(), subprocess rendering, lower DPI previews
• 5. Organize project folder with assets/, raw output/, output/, ready‑to‑submit, packs/, research/, scripts/
• 6. Package products with pack_product.py to generate cover and thumbnail, output to _system folder

Доказательства

I ran a batch of 39 products through my pipeline on a server with 1 vCPU and 2 GB of RAM. Around product 14, the whole thing died mid-render with a MemoryError and no partial output.

Источник

The first failure that showed up repeatedly was not rendering at all. Adobe's bulk upload CSV has a strict 5-column header, Vecteezy's has 4 columns, Dreamstime's has 15.

Источник

Running the batch twice means double-posting assets, and that is the fastest way to get a contributor account flagged.

Источник

The 3 a.m. MemoryError taught me to measure, not guess. Average memory usage told me nothing. What mattered was peak RSS, which I check with: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Источник

With those habits, the same batch that crashed at product 14 completed all 39 products, about 856 MB of output, on the same 2 GB machine.

Источник