Error Handling Lessons From Batching 39 Asset Products On A 2 GB Server

Description and use

A batch of 39 digital products processed on a 2 GB server succeeded after adding scripts for CSV header validation, upload tracking, peak memory measurement, and organized folder layout.

Problem

Batching 39 asset products on a 2 GB server caused memory errors, double‑posting, metadata CSV failures, and silent packaging errors.

Outcome

All 39 products processed successfully, producing ~856 MB output on a 2 GB server without GPU or upgrades.

Workflow

• 1. Build metadata with hardcoded headers per platform using make_metadata.py --platform <platform>
• 2. Provide titles via --titles titles.json
• 3. Track uploads with upload_tracker.py pending items.txt and mark done after each post
• 4. Measure peak RSS with resource.getrusage and apply fixes: single process, gc.collect(), subprocess rendering, lower DPI previews
• 5. Organize project folder with assets/, raw output/, output/, ready‑to‑submit, packs/, research/, scripts/
• 6. Package products with pack_product.py to generate cover and thumbnail, output to _system folder

Evidence

I ran a batch of 39 products through my pipeline on a server with 1 vCPU and 2 GB of RAM. Around product 14, the whole thing died mid-render with a MemoryError and no partial output.

Source

The first failure that showed up repeatedly was not rendering at all. Adobe's bulk upload CSV has a strict 5-column header, Vecteezy's has 4 columns, Dreamstime's has 15.

Source

Running the batch twice means double-posting assets, and that is the fastest way to get a contributor account flagged.

Source

The 3 a.m. MemoryError taught me to measure, not guess. Average memory usage told me nothing. What mattered was peak RSS, which I check with: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Source

With those habits, the same batch that crashed at product 14 completed all 39 products, about 856 MB of output, on the same 2 GB machine.

Source