10 Tasks in Parallel: Killing the Queue Wait with orchestrate-codex-worker.sh

Описание и применение

Параллельный запуск 10 задач с Codex уменьшает общее время выполнения с 15 минут до ~90 секунд, обеспечивая реальное время статуса и упрощая обнаружение ошибок.

Задача

Sequential execution of Codex tasks causes long idle CPU time and opaque failure visibility, delaying batch processing and making debugging hard.

Результат

Parallel worker design reduces total batch time from ~15 minutes to ~90 seconds, eliminates idle CPU, provides real‑time status via status files, and simplifies failure detection and aggregation.

Рабочий процесс

• Launch launchd job at scheduled times (08:00, 10:35).
• Run claude‑quota‑guard.py to ensure API quota.
• Invoke run‑and‑notify.sh to send start notification.
• Parent script article‑daily‑stock.sh starts N orchestrate‑codex‑worker.sh processes in background, each with task, handoff, and status files.
• Each worker writes "running" to its status file, builds prompt, calls codex exec with flags (-p yolo, -m gpt‑5.4, --color never, -C "$(pwd)", -o output_file).
• On success, worker writes structured handoff file (Summary, Files Changed, Validation, Remaining Risks) and updates status to "completed". On failure, writes failure summary and status "failed".
• Parent script waits for all workers, aggregates handoff files, generates report, and sends completion notification.

Доказательства

Run 10 tasks in parallel and total elapsed time converges to "the slowest worker's processing time" + "startup overhead."

Источник

Each worker keeps writing its own state to a status-file, the parent can read every job's progress even before it joins with wait.

Источник