Anthropic Batch API: 112 of My 1,842 Jobs Never Came Back

Описание и применение

Использование Batch API Anthropic для ночных задач привело к 6,1 % незаметных ошибок из‑за неверного сопоставления результатов. После внедрения проверки статусов, использования custom_id и обработки обрезанных ответов ошибки исчезли, а стоимость снизилась на 50 %.

Задача

Assuming batch processing status "ended" means all requests succeeded caused silent failures and mis‑aligned results in a production pipeline.

Результат

Reduced silent failures from 6.1% to 0%, ensured correct report mapping, and cut batch cost by 50% by using batch for nightly scoring jobs.

Рабочий процесс

• 1. Create batch with requests, each having a unique custom_id (e.g., session UUID).
• 2. Poll batch until processing_status == "ended".
• 3. Retrieve batch.request_counts to verify counts of succeeded, errored, expired, canceled.
• 4. Stream results JSONL and build a dict mapping custom_id to result entry.
• 5. For each pending session, fetch entry by custom_id; if missing, mark for retry.
• 6. If entry.result.type == "succeeded" and entry.result.message exists, parse JSON; handle stop_reason "max_tokens" by retrying or truncating safely.
• 7. Save report or score; otherwise log error and notify.
• 8. Monitor metrics and alert on high error or expiration rates.

Доказательства

batch.processing_status == "ended" does not mean success. Read request_counts and switch on each result line's result.type

Источник

Results come back in arbitrary order. Zipping them against your input list silently attaches the wrong output to the wrong record.

Источник