Anthropic Batch API: 112 of My 1,842 Jobs Never Came Back

Description and use

Using Anthropic Batch API for nightly jobs caused a 6.1 % silent failure rate due to mis‑aligned results. After adding per‑request status checks, custom_id mapping, and handling truncated responses, failures dropped to 0 % and cost was cut by 50 %.

Problem

Assuming batch processing status "ended" means all requests succeeded caused silent failures and mis‑aligned results in a production pipeline.

Outcome

Reduced silent failures from 6.1% to 0%, ensured correct report mapping, and cut batch cost by 50% by using batch for nightly scoring jobs.

Workflow

• 1. Create batch with requests, each having a unique custom_id (e.g., session UUID).
• 2. Poll batch until processing_status == "ended".
• 3. Retrieve batch.request_counts to verify counts of succeeded, errored, expired, canceled.
• 4. Stream results JSONL and build a dict mapping custom_id to result entry.
• 5. For each pending session, fetch entry by custom_id; if missing, mark for retry.
• 6. If entry.result.type == "succeeded" and entry.result.message exists, parse JSON; handle stop_reason "max_tokens" by retrying or truncating safely.
• 7. Save report or score; otherwise log error and notify.
• 8. Monitor metrics and alert on high error or expiration rates.

Evidence

batch.processing_status == "ended" does not mean success. Read request_counts and switch on each result line's result.type

Source

Results come back in arbitrary order. Zipping them against your input list silently attaches the wrong output to the wrong record.

Source