Idempotent Video Requests: Prevent Duplicate Generation During Retries in Production

Description and use

Implementing a durable idempotency record for video generation requests prevents duplicate jobs on retries, saving bandwidth and simplifying audit.

Problem

When a client retries a video generation request after a timeout, the same job may be submitted multiple times, causing duplicate video generation, wasted bandwidth, and extra cleanup work.

Outcome

Implementing a durable idempotency record per request key eliminates duplicate video generations during retries, reduces bandwidth and cleanup costs, and provides clear lineage and terminal state persistence.

Workflow

• 1. Client generates a stable request_key and source_asset_id.
• 2. Worker calls start_or_resume: retrieves or creates a VideoRequest record keyed by request_key.
• 3. If generation_id is null, worker submits job to provider and attaches the returned generation_id to the record (conditional write).
• 4. Subsequent retries read the record; if generation_id exists, they skip submission and wait for the job to finish.
• 5. Worker polls provider using generation_id; upon terminal state (ready/failed) updates record state.
• 6. All stages validate presence of source asset, generation_id, and terminal status before proceeding.
• 7. Record lineage with timestamps and byte counts for audit.
• 8. Handle provider rate limits by honoring Retry-After and keeping the same request_key.
• 9. Ensure finite polling and persist terminal states for cleanup jobs.

Evidence

The experiment is simple: submit one logical request, interrupt the client after submission, replay the same request five times, and verify that the ledger contains one generation ID and one source-to-derivative lineage.

Source

A missing source asset, an unknown generation ID, or a non-terminal status should stop the pipeline. It should not create a second video.

Source