Design Publishing Queues That Survive Image Provider Outages

Description and use

Publishing queues that survive image provider outages keep articles by deferring media processing, bounding retries, and recording provenance, ensuring editorial content is not lost during temporary external failures.

Problem

Publishing pipelines often reject articles when external image providers fail, causing loss of editorial content.

Outcome

Articles no longer get dropped during temporary image provider outages; media stage retries are bounded, provenance is recorded, and deployment is verified, improving reliability.

Workflow

• Validate article editorially
• Check hero image availability
• If image fetch fails and error is transient, defer media stage with retryAfter 1h
• Persist article state as MEDIA_PENDING
• Retry provider queries within bounded limits (e.g., 2 retries, alternate search phrases)
• If a valid asset is found, bind provenance and transform image
• Verify transformed raster (dimensions, hash, MIME)
• Deploy asset and confirm public URL
• If deployment fails, keep article in pending state for retry

Evidence

The retry design must respect Cloudflare Workflow limits and the provider's Pexels API documentation; provider failure does not decide whether the article itself is valid.

Source

A temporary outage has changed editorial history. A better state model is: text VALIDATED MEDIA_PENDING MEDIA_SELECTED READY_TO_PUBLISH PUBLISHED

Source

If an asset selection has already been recorded for article A revision 3, the retry should inspect that record before searching again.

Source