The Canary Workflow: A Fake n8n Job That Pages Me Before Clients Notice

Description and use

Problem: client workflows in n8n can fail silently. Solution: an hourly canary workflow that probes the LLM API, Gmail, webhook, and n8n API, alerting via Telegram. Over six months it caught four incidents before clients noticed, preventing complaints.

Problem

Client workflows can fail silently on the platform layer, causing clients to notice outages before the owner does.

Outcome

In six months the canary detected four incidents—revoked OAuth token, model deprecation, queue backup, TLS cert expiry—before any client noticed, preventing any client complaints.

Workflow

• Schedule canary to run hourly
• Probe LLM API with 10‑token prompt and 30s timeout
• Probe Gmail by sending and searching for a unique email
• Probe webhook by POSTing signed payload and checking 200 within 10s
• Probe n8n API for own execution history
• Log results and page via Telegram on failure
• If two probes fail, start 15‑minute re‑check loop

Evidence

The canary tells me first, every time.

Source

The canary runs every hour and produces nothing of value. Its only job is to touch the four things every real workflow in my fleet depends on: LLM API, Gmail round‑trip, Webhook echo, n8n API self‑query.

Source

In six months, four real incidents — and in all four, the canary paged me before any client noticed.

Source

Rule 1: the canary must be cheaper than what it protects.

Source