I Rotated 4 Secrets in 6 Days. 3 of Them Broke Production.

Description and use

The author explains how incomplete inventory caused silent failures during secret rotation and how they added verification steps to prevent future outages.

Problem

Secret rotation caused silent failures across multiple systems because inventory of credential usage was incomplete and stale copies existed.

Outcome

By adding inventory checks and verification steps, the author prevented silent outages and ensured all consumers were updated before deleting old secrets.

Workflow

• 1. Run inventory query to list all credential references in n8n workflows.
• 2. Identify stale or missing credential IDs and high-impact credentials (e.g., Postgres).
• 3. For each credential to rotate, perform pre-rotation verification: confirm current value works for all consumers.
• 4. Rotate the secret (update credential, environment variables, nginx snippets, etc.).
• 5. Immediately verify new value by triggering a test request to each consumer endpoint.
• 6. Monitor for failures (e.g., 401, 403, 404) and trigger alerts if any occur.
• 7. Once all consumers confirm success, delete the old credential value.
• 8. Log rotation details and update documentation.

Evidence

The single most-used credential (a Postgres connection) sits in 233 nodes in 80 workflows, 40 of them active.

Source

Three failures were silent: inbound messages dropped, calls rejected, media sync stopped.

Source