I Rotated 4 Secrets in 6 Days. 3 of Them Broke Production.

说明与用途

作者说明了不完整的库存导致秘密轮换时的隐藏故障，并介绍了如何添加验证步骤来防止未来的停机。

问题

Secret rotation caused silent failures across multiple systems because inventory of credential usage was incomplete and stale copies existed.

结果

By adding inventory checks and verification steps, the author prevented silent outages and ensured all consumers were updated before deleting old secrets.

工作流

• 1. Run inventory query to list all credential references in n8n workflows.
• 2. Identify stale or missing credential IDs and high-impact credentials (e.g., Postgres).
• 3. For each credential to rotate, perform pre-rotation verification: confirm current value works for all consumers.
• 4. Rotate the secret (update credential, environment variables, nginx snippets, etc.).
• 5. Immediately verify new value by triggering a test request to each consumer endpoint.
• 6. Monitor for failures (e.g., 401, 403, 404) and trigger alerts if any occur.
• 7. Once all consumers confirm success, delete the old credential value.
• 8. Log rotation details and update documentation.

证据

The single most-used credential (a Postgres connection) sits in 233 nodes in 80 workflows, 40 of them active.

来源

Three failures were silent: inbound messages dropped, calls rejected, media sync stopped.

来源