I Rotated 4 Secrets in 6 Days. 3 of Them Broke Production.

Описание и применение

Автор описал, как некомплексный инвентаризационный процесс привёл к скрытым сбоям при смене секретов и как внедрил проверки и аудит, чтобы избежать подобных проблем.

Задача

Secret rotation caused silent failures across multiple systems because inventory of credential usage was incomplete and stale copies existed.

Результат

By adding inventory checks and verification steps, the author prevented silent outages and ensured all consumers were updated before deleting old secrets.

Рабочий процесс

• 1. Run inventory query to list all credential references in n8n workflows.
• 2. Identify stale or missing credential IDs and high-impact credentials (e.g., Postgres).
• 3. For each credential to rotate, perform pre-rotation verification: confirm current value works for all consumers.
• 4. Rotate the secret (update credential, environment variables, nginx snippets, etc.).
• 5. Immediately verify new value by triggering a test request to each consumer endpoint.
• 6. Monitor for failures (e.g., 401, 403, 404) and trigger alerts if any occur.
• 7. Once all consumers confirm success, delete the old credential value.
• 8. Log rotation details and update documentation.

Доказательства

The single most-used credential (a Postgres connection) sits in 233 nodes in 80 workflows, 40 of them active.

Источник

Three failures were silent: inbound messages dropped, calls rejected, media sync stopped.

Источник