A client pinged me on a Saturday morning: "your site is down."

Description and use

A client reported site downtime, but the issue was DNS failure undetected by standard uptime checks. Implemented DNS monitoring, a second provider, and a 10‑minute restore runbook to quickly recover, though trust was impacted.

Problem

A client reported site downtime, but the issue was DNS failure, not the application.

Outcome

The client restored DNS service within 10 minutes, preventing further customer impact, but trust was damaged due to delayed communication.

Workflow

• Confirm scope: dig failing names against public resolvers and ISP resolver.
• Decide path: if provider down, switch NS to second provider; if own mistake, restore exported zone to primary.
• Execute runbook: use exact NS strings, export filename, and verification command.
• Verify and communicate: test from mobile data, post status update.

Evidence

The thing that broke sits one layer above every request they serve, so every tool they own said everything was fine while every customer saw a "server not found" page.

Source

The 10‑minute restore path, as a checklist: ...

Source