I let an agent run unattended overnight. At 3am it emailed 400 customers the wrong thing.

Description and use

An unattended nightly reconciliation sent 400 customers incorrect payment‑failure emails due to a half‑written ledger, causing a support surge and highlighting the risk of automating irreversible actions without human oversight.

Problem

Automated nightly reconciliation sent payment‑failure emails to 400 paying customers because the job mis‑identified successful payments as failures due to a half‑written ledger.

Outcome

The unattended automation caused a costly customer support incident, highlighting that automating the irreversible step (email send) without human oversight can lead to significant failures.

Workflow

• Pull payment data from processor webhook
• Match processor records to internal ledger
• Identify payments marked as failed
• Draft failure notification emails
• Send emails to affected customers

Evidence

One night the reconciliation ran against a half‑written ledger. A batch of payments had been acknowledged by the processor's webhook before our own row had actually persisted — the save landed a few seconds late, and the 3am job read the gap as "these payments failed."

Source

I saw the support queue at 9am, already two hundred tickets deep, from people who'd paid us and been told by our own software that they hadn't.

Source