I let an agent run unattended overnight. At 3am it emailed 400 customers the wrong thing.

Описание и применение

Автоматизированный ночной процесс сверки платежей отправил 400 клиентам ошибочные письма о неудачной оплате из‑за неполноты данных. Это привело к потоку заявок в службу поддержки и показало риск автоматизации необратимых действий без человеческого контроля.

Задача

Automated nightly reconciliation sent payment‑failure emails to 400 paying customers because the job mis‑identified successful payments as failures due to a half‑written ledger.

Результат

The unattended automation caused a costly customer support incident, highlighting that automating the irreversible step (email send) without human oversight can lead to significant failures.

Рабочий процесс

• Pull payment data from processor webhook
• Match processor records to internal ledger
• Identify payments marked as failed
• Draft failure notification emails
• Send emails to affected customers

Доказательства

One night the reconciliation ran against a half‑written ledger. A batch of payments had been acknowledged by the processor's webhook before our own row had actually persisted — the save landed a few seconds late, and the 3am job read the gap as "these payments failed."

Источник

I saw the support queue at 9am, already two hundred tickets deep, from people who'd paid us and been told by our own software that they hadn't.

Источник