Our Service Desk and a Customer's Spent a Weekend Opening Tickets for Each Other

Описание и применение

Автоматические подтверждения двух систем поддержки создали бесконечный цикл, породив более 11 000 дублирующих заявок. После внедрения проверки заголовков, ограничения количества подтверждений и перенаправления сообщений известных клиентов цикл остановлен.

Задача

Automatic acknowledgements from two service desk systems triggered an infinite loop, creating over 11,000 duplicate tickets on each side.

Результат

The loop created 11,400 tickets on each side; after implementing header checks, rate limiting, and queue adjustments, the system now prevents similar loops and controls ticket volume.

Рабочий процесс

• 1. Customer’s portal notifications sent from customer’s service desk mailbox.
• 2. Company’s service desk received notification and created a ticket, sending an acknowledgement.
• 3. Customer’s service desk treated acknowledgement as a new request and created a ticket, sending another acknowledgement.
• 4. Company’s service desk repeated the process, creating a loop of tickets.
• 5. After the incident, company’s service desk added automatic message header detection and rate limiting.
• 6. Configured queues to ignore acknowledgements from known customer/supplier mailboxes.
• 7. Implemented surge handling to alert on‑call engineer after ten minutes of high volume.

Доказательства

Our service desk tool sends an automatic acknowledgement to every email it receives, with a ticket number and a line saying we will be in touch.

Источник

By Monday morning our queue held eleven thousand four hundred tickets and theirs held about the same.

Источник

Our desk now marks every automatic message with the standard header that tells other systems it was generated automatically, and it no longer acknowledges incoming mail that carries the same header or its common equivalents.

Источник

No sender receives more than five acknowledgements in an hour. A sixth message from the same address in an hour opens one ticket for a person to look at, with the rest attached to it.

Источник