Our Service Desk and a Customer's Spent a Weekend Opening Tickets for Each Other

说明与用途

两个支持系统的自动确认导致无限循环，产生超过11,000个重复工单。通过检查头信息、限制确认次数和路由已知客户邮件，已停止循环。

问题

Automatic acknowledgements from two service desk systems triggered an infinite loop, creating over 11,000 duplicate tickets on each side.

结果

The loop created 11,400 tickets on each side; after implementing header checks, rate limiting, and queue adjustments, the system now prevents similar loops and controls ticket volume.

工作流

• 1. Customer’s portal notifications sent from customer’s service desk mailbox.
• 2. Company’s service desk received notification and created a ticket, sending an acknowledgement.
• 3. Customer’s service desk treated acknowledgement as a new request and created a ticket, sending another acknowledgement.
• 4. Company’s service desk repeated the process, creating a loop of tickets.
• 5. After the incident, company’s service desk added automatic message header detection and rate limiting.
• 6. Configured queues to ignore acknowledgements from known customer/supplier mailboxes.
• 7. Implemented surge handling to alert on‑call engineer after ten minutes of high volume.

证据

Our service desk tool sends an automatic acknowledgement to every email it receives, with a ticket number and a line saying we will be in touch.

来源

By Monday morning our queue held eleven thousand four hundred tickets and theirs held about the same.

来源

Our desk now marks every automatic message with the standard header that tells other systems it was generated automatically, and it no longer acknowledges incoming mail that carries the same header or its common equivalents.

来源

No sender receives more than five acknowledgements in an hour. A sixth message from the same address in an hour opens one ticket for a person to look at, with the rest attached to it.

来源