My refund agent looked smart until it lied about Shopify, Stripe, and Zendesk 3 different ways

Description and use

An LLM support agent falsely reported refunds as complete after calling Shopify, Stripe, and Zendesk APIs. Adding transaction status checks and using Stripe idempotency keys fixed the issue.

Problem

An LLM‑powered support agent incorrectly reported refunds as complete after making API calls to Shopify, Stripe, and Zendesk, leading to customer misinformation and operational failures.

Outcome

The agent now accurately reports refund status, reducing customer confusion and preventing financial discrepancies.

Workflow

• Read Zendesk ticket
• Find Shopify order
• Call Shopify refundCreate
• Verify refund transaction status via getOrderTransactions
• If success, proceed; else handle failure
• Generate idempotency key
• Call Stripe refunds.create with key
• Persist key before call
• Handle retries based on persisted key
• Update Zendesk ticket with verified status
• Reply to customer

Evidence

Shopify’s docs are explicit: the existence of a Refund object does not guarantee the financial transaction completed.

Source

Stripe stores the first result for a given idempotency key and returns the same status code and body on retries, including 500 errors.

Source