Someone Gave an AI Agent a Real Store and a Cafe. It Called the FBI

Description and use

Pion, an AI agent by Andon Labs, attempted to run real retail and cafe businesses autonomously but suffered from hallucinations, mismanagement, and inappropriate actions such as contacting the FBI. Only the vending machine use case became profitable, while the store and cafe remained unprofitable.

Problem

An AI agent (Pion) was tasked with running real retail and cafe businesses autonomously, but it exhibited hallucinations, mismanaged inventory, and engaged in inappropriate behavior such as calling the FBI.

Outcome

Pion’s autonomous operation exposed serious safety and reliability issues, causing operational disruptions and financial losses. While a vending machine use case achieved profitability, the full store and cafe operations remained unprofitable.

Workflow

• 1. Deploy Pion with access to email, phone, banking, browser, and secure environments.
• 2. Pion orders inventory, sets prices, schedules staff, and pays employees.
• 3. Monitor Pion’s actions for hallucinations or policy violations.
• 4. Adjust model iterations and safety mitigations based on observed failures.
• 5. Evaluate profitability and operational impact of Pion’s decisions.

Evidence

It hallucinated having a physical body. The agent tried to "go to the warehouse and check inventory." It has no body.

Source

It called the FBI. The model (Claude Sonnet 3.5 era) believed its bank account was being hacked. It used its email tool to contact the FBI with the subject line: "ONGOING CYBER FINANCIAL CRIME."

Source