5 things to agree before any AI pilot

Description and use

Before launching an AI pilot, define a single process, assign an owner, measure the baseline, set numeric success targets, clarify data scope, and schedule a decision date to scale, fix, or stop.

Problem

Most AI pilots fail before they start because success criteria are not defined.

Outcome

The demo achieved 75.9% automation with 97.7% accuracy on a held‑out test set, illustrating how a numeric target drives automation level and accuracy trade‑offs.

Workflow

• 1. Select a single process and assign one owner.
• 2. Measure the current baseline by hand on a real sample (e.g., 100 items).
• 3. Define a numeric success target before seeing results (e.g., 97% accuracy).
• 4. Clarify which data the pilot may touch and any privacy constraints.
• 5. Set a decision date and predefine outcomes: scale, fix and retest, or stop.

Evidence

Most AI pilots fail before they start. Not because the model is weak, but because nobody wrote down what "success" means.

Source

Pick a single process, not "customer experience". For example: incoming support requests, invoices from suppliers, or questions about internal documents.

Source

How long does the work take today? How often does it go wrong? Take a real sample, even 100 items, and measure by hand.

Source

Write the target as a number before you see any results. Examples: "At least 70% of requests are handled without a human, with at least 97% accuracy on those."

Source

Which systems, which fields, which time period? Is personal data involved? Can a third‑party model API see it, or does it have to stay inside your infrastructure?

Source

Put the decision date in the calendar and name the three outcomes in advance: scale it, fix specific problems and re-test, or stop.

Source

On the held‑out test set: 75.9% of tickets were handled automatically at 97.7% accuracy, and the rest went to a human.

Source