90/10: what we took away from our AI coding agent, and why

Description and use

Splitting responsibilities between scripted steps and an AI agent improved CI/CD pipeline stability and debugging, reducing unpredictable failures.

Problem

Integrating an AI coding agent into a CI/CD pipeline caused unpredictable failures and debugging challenges.

Outcome

Splitting responsibilities reduced failure variability, improved debugging, cut costs, and maintained consistent git workflows.

Workflow

• 1. Developer writes ticket in plain English.
• 2. Ticket moved to Approved; orchestrator provisions stack and copies dev database.
• 3. Ticket moved to Working; agent performs engineering inside the stack.
• 4. Agent posts preview; developer tests and replies in thread.
• 5. Ticket moved to Git Push; orchestrator commits, merges, opens PRs, merges PRs, posts links.
• 6. Ticket moved to VM Update; orchestrator waits for CI, deploys images, runs health checks, rolls back if unhealthy.
• 7. Ticket moved to Completed; orchestrator notifies developer and tears down stack.

Evidence

The first version of our agent pipeline let the coding agent run the git and deploy steps. It didn't fail often, but when it did, it failed in a different way each time, and I couldn't debug that.

Source

Today roughly 90% of the pipeline is plain scripted code, and the agent does one job.

Source

The split we ended up with follows one line. If a step is the same on every ticket, it is a script. The agent keeps the engineering.

Source