Running an issue-fixing agent on a $6 Fly Machine: a queue, three budgets, and an honest test report

Description and use

Automating issue resolution in a repository using an LLM agent on Fly.io, without continuous LLM billing or git access, with limits on iterations and cost.

Problem

Automating issue resolution in a repository without continuous LLM billing and without exposing git credentials to the LLM.

Outcome

In 10 consecutive issues on a real TypeScript monorepo, median cost per issue was ~$0.31, worst case $1.85, with typical run ~80 iterations. All 10 PRs were reviewed by a human before merge; fly-swe never merged automatically.

Workflow

• Receive GitHub webhook for a new issue with label bug/docs/chore
• Verify webhook authenticity and deduplicate by delivery ID
• Write event to SQLite queue in WAL mode
• Worker polls queue and starts a task
• Run mini-swe-agent with hard limits: max_iterations, max_cost_per_task_usd, max_daily_cost_usd, and cost_tracking strict
• Agent edits files in working tree and emits completion signal
• Harness performs git add, commit, push to a WIP branch, and opens a PR
• Harness re-runs the test command specified by the agent to verify changes
• If tests fail, harness reports failure on the issue and retries automatically
• If a budget limit is hit, harness force-pushes partial work to a WIP branch and resumes next trigger with a summary of prior attempts

Evidence

The whole thing is a FastAPI webhook receiver and a worker, run together by honcho on a single shared-cpu-1x Fly Machine with a 1 GB volume.

Source

The LLM never touches git. The agent edits files and leaves them in the working tree. Then it emits a completion signal, and everything after that belongs to the harness: git add, commit, push and PR.

Source

Three budgets, enforced in code: max_iterations, max_cost_per_task_usd, max_daily_cost_usd.

Source

After the agent's loop ends, the harness re-runs the test command the agent says it used, in the same environment.

Source

10 consecutive issues (bugs and docs) on a TypeScript monorepo with an SDK and a docs site, using GLM-5.3: median cost per issue: ~$0.31; worst case $1.85 (225 iterations, two verification retries).

Source