A Free‑Tier Trace Loop for Agent Run Regression

Description and use

Automated trace comparison loop for agent runs using free MonkeyCode resources, enabling regression detection without cost.

Problem

Agent runs are stochastic; fixing one bug often reveals another. Detecting regressions requires comparing full trace logs, which traditionally needs paid server and model resources.

Outcome

Implemented a zero‑cost trace comparison loop that automatically detects regressions in agent behavior using free MonkeyCode resources.

Workflow

• 1. Configure MonkeyCode API key and base URL via environment variables.
• 2. Run FastAPI service that stores each agent run’s events in SQLite.
• 3. After an agent run finishes, POST the list of tool calls, inputs, and outputs to /traces endpoint.
• 4. To compare two runs, send a GET request to /compare?run1=<run_id1>&run2=<run_id2>.
• 5. The service calls MonkeyCode’s model to produce a bullet‑point diff and verdict.
• 6. Review the diff to identify behavioral changes such as wrong tool usage or argument order.

Evidence

MonkeyCode is an open‑source project that offers two things relevant to this workflow: free model access and a free server option.

Source

The loop I use is simple: record every tool call and input/output pair from an agent run into a structured store; serialize each run into a single document; then ask a model to diff two documents and highlight behavioral changes.

Source