A Free‑Tier Trace Loop for Agent Run Regression

Описание и применение

Автоматический цикл сравнения трассировок агентских запусков с бесплатными ресурсами MonkeyCode, позволяющий выявлять регрессии без затрат.

Задача

Agent runs are stochastic; fixing one bug often reveals another. Detecting regressions requires comparing full trace logs, which traditionally needs paid server and model resources.

Результат

Implemented a zero‑cost trace comparison loop that automatically detects regressions in agent behavior using free MonkeyCode resources.

Рабочий процесс

• 1. Configure MonkeyCode API key and base URL via environment variables.
• 2. Run FastAPI service that stores each agent run’s events in SQLite.
• 3. After an agent run finishes, POST the list of tool calls, inputs, and outputs to /traces endpoint.
• 4. To compare two runs, send a GET request to /compare?run1=<run_id1>&run2=<run_id2>.
• 5. The service calls MonkeyCode’s model to produce a bullet‑point diff and verdict.
• 6. Review the diff to identify behavioral changes such as wrong tool usage or argument order.

Доказательства

MonkeyCode is an open‑source project that offers two things relevant to this workflow: free model access and a free server option.

Источник

The loop I use is simple: record every tool call and input/output pair from an agent run into a structured store; serialize each run into a single document; then ask a model to diff two documents and highlight behavioral changes.

Источник