Free AI Compute for SQL Review: A Structured Debate and a Decision Rule

Описание и применение

Автоматизированный рецепт для еженедельной проверки точности ревьюеров SQL, генерируемого ИИ, с использованием бесплатного уровня MonkeyCode и sandbox‑сервера PostgreSQL.

Задача

Teams lack a systematic way to measure how accurately humans review AI‑generated SQL, leading to costly mistakes that go unnoticed.

Результат

Teams can run weekly blind calibration tests for ~40k tokens, covering ~250 rounds on the free tier, and obtain actionable accuracy metrics that inform review practices and gate decisions.

Рабочий процесс

• Create sandbox schema with python calibrate_reviewer.py init
• Define or extend test cases with ground‑truth verdicts
• Run each case through EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) and wrap writes in a transaction that rolls back
• Emit review_pack.json containing plans but no verdicts
• Have reviewers classify each query as correct, slow, or dangerous in a separate file
• Score the verdicts with python calibrate_reviewer.py score review_verdicts.json and review the accuracy report

Доказательства

A blind calibration harness costs almost nothing when tokens and a sandbox server are free, so the measurement can run weekly instead of annually.

Источник

The harness below is a condensed version of the one I run for review calibration. It targets Postgres 14 or newer, works on the free server option or any local sandbox, and never executes a write outside a rolled‑back transaction.

Источник