Free AI Compute for SQL Review: A Structured Debate and a Decision Rule

Description and use

An automated recipe for weekly blind calibration of AI‑generated SQL reviewers using MonkeyCode’s free tier and a PostgreSQL sandbox.

Problem

Teams lack a systematic way to measure how accurately humans review AI‑generated SQL, leading to costly mistakes that go unnoticed.

Outcome

Teams can run weekly blind calibration tests for ~40k tokens, covering ~250 rounds on the free tier, and obtain actionable accuracy metrics that inform review practices and gate decisions.

Workflow

• Create sandbox schema with python calibrate_reviewer.py init
• Define or extend test cases with ground‑truth verdicts
• Run each case through EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) and wrap writes in a transaction that rolls back
• Emit review_pack.json containing plans but no verdicts
• Have reviewers classify each query as correct, slow, or dangerous in a separate file
• Score the verdicts with python calibrate_reviewer.py score review_verdicts.json and review the accuracy report

Evidence

A blind calibration harness costs almost nothing when tokens and a sandbox server are free, so the measurement can run weekly instead of annually.

Source

The harness below is a condensed version of the one I run for review calibration. It targets Postgres 14 or newer, works on the free server option or any local sandbox, and never executes a write outside a rolled‑back transaction.

Source