226 Scenarios, 43 Detectors, 1 Clean Run: Building a Test Harness That Can't Lie

Description and use

A 226‑scenario harness tests 43 detectors without Docker or Postgres, using real LLM calls to uncover hidden bugs.

Problem

Detectors that do nothing can be indistinguishable from those that crash, making it hard to trust test results.

Outcome

The harness executes 226 scenarios in ~12 s, catching 6/9 floating‑point bugs, off‑by‑one errors, silent skips, and logic bugs that would otherwise go unnoticed.

Workflow

• Define a Scenario dataclass with id, detector, expected fire, build function, severity, pool, and phase.
• Create hand‑built traces that trigger a single property per scenario.
• Implement ScriptedPool to fake a Postgres pool with a dict, avoiding Docker or asyncpg.
• Use _has_valid_pool() heuristic to distinguish real pools from mocks.
• Run each scenario, capturing client stats before and after to verify real LLM calls.
• Check six conditions per detector: fire status, severity, detector identity, explanation, evidence, and model call.
• For LLM detectors, parse the model’s JSON verdict and compare to expected hallucination flag.
• Split scenarios into four phases (matrix, precision, llm, clean) to target specific bug classes.
• Execute all scenarios and report failures.

Evidence

The trick is in how the detectors decide whether they have a real pool.

Source

Six identical calls, threshold of five, should fire a warning.

Source

The twelve LLM scenarios hit a real model.

Source