226 Scenarios, 43 Detectors, 1 Clean Run: Building a Test Harness That Can't Lie

Описание и применение

Создана система тестирования из 226 сценариев, проверяющая 43 детектора без Docker и Postgres, с реальными вызовами LLM, выявляющая скрытые ошибки.

Задача

Detectors that do nothing can be indistinguishable from those that crash, making it hard to trust test results.

Результат

The harness executes 226 scenarios in ~12 s, catching 6/9 floating‑point bugs, off‑by‑one errors, silent skips, and logic bugs that would otherwise go unnoticed.

Рабочий процесс

• Define a Scenario dataclass with id, detector, expected fire, build function, severity, pool, and phase.
• Create hand‑built traces that trigger a single property per scenario.
• Implement ScriptedPool to fake a Postgres pool with a dict, avoiding Docker or asyncpg.
• Use _has_valid_pool() heuristic to distinguish real pools from mocks.
• Run each scenario, capturing client stats before and after to verify real LLM calls.
• Check six conditions per detector: fire status, severity, detector identity, explanation, evidence, and model call.
• For LLM detectors, parse the model’s JSON verdict and compare to expected hallucination flag.
• Split scenarios into four phases (matrix, precision, llm, clean) to target specific bug classes.
• Execute all scenarios and report failures.

Доказательства

The trick is in how the detectors decide whether they have a real pool.

Источник

Six identical calls, threshold of five, should fire a warning.

Источник

The twelve LLM scenarios hit a real model.

Источник