226 Scenarios, 43 Detectors, 1 Clean Run: Building a Test Harness That Can't Lie
说明与用途
构建了226个手工场景的测试平台,测试43个检测器,无需Docker或Postgres,使用真实LLM调用发现隐藏错误。
问题
Detectors that do nothing can be indistinguishable from those that crash, making it hard to trust test results.
结果
The harness executes 226 scenarios in ~12 s, catching 6/9 floating‑point bugs, off‑by‑one errors, silent skips, and logic bugs that would otherwise go unnoticed.
工作流
- Define a Scenario dataclass with id, detector, expected fire, build function, severity, pool, and phase.
- Create hand‑built traces that trigger a single property per scenario.
- Implement ScriptedPool to fake a Postgres pool with a dict, avoiding Docker or asyncpg.
- Use _has_valid_pool() heuristic to distinguish real pools from mocks.
- Run each scenario, capturing client stats before and after to verify real LLM calls.
- Check six conditions per detector: fire status, severity, detector identity, explanation, evidence, and model call.
- For LLM detectors, parse the model’s JSON verdict and compare to expected hallucination flag.
- Split scenarios into four phases (matrix, precision, llm, clean) to target specific bug classes.
- Execute all scenarios and report failures.
证据
The trick is in how the detectors decide whether they have a real pool.
Six identical calls, threshold of five, should fire a warning.
The twelve LLM scenarios hit a real model.
- 记录日期: 2026-10-19T06:41:30.499Z