Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

226 Scenarios, 43 Detectors, 1 Clean Run: Building a Test Harness That Can't Lie

说明与用途

构建了226个手工场景的测试平台,测试43个检测器,无需Docker或Postgres,使用真实LLM调用发现隐藏错误。

问题

Detectors that do nothing can be indistinguishable from those that crash, making it hard to trust test results.

结果

The harness executes 226 scenarios in ~12 s, catching 6/9 floating‑point bugs, off‑by‑one errors, silent skips, and logic bugs that would otherwise go unnoticed.

工作流

  • Define a Scenario dataclass with id, detector, expected fire, build function, severity, pool, and phase.
  • Create hand‑built traces that trigger a single property per scenario.
  • Implement ScriptedPool to fake a Postgres pool with a dict, avoiding Docker or asyncpg.
  • Use _has_valid_pool() heuristic to distinguish real pools from mocks.
  • Run each scenario, capturing client stats before and after to verify real LLM calls.
  • Check six conditions per detector: fire status, severity, detector identity, explanation, evidence, and model call.
  • For LLM detectors, parse the model’s JSON verdict and compare to expected hallucination flag.
  • Split scenarios into four phases (matrix, precision, llm, clean) to target specific bug classes.
  • Execute all scenarios and report failures.

证据

The trick is in how the detectors decide whether they have a real pool.

来源

Six identical calls, threshold of five, should fire a warning.

来源

The twelve LLM scenarios hit a real model.

来源

  • 记录日期: 2026-10-19T06:41:30.499Z