Detecting False Zeros in Coding Agent Probes via Control Tests

Описание и применение

Исследование показывает, что кодовые агенты и пробы часто возвращают «ноль» (отсутствие данных) при сбоях подключения или неверных предположениях о схеме данных, а не при реальном отсутствии данных. Это создает «ложный ноль», который маскирует ошибки. Решение — использование положительных и отрицательных контролей для проверки работоспособности инструментов перед доверием их результатам.

Задача

Coding agents and monitoring probes often return 'zero' (indicating no issues or missing data) when they fail to connect to the correct data source or schema, rather than when the data is genuinely absent. This 'false zero' creates a blind spot where instruments report success despite being broken or misconfigured.

Результат

Four distinct probe failures (encoding, schema, type, vocabulary) all returned 'zero', mimicking a healthy 'not found' state. Applying positive and negative controls revealed the probes were broken or misaligned, not that the data was absent.

Рабочий процесс

• Identify probes or agent outputs that return 'zero' or 'null' as a primary signal of health or absence.
• Define a 'positive control': a known instance of the target data that the probe should definitely find.
• Define a 'negative control': a known instance of non-target data that the probe should definitely ignore.
• Run the probe against the positive control to verify it fires correctly.
• Run the probe against the negative control to verify it discriminates correctly.
• Only trust the 'zero' result from the actual production data if both controls passed.
• If controls fail, debug the probe's connection, schema, or logic before trusting its output.

Доказательства

A missing fact produces wrong work rather than absent work

Источник

instruments built on reads look for a hole already filled

Источник

probes returning a FALSE ZERO (naive==0 while correct>0): 4