Temperature Zero Is Not a Reproducibility Guarantee: Five Agent Myths, Tested

说明与用途

温度0并不保证可复现性：五个代理神话的实践检验。

问题

Developers assume agent runs are deterministic when temperature is set to zero, local tests pass, retries fix failures, warm caches guarantee reproducibility, and free compute incurs no cost.

结果

Implementing systematic variance checks, environment fingerprinting, flake rate analysis, lockfile enforcement, and cost tracking transforms agent pipelines from flaky, opaque processes into measurable, reproducible workflows.

工作流

• 1. Run the same prompt multiple times (e.g., 20) and hash outputs to detect variance.
• 2. Fingerprint local and runner environments and diff to find discrepancies.
• 3. Execute the same job 20 times to calculate flake rate and identify causes.
• 4. Verify that dependency lockfiles are unchanged after installs.
• 5. Track agent minutes, queued minutes, and retry multiplier from run logs.

证据

Setting temperature=0 reduces sampling randomness. It does not remove run-to-run variance.

来源

Local green proves your laptop works. It says nothing about the machine that will run your code.

来源

Retries hide flakiness. They do not repair it, and they quietly raise your cost per merged change.

来源

The cache key is not the build. The lockfile is.

来源

Every free tier has a budget. It is usually wall-clock time and concurrency instead of dollars.

来源