Temperature Zero Is Not a Reproducibility Guarantee: Five Agent Myths, Tested

Description and use

Temperature 0 does not guarantee reproducibility: five agent myths tested in practice.

Problem

Developers assume agent runs are deterministic when temperature is set to zero, local tests pass, retries fix failures, warm caches guarantee reproducibility, and free compute incurs no cost.

Outcome

Implementing systematic variance checks, environment fingerprinting, flake rate analysis, lockfile enforcement, and cost tracking transforms agent pipelines from flaky, opaque processes into measurable, reproducible workflows.

Workflow

• 1. Run the same prompt multiple times (e.g., 20) and hash outputs to detect variance.
• 2. Fingerprint local and runner environments and diff to find discrepancies.
• 3. Execute the same job 20 times to calculate flake rate and identify causes.
• 4. Verify that dependency lockfiles are unchanged after installs.
• 5. Track agent minutes, queued minutes, and retry multiplier from run logs.

Evidence

Setting temperature=0 reduces sampling randomness. It does not remove run-to-run variance.

Source

Local green proves your laptop works. It says nothing about the machine that will run your code.

Source

Retries hide flakiness. They do not repair it, and they quietly raise your cost per merged change.

Source

The cache key is not the build. The lockfile is.

Source

Every free tier has a budget. It is usually wall-clock time and concurrency instead of dollars.

Source