Cents Matter: Does a Model Notice When One Financial Fact Changes?

Description and use

LLM models show varying accuracy in precise monetary calculations within a synthetic dataset, revealing errors in rounding, locale handling, and event identification.

Problem

Models often misinterpret or miscalculate small monetary differences, leading to incorrect warnings or corrections in financial data processing.

Outcome

Gemini and gpt-oss-20b achieved near‑perfect individual case accuracy, while Claude Haiku struggled. Pair accuracy exposed that Haiku often failed to adjust answers when a single fact changed. Evidence sufficiency tests showed all models abstained correctly on underspecified cases, but Haiku also abstained incorrectly on a determinate case.

Workflow

• 1. Construct 40 synthetic cases with 20 minimal pairs varying one fact each.
• 2. Define expected outputs: status, expected cents, discrepancy cents.
• 3. Run each model (Gemini 3.7 Flash, gpt-oss-20b, Claude Haiku 4.5) via Kaggle Benchmarks SDK with fixed temperature and seed.
• 4. Parse structured JSON responses and compare against reference using deterministic scorer.
• 5. Aggregate case, pair, and evidence sufficiency scores.
• 6. Analyze failure patterns and document insights.

Evidence

Gemini 3.7 Flash 39/40 (97.5%)

Source

All three completed the same 40 cases on October 7, 2026, with zero execution errors.

Source

The benchmark checks the proposed amount and discrepancy, not just the warning label.

Source