Cents Matter: Does a Model Notice When One Financial Fact Changes?

说明与用途

LLM模型在合成数据集中的精确货币计算表现出不同的准确性，揭示了舍入、地区设置和事件识别中的错误。

问题

Models often misinterpret or miscalculate small monetary differences, leading to incorrect warnings or corrections in financial data processing.

结果

Gemini and gpt-oss-20b achieved near‑perfect individual case accuracy, while Claude Haiku struggled. Pair accuracy exposed that Haiku often failed to adjust answers when a single fact changed. Evidence sufficiency tests showed all models abstained correctly on underspecified cases, but Haiku also abstained incorrectly on a determinate case.

工作流

• 1. Construct 40 synthetic cases with 20 minimal pairs varying one fact each.
• 2. Define expected outputs: status, expected cents, discrepancy cents.
• 3. Run each model (Gemini 3.7 Flash, gpt-oss-20b, Claude Haiku 4.5) via Kaggle Benchmarks SDK with fixed temperature and seed.
• 4. Parse structured JSON responses and compare against reference using deterministic scorer.
• 5. Aggregate case, pair, and evidence sufficiency scores.
• 6. Analyze failure patterns and document insights.

证据

Gemini 3.7 Flash 39/40 (97.5%)

来源

All three completed the same 40 cases on October 7, 2026, with zero execution errors.

来源

The benchmark checks the proposed amount and discrepancy, not just the warning label.

来源