Cents Matter: Does a Model Notice When One Financial Fact Changes?

Описание и применение

Модели LLM демонстрируют разную точность при точном расчёте денежных сумм в синтетическом наборе данных, выявляя ошибки округления, локали и идентификации событий.

Задача

Models often misinterpret or miscalculate small monetary differences, leading to incorrect warnings or corrections in financial data processing.

Результат

Gemini and gpt-oss-20b achieved near‑perfect individual case accuracy, while Claude Haiku struggled. Pair accuracy exposed that Haiku often failed to adjust answers when a single fact changed. Evidence sufficiency tests showed all models abstained correctly on underspecified cases, but Haiku also abstained incorrectly on a determinate case.

Рабочий процесс

• 1. Construct 40 synthetic cases with 20 minimal pairs varying one fact each.
• 2. Define expected outputs: status, expected cents, discrepancy cents.
• 3. Run each model (Gemini 3.7 Flash, gpt-oss-20b, Claude Haiku 4.5) via Kaggle Benchmarks SDK with fixed temperature and seed.
• 4. Parse structured JSON responses and compare against reference using deterministic scorer.
• 5. Aggregate case, pair, and evidence sufficiency scores.
• 6. Analyze failure patterns and document insights.

Доказательства

Gemini 3.7 Flash 39/40 (97.5%)

Источник

All three completed the same 40 cases on October 7, 2026, with zero execution errors.

Источник

The benchmark checks the proposed amount and discrepancy, not just the warning label.

Источник