Model selection is an architecture decision

说明与用途

基于每次成功任务成本的指标选择LLM，最终选定Gemini‑1.5‑Flash作为处理银行对账单的最具成本效益的模型。

问题

Choosing the most cost‑effective LLM for parsing noisy bank statements into clean expense reports, where the key KPI is cost per successful parsing.

结果

Gemini‑1.5‑Flash emerged as the most cost‑efficient model, reducing cost per successful parsing to $0.007 compared to $0.013 for GPT‑3.5‑turbo and $0.019 for Claude‑3 Opus, while meeting latency and accuracy targets.

工作流

• 1. Define relevant dimensions: capability, reasoning, latency, context length, modality, cost.
• 2. Build a benchmark that feeds real statements, requests JSON line items, and measures success by JSON parse and total match.
• 3. Run benchmark against candidate models (OpenAI gpt‑3.5‑turbo‑16k, AWS Bedrock Claude‑3 Opus, Google Gemini‑1.5‑Flash).
• 4. Calculate cost‑per‑successful‑task as (cost per call)/(success rate).
• 5. Select model with lowest cost‑per‑successful‑task (Gemini‑Flash).
• 6. Version‑track the chosen model and re‑benchmark on upgrades.
• 7. Apply same process to other projects (e.g., ed‑tech quiz scoring).

证据

After a weekend of latency checks, cost tallies and exploding error logs I realized the “best” model on every public leaderboard still missed our real KPI: cost per successful parsing.

来源

I built a tiny benchmark that mirrors the exact workflow: feed a real statement, ask for JSON line items, measure success (JSON parses, totals match).

来源

Cost‑per‑successful‑task = (cost per call) ÷ (success rate).

来源

Gemini‑Flash became the clear winner for our budget‑tight pipeline, even though its raw accuracy looked lower on paper.

来源

The trick is to version‑track the exact model you use (e.g., gpt-3.5-turbo-0613) and re‑run the benchmark whenever a new upgrade lands.

来源