Fine-Tuned Qwen2.5-1.5B vs Claude Opus 5 for JSON Extraction

说明与用途

比较显示，经过微调的小型模型 Qwen2.5-1.5B 在特定领域的 JSON 提取任务中优于 Claude Opus 5，字段匹配准确率达到 96.72%，而 Claude 为 81.81%，同时成本更低且对数据的控制力更强。

问题

Determine if a fine-tuned small language model (Qwen2.5-1.5B) can outperform a state-of-the-art frontier model (Claude Opus 5) on a domain-specific JSON extraction task.

结果

The fine-tuned Qwen2.5-1.5B achieved 96.72% field match accuracy and 62% exact match accuracy, compared to Claude Opus 5's 81.81% field match and 0% exact match. The small model was also much less expensive and provided more control over data.

工作流

• Prepare a domain-specific dataset for JSON extraction.
• Fine-tune Qwen2.5-1.5B-Instruct using QLoRA.
• Define evaluation metrics: Exact Match (binary per sample) and Field Match (partial credit per sample).
• Run the fine-tuned Qwen2.5-1.5B on the test set.
• Run Claude Opus 5 on the identical test set using the same metrics.
• Compare results across accuracy, cost, and data control.

证据

Despite being ~1,000x smaller than Claude Opus-5, our fine-tuned Qwen2.5-1.5B achieved: ~15 percentage points higher field match accuracy (96.72% vs 81.81%), 62% exact match accuracy vs Claude's 0%

来源