Fine-Tuned Qwen2.5-1.5B vs Claude Opus 5 for JSON Extraction

Description and use

Comparison shows that a fine-tuned small model Qwen2.5-1.5B outperforms Claude Opus 5 in domain-specific JSON extraction, achieving 96.72% field match accuracy vs 81.81% for Claude, while being more cost-effective and offering greater data control.

Problem

Determine if a fine-tuned small language model (Qwen2.5-1.5B) can outperform a state-of-the-art frontier model (Claude Opus 5) on a domain-specific JSON extraction task.

Outcome

The fine-tuned Qwen2.5-1.5B achieved 96.72% field match accuracy and 62% exact match accuracy, compared to Claude Opus 5's 81.81% field match and 0% exact match. The small model was also much less expensive and provided more control over data.

Workflow

• Prepare a domain-specific dataset for JSON extraction.
• Fine-tune Qwen2.5-1.5B-Instruct using QLoRA.
• Define evaluation metrics: Exact Match (binary per sample) and Field Match (partial credit per sample).
• Run the fine-tuned Qwen2.5-1.5B on the test set.
• Run Claude Opus 5 on the identical test set using the same metrics.
• Compare results across accuracy, cost, and data control.

Evidence

Despite being ~1,000x smaller than Claude Opus-5, our fine-tuned Qwen2.5-1.5B achieved: ~15 percentage points higher field match accuracy (96.72% vs 81.81%), 62% exact match accuracy vs Claude's 0%

Source