Fine-Tuned Qwen2.5-1.5B vs Claude Opus 5 for JSON Extraction

Описание и применение

Сравнение показывает, что дообученная малая модель Qwen2.5-1.5B превосходит Claude Opus 5 в извлечении JSON для специфичной области, достигая 96.72% точности полей против 81.81% у Claude, при этом будучи дешевле и обеспечивая больший контроль над данными.

Задача

Determine if a fine-tuned small language model (Qwen2.5-1.5B) can outperform a state-of-the-art frontier model (Claude Opus 5) on a domain-specific JSON extraction task.

Результат

The fine-tuned Qwen2.5-1.5B achieved 96.72% field match accuracy and 62% exact match accuracy, compared to Claude Opus 5's 81.81% field match and 0% exact match. The small model was also much less expensive and provided more control over data.

Рабочий процесс

• Prepare a domain-specific dataset for JSON extraction.
• Fine-tune Qwen2.5-1.5B-Instruct using QLoRA.
• Define evaluation metrics: Exact Match (binary per sample) and Field Match (partial credit per sample).
• Run the fine-tuned Qwen2.5-1.5B on the test set.
• Run Claude Opus 5 on the identical test set using the same metrics.
• Compare results across accuracy, cost, and data control.

Доказательства

Despite being ~1,000x smaller than Claude Opus-5, our fine-tuned Qwen2.5-1.5B achieved: ~15 percentage points higher field match accuracy (96.72% vs 81.81%), 62% exact match accuracy vs Claude's 0%

Источник