My LoRA improved routing and made JSON less reliable

Description and use

Fine‑tuning Qwen2.5‑0.5B‑Instruct with a LoRA adapter improved routing accuracy to 72 % and schema validity to 95 %, but reduced strict JSON validity to 95.5 % and left calculation accuracy at zero.

Problem

Improve routing accuracy of a small model while maintaining JSON validity.

Outcome

The LoRA adapter substantially increased routing accuracy and schema compliance, but reduced strict JSON validity and left calculation accuracy at zero.

Workflow

• 1. Construct synthetic dataset with 1,026 training, 291 validation, 576 test rows across 23 families and 7 clusters.
• 2. Train LoRA adapter (rank 8, scale 20, 16 layers, lr 1e‑4, batch 1, 1,500 iterations, seed 42) on Apple silicon using MLX.
• 3. Evaluate base and adapter with greedy decoding, 96‑token limit, same prompt.
• 4. Score outputs with frozen scorer: strict JSON, schema‑valid, correct route, strict all‑field match, cross‑language equal and correct.
• 5. Analyze failures: missing‑information requests, unsupported operations, repeated keys.
• 6. Publish adapter, dataset, and Kaggle notebook for reproducibility.
• 7. Plan next experiments on missing‑information handling and constrained decoding.

Evidence

Strict all-field accuracy rose from 7.6% to 47.2%. JSON validity fell from 98.6% to 95.5%.

Source

The base model usually produced parseable JSON, but often used fields or routes outside the contract.

Source

The adapter’s strict calculation score was zero.

Source

I would keep this test frozen and build a separate experiment around the weaknesses it exposed: missing-information requests, unsupported requests, and operations absent from training.

Source