My LoRA improved routing and made JSON less reliable

Описание и применение

Улучшение маршрутизации модели Qwen2.5‑0.5B‑Instruct с помощью LoRA повысило точность маршрутизации до 72 % и схему‑валидность до 95 %, но снизило валидность JSON до 95,5 % и оставило точность вычислений нулевой.

Задача

Improve routing accuracy of a small model while maintaining JSON validity.

Результат

The LoRA adapter substantially increased routing accuracy and schema compliance, but reduced strict JSON validity and left calculation accuracy at zero.

Рабочий процесс

• 1. Construct synthetic dataset with 1,026 training, 291 validation, 576 test rows across 23 families and 7 clusters.
• 2. Train LoRA adapter (rank 8, scale 20, 16 layers, lr 1e‑4, batch 1, 1,500 iterations, seed 42) on Apple silicon using MLX.
• 3. Evaluate base and adapter with greedy decoding, 96‑token limit, same prompt.
• 4. Score outputs with frozen scorer: strict JSON, schema‑valid, correct route, strict all‑field match, cross‑language equal and correct.
• 5. Analyze failures: missing‑information requests, unsupported operations, repeated keys.
• 6. Publish adapter, dataset, and Kaggle notebook for reproducibility.
• 7. Plan next experiments on missing‑information handling and constrained decoding.

Доказательства

Strict all-field accuracy rose from 7.6% to 47.2%. JSON validity fell from 98.6% to 95.5%.

Источник

The base model usually produced parseable JSON, but often used fields or routes outside the contract.

Источник

The adapter’s strict calculation score was zero.

Источник

I would keep this test frozen and build a separate experiment around the weaknesses it exposed: missing-information requests, unsupported requests, and operations absent from training.

Источник