My LoRA improved routing and made JSON less reliable

说明与用途

使用 LoRA 微调 Qwen2.5‑0.5B‑Instruct 模型后，路由准确率提升至 72%，模式有效率提升至 95%，但严格 JSON 有效率下降至 95.5%，计算准确率仍为零。

问题

Improve routing accuracy of a small model while maintaining JSON validity.

结果

The LoRA adapter substantially increased routing accuracy and schema compliance, but reduced strict JSON validity and left calculation accuracy at zero.

工作流

• 1. Construct synthetic dataset with 1,026 training, 291 validation, 576 test rows across 23 families and 7 clusters.
• 2. Train LoRA adapter (rank 8, scale 20, 16 layers, lr 1e‑4, batch 1, 1,500 iterations, seed 42) on Apple silicon using MLX.
• 3. Evaluate base and adapter with greedy decoding, 96‑token limit, same prompt.
• 4. Score outputs with frozen scorer: strict JSON, schema‑valid, correct route, strict all‑field match, cross‑language equal and correct.
• 5. Analyze failures: missing‑information requests, unsupported operations, repeated keys.
• 6. Publish adapter, dataset, and Kaggle notebook for reproducibility.
• 7. Plan next experiments on missing‑information handling and constrained decoding.

证据

Strict all-field accuracy rose from 7.6% to 47.2%. JSON validity fell from 98.6% to 95.5%.

来源

The base model usually produced parseable JSON, but often used fields or routes outside the contract.

来源

The adapter’s strict calculation score was zero.

来源

I would keep this test frozen and build a separate experiment around the weaknesses it exposed: missing-information requests, unsupported requests, and operations absent from training.

来源