Laya vs Jev in Practice: Zero‑Shot, Fine‑Tuning, and the Real Cost of an Open‑Source Decision Model

Description and use

Fine‑tuning Laya on domain‑specific synthetic data closed the performance gap with Jev, raising exact‑tuple accuracy from 11.43 % to 85.71 % on the same held‑out benchmark.

Problem

Assessing the performance gap between an open‑weight decision model (Laya) and a proprietary model (Jev) on a ticket triage benchmark, and determining the impact of domain adaptation.

Outcome

Fine‑tuning Laya on domain‑specific synthetic data closed the performance gap with Jev, raising exact‑tuple accuracy from 11.43 % to 85.71 % on the same held‑out benchmark.

Workflow

• 1. Retrieve frozen held‑out set of 70 tickets and labels.
• 2. Evaluate base Laya zero‑shot on the set, record metrics.
• 3. Create 1,120 synthetic TRAIN tickets and 280 VALIDATION tickets using deterministic generator.
• 4. Fine‑tune Laya from the same base checkpoint on TRAIN tickets for 4 epochs with mixed precision and gradient accumulation.
• 5. Validate checkpoints on VALIDATION set, select checkpoint with highest exact‑tuple accuracy (epoch 4).
• 6. Freeze adapted checkpoint (commit 00a9a64).
• 7. Evaluate frozen adapted checkpoint on the held‑out set, record metrics.
• 8. Compare results to Jev 1.13 benchmark (66/70).

Evidence

Laya went from 8/70 correct tuples zero‑shot to 60/70 after domain adaptation

Source

The clearest pattern was 40/40 expected LOW priorities predicted as MEDIUM

Source

We created 1,120 TRAIN tickets and 280 VALIDATION tickets, also synthetic

Source

Fine‑tuning started from the same base checkpoint as A

Source