NIRNAY: a 450M open model that beats Jev

Description and use

NIRNAY is a 450 M‑parameter open model that outperforms Jev 1.13.0 on the Banking77 benchmark (F1 = 0.8792) while running locally in ~200 ms and requiring no API key.

Problem

Fine‑tuned open models can outperform zero‑shot APIs on intent classification tasks.

Outcome

NIRNAY phase_b achieved 0.8792 F1 on Banking77, surpassing Jev 1.13.0 (0.803) while running locally in ~200 ms and requiring no API key.

Workflow

• Fine‑tune a 421 M Laya fork with ~30 M additions on the Banking77 training split.
• Apply affine‑free LayerNorm to the concept encoder output before quantization to prevent scale runaway.
• Add a load‑balancing auxiliary loss to avoid usage collapse during training.
• Use held‑out probes every 250 steps, retain best checkpoint, and abort if accuracy halves.
• Evaluate the trained model on the full Banking77 test set using the provided harness.
• Deploy the model locally via Hugging Face checkpoint without GPU.
• Run inference in ~200 ms per example.
• Compare performance to Jev 1.13.0 and other baselines.
• Publish code, data, and evaluation scripts on GitHub.

Evidence

NIRNAY phase_b scores 0.8792 on the full Banking77 test set (3,080 cases).

Source

Jev 1.13.0 scores 0.803 on the same cases (jevbench.xyz recorded run).

Source

Our runs local in ~200ms, no API key, Apache-2.0.

Source

Fine‑tuned on the train split, Jev answered zero‑shot. Same test cases, different preparation.

Source