NIRNAY: a 450M open model that beats Jev

Описание и применение

NIRNAY – 450‑млн параметров открытая модель, превосходящая Jev 1.13.0 на тесте Banking77 с F1‑значением 0.8792 и временем инференса ~200 мс без ключа API.

Задача

Fine‑tuned open models can outperform zero‑shot APIs on intent classification tasks.

Результат

NIRNAY phase_b achieved 0.8792 F1 on Banking77, surpassing Jev 1.13.0 (0.803) while running locally in ~200 ms and requiring no API key.

Рабочий процесс

• Fine‑tune a 421 M Laya fork with ~30 M additions on the Banking77 training split.
• Apply affine‑free LayerNorm to the concept encoder output before quantization to prevent scale runaway.
• Add a load‑balancing auxiliary loss to avoid usage collapse during training.
• Use held‑out probes every 250 steps, retain best checkpoint, and abort if accuracy halves.
• Evaluate the trained model on the full Banking77 test set using the provided harness.
• Deploy the model locally via Hugging Face checkpoint without GPU.
• Run inference in ~200 ms per example.
• Compare performance to Jev 1.13.0 and other baselines.
• Publish code, data, and evaluation scripts on GitHub.

Доказательства

NIRNAY phase_b scores 0.8792 on the full Banking77 test set (3,080 cases).

Источник

Jev 1.13.0 scores 0.803 on the same cases (jevbench.xyz recorded run).

Источник

Our runs local in ~200ms, no API key, Apache-2.0.

Источник

Fine‑tuned on the train split, Jev answered zero‑shot. Same test cases, different preparation.

Источник