NIRNAY: a 450M open model that beats Jev

说明与用途

NIRNAY 是一个 450 M 参数的开源模型，在 Banking77 基准上以 0.8792 的 F1 分数超过 Jev 1.13.0，且本地推理时间约 200 ms，无需 API 密钥。

问题

Fine‑tuned open models can outperform zero‑shot APIs on intent classification tasks.

结果

NIRNAY phase_b achieved 0.8792 F1 on Banking77, surpassing Jev 1.13.0 (0.803) while running locally in ~200 ms and requiring no API key.

工作流

• Fine‑tune a 421 M Laya fork with ~30 M additions on the Banking77 training split.
• Apply affine‑free LayerNorm to the concept encoder output before quantization to prevent scale runaway.
• Add a load‑balancing auxiliary loss to avoid usage collapse during training.
• Use held‑out probes every 250 steps, retain best checkpoint, and abort if accuracy halves.
• Evaluate the trained model on the full Banking77 test set using the provided harness.
• Deploy the model locally via Hugging Face checkpoint without GPU.
• Run inference in ~200 ms per example.
• Compare performance to Jev 1.13.0 and other baselines.
• Publish code, data, and evaluation scripts on GitHub.

证据

NIRNAY phase_b scores 0.8792 on the full Banking77 test set (3,080 cases).

来源

Jev 1.13.0 scores 0.803 on the same cases (jevbench.xyz recorded run).

来源

Our runs local in ~200ms, no API key, Apache-2.0.

来源

Fine‑tuned on the train split, Jev answered zero‑shot. Same test cases, different preparation.

来源