Building laya-triage, part 2: today's issues, a new model, and OpenAI's Decisions API

说明与用途

laya‑triage 1.2版将新问题分类准确率提升至 82.3%，新增 backlog 模式、运行摘要和实验性仓库投票功能，并保持免费且使用校准的置信度阈值。

问题

Accurate automatic labeling of new GitHub issues into bug, feature, question, or docs categories has degraded over time, and existing models struggle with recent issue data and confidence calibration.

结果

v1.2 of laya-triage achieves 82.3% accuracy on recent issues, 88.8% on NLBSE'23, and outperforms OpenAI Decisions API in terms of wrong label rate (12 vs 20–21). It remains free to run and offers calibrated confidence to reduce mislabeling.

工作流

• Collect 10,026 closed issues from 2025‑2026 across 288 active repos; filter by maintainer labels; evaluate v1.0 accuracy.
• Fine‑tune v1.0 on 127,140 recent issues; observe 81.9% recent accuracy but drop on NLBSE'23.
• Train v1.1 from scratch on combined NLBSE'23 + recent data; achieve 88.8% on NLBSE'23 and 82.3% on recent issues.
• Add v1.2 features: backlog mode, run summaries, experimental repo memory voting, expanded label names.
• Measure OpenAI Decisions API on same datasets; compare accuracy and cost.
• Publish all scripts and raw results in bench/ directory.
• Deploy laya-triage via GitHub Action with dry‑run mode.

证据

v1.0 scored 88.8% on the NLBSE'23 benchmark, and I was happy with that. Then I tested it on issues people are writing today, and it dropped to 79.8%.

来源

v1.1 keeps 88.8% on NLBSE'23 and reached 82.3% on recent issues.

来源

v1.2 keeps the v1.1 models and adds: Backlog mode, Run summaries, Repo memory (experimental), More label names.

来源

The Decisions API is still in beta, so these numbers may change. I measured it on October 6, 2026.

来源