Building laya-triage, part 2: today's issues, a new model, and OpenAI's Decisions API
Описание и применение
Версия 1.2 GitHub‑действия laya‑triage повышает точность классификации новых проблем до 82,3 % на свежих данных, добавляет режим обработки старых задач, сводки выполнения и экспериментальный механизм голосования по репозиторию, при этом оставаясь бесплатной и с калиброванным порогом уверенности.
Задача
Accurate automatic labeling of new GitHub issues into bug, feature, question, or docs categories has degraded over time, and existing models struggle with recent issue data and confidence calibration.
Результат
v1.2 of laya-triage achieves 82.3% accuracy on recent issues, 88.8% on NLBSE'23, and outperforms OpenAI Decisions API in terms of wrong label rate (12 vs 20–21). It remains free to run and offers calibrated confidence to reduce mislabeling.
Рабочий процесс
- Collect 10,026 closed issues from 2025‑2026 across 288 active repos; filter by maintainer labels; evaluate v1.0 accuracy.
- Fine‑tune v1.0 on 127,140 recent issues; observe 81.9% recent accuracy but drop on NLBSE'23.
- Train v1.1 from scratch on combined NLBSE'23 + recent data; achieve 88.8% on NLBSE'23 and 82.3% on recent issues.
- Add v1.2 features: backlog mode, run summaries, experimental repo memory voting, expanded label names.
- Measure OpenAI Decisions API on same datasets; compare accuracy and cost.
- Publish all scripts and raw results in bench/ directory.
- Deploy laya-triage via GitHub Action with dry‑run mode.
Доказательства
v1.0 scored 88.8% on the NLBSE'23 benchmark, and I was happy with that. Then I tested it on issues people are writing today, and it dropped to 79.8%.
v1.1 keeps 88.8% on NLBSE'23 and reached 82.3% on recent issues.
v1.2 keeps the v1.1 models and adds: Backlog mode, Run summaries, Repo memory (experimental), More label names.
The Decisions API is still in beta, so these numbers may change. I measured it on October 6, 2026.
- Дата записи: 2026-10-07T17:41:10.505Z