Building laya-triage, part 2: today's issues, a new model, and OpenAI's Decisions API

Description and use

Version 1.2 of the laya‑triage GitHub Action improves new issue classification accuracy to 82.3 % on recent data, adds backlog mode, run summaries, and experimental repository voting, while remaining free and using calibrated confidence thresholds.

Problem

Accurate automatic labeling of new GitHub issues into bug, feature, question, or docs categories has degraded over time, and existing models struggle with recent issue data and confidence calibration.

Outcome

v1.2 of laya-triage achieves 82.3% accuracy on recent issues, 88.8% on NLBSE'23, and outperforms OpenAI Decisions API in terms of wrong label rate (12 vs 20–21). It remains free to run and offers calibrated confidence to reduce mislabeling.

Workflow

• Collect 10,026 closed issues from 2025‑2026 across 288 active repos; filter by maintainer labels; evaluate v1.0 accuracy.
• Fine‑tune v1.0 on 127,140 recent issues; observe 81.9% recent accuracy but drop on NLBSE'23.
• Train v1.1 from scratch on combined NLBSE'23 + recent data; achieve 88.8% on NLBSE'23 and 82.3% on recent issues.
• Add v1.2 features: backlog mode, run summaries, experimental repo memory voting, expanded label names.
• Measure OpenAI Decisions API on same datasets; compare accuracy and cost.
• Publish all scripts and raw results in bench/ directory.
• Deploy laya-triage via GitHub Action with dry‑run mode.

Evidence

v1.0 scored 88.8% on the NLBSE'23 benchmark, and I was happy with that. Then I tested it on issues people are writing today, and it dropped to 79.8%.

Source

v1.1 keeps 88.8% on NLBSE'23 and reached 82.3% on recent issues.

Source

v1.2 keeps the v1.1 models and adds: Backlog mode, Run summaries, Repo memory (experimental), More label names.

Source

The Decisions API is still in beta, so these numbers may change. I measured it on October 6, 2026.

Source