How well can a small, local model classify AI incidents?

说明与用途

本地运行的 Llama 3.1 8B 模型在 AIID 事件分类中，7 大类准确率 62.1%，24 小类 47.6%，可用于预先排序和风险清单生成。

问题

Automated sorting of AI Incident Database entries into harm taxonomy labels is currently manual and does not scale with growing data.

结果

The small local model achieves moderate agreement with human labels, especially at the broad domain level, making it useful for triage but not for fine‑grained labeling.

工作流

• Pull weekly AIID export and flatten into a table
• For each incident, feed title, description, and allowed labels to the model and request one label
• Compare model output to human labels for ~1,500 incidents
• Compute accuracy and Cohen’s kappa for 7‑domain and 24‑subdomain levels

证据

Across 1,495 incidents, with not one malformed answer, here's where it landed: 24‑way subdomain accuracy 47.6% (kappa 0.41), 7‑domain accuracy 62.1% (kappa 0.51).

来源

The model is much better at the broad domain than the specific subdomain. Kappa of 0.41 on the fine‑grained labels is fair agreement. At the domain level it climbs to 0.51, solidly moderate.

来源