How well can a small, local model classify AI incidents?

Description and use

A locally‑run Llama 3.1 8B model classifies AIID incidents with 62.1% accuracy on 7 broad domains and 47.6% on 24 subdomains, enabling pre‑sorting and risk checklist generation.

Problem

Automated sorting of AI Incident Database entries into harm taxonomy labels is currently manual and does not scale with growing data.

Outcome

The small local model achieves moderate agreement with human labels, especially at the broad domain level, making it useful for triage but not for fine‑grained labeling.

Workflow

• Pull weekly AIID export and flatten into a table
• For each incident, feed title, description, and allowed labels to the model and request one label
• Compare model output to human labels for ~1,500 incidents
• Compute accuracy and Cohen’s kappa for 7‑domain and 24‑subdomain levels

Evidence

Across 1,495 incidents, with not one malformed answer, here's where it landed: 24‑way subdomain accuracy 47.6% (kappa 0.41), 7‑domain accuracy 62.1% (kappa 0.51).

Source

The model is much better at the broad domain than the specific subdomain. Kappa of 0.41 on the fine‑grained labels is fair agreement. At the domain level it climbs to 0.51, solidly moderate.

Source