Your 94% accuracy model might still be a bad model: auditing ML across 7 dimensions

Description and use

A 94% accurate model can still be bad due to missing checks on robustness, explainability, calibration, data quality, fairness, and complexity. The ai‑critic library audits these seven dimensions, producing a transparent report and overall score while excluding inconclusive metrics.

Problem

A model with high accuracy can still fail in production due to overconfidence, noise sensitivity, or subgroup performance gaps.

Outcome

Using ai‑critic, a Random Forest classifier on the breast cancer dataset produced an audit report with an overall score and verdict, demonstrating how to surface hidden model weaknesses beyond accuracy.

Workflow

• 1. Import ai_critic and a dataset (e.g., load_breast_cancer).
• 2. Split data into train/test sets.
• 3. Train a model (e.g., RandomForestClassifier).
• 4. Call ai_critic.audit(model, (X_test, y_test), task_type="classification") to generate a report.
• 5. Inspect report.overall_score and report.overall_verdict.value.
• 6. Optionally enforce a quality gate with ai_critic.gate(report, min_score=0.75).

Evidence

We usually judge a model by one number. But a classifier with great accuracy can be overconfident, collapse under small input noise, or perform much worse for one subgroup.

Source

ai-critic is an open-source Python library that audits a model across seven dimensions: performance, robustness, explainability, calibration, data quality, fairness, and complexity.

Source

If a dimension can’t be measured (no sensitive groups declared, no probabilities for calibration), it’s marked inconclusive or not_applicable and excluded from the overall score, instead of getting a flattering default.

Source

ai_critic.gate(report, min_score=0.75) # Raises if below threshold

Source