Your 94% accuracy model might still be a bad model: auditing ML across 7 dimensions

说明与用途

94% 准确率的模型可能仍然存在问题，因为缺乏对鲁棒性、可解释性、校准、数据质量、公平性和复杂度的评估。ai‑critic 库会对这七个维度进行审核，生成透明的报告和总体得分，并排除不确定的指标。

问题

A model with high accuracy can still fail in production due to overconfidence, noise sensitivity, or subgroup performance gaps.

结果

Using ai‑critic, a Random Forest classifier on the breast cancer dataset produced an audit report with an overall score and verdict, demonstrating how to surface hidden model weaknesses beyond accuracy.

工作流

• 1. Import ai_critic and a dataset (e.g., load_breast_cancer).
• 2. Split data into train/test sets.
• 3. Train a model (e.g., RandomForestClassifier).
• 4. Call ai_critic.audit(model, (X_test, y_test), task_type="classification") to generate a report.
• 5. Inspect report.overall_score and report.overall_verdict.value.
• 6. Optionally enforce a quality gate with ai_critic.gate(report, min_score=0.75).

证据

We usually judge a model by one number. But a classifier with great accuracy can be overconfident, collapse under small input noise, or perform much worse for one subgroup.

来源

ai-critic is an open-source Python library that audits a model across seven dimensions: performance, robustness, explainability, calibration, data quality, fairness, and complexity.

来源

If a dimension can’t be measured (no sensitive groups declared, no probabilities for calibration), it’s marked inconclusive or not_applicable and excluded from the overall score, instead of getting a flattering default.

来源

ai_critic.gate(report, min_score=0.75) # Raises if below threshold

来源