What is the most common mistake in AI evaluation?

• A. Using cross-validation to estimate performance.
• B. Evaluating with a variety of metrics.
• C. Relying only on accuracy.
• D. Ignoring data quality.

Answer: (C)

Relying only on accuracy to judge AI performance is the most common pitfall in evaluation. Accuracy can be misleading, especially when classes are imbalanced or when different mistakes carry different costs. A model could achieve high accuracy by predicting the majority class all the time and still miss the important minority cases, misclassify costly instances, or provide poorly calibrated probabilities. This hides meaningful differences in how the model behaves across conditions and real-world impact.

To evaluate properly, use a suite of metrics that expose different aspects of performance. Precision and recall (and the F1 score) reveal trade-offs between false positives and false negatives; ROC-AUC or PR-AUC shows how well the model ranks positive instances; calibration tells you whether predicted probabilities reflect true frequencies. A confusion matrix helps visualize where errors occur, and domain-specific costs can guide whether a small change in one metric is worth it. Cross-validation helps ensure estimates generalize beyond a single dataset, and attention to data quality remains crucial, but the key takeaway is that relying solely on accuracy misses the fuller picture of model behavior.