Is Model A Really Better Than B?

Determining whether Model A's 0.5% offline metric gain over Model B is statistically significant or random sampling noise.

🔴 advanced5 min readevaluation

Comparing two ML models requires statistical hypothesis testing to prove that Model A's performance gain over Model B is statistically significant. Standard Student's t-test violates independence assumptions when run on overlapping cross-validation folds (Dietterich, 1998). Recommended techniques include 5x2-fold Cross-Validation Paired t-test, McNemar's Test (for paired binary classification outputs), and Non-Parametric Bootstrap Resampling to construct confidence intervals around metric differences.

Why Point Estimates Are Misleading

Model A scores AUC = 0.885; Model B scores AUC = 0.880.

Is Model A genuinely superior, or did it get lucky on a favorable test set split?

                       95% Confidence Intervals of Metric Difference ΔAUC
  Scenario 1: [==== (0.001 to 0.009) ====]  ──► Statistically Significant (Interval excludes 0!)
  Scenario 2: [====== (-0.003 to 0.013) ======]  ──► NOT Statistically Significant (Interval crosses 0!)

If the 95% confidence interval of $\Delta = \text{Metric}_A - \text{Metric}_B$ crosses zero, Model A is NOT provably better than Model B.

Three Rigorous Model Comparison Tests

1. McNemar's Test (Paired Binary Predictions)

Fast, non-parametric test operating on a $2 \times 2$ contingency table of paired predictions on the same fixed test set:

                       Model B Correct       Model B Incorrect
  Model A Correct            n_00                   n_01  (A right, B wrong)
  Model A Incorrect          n_10 (B right, A wrong) n_11

McNemar Test Statistic (Chi-Square with 1 degree of freedom):

\chi^2 = \frac{( |n_{01} - n_{10}| - 1 )^2}{n_{01} + n_{10}}

If $\chi^2 > 3.841$ ( $p < 0.05$ ), the disagreement rates are statistically significant!

2. 5x2-Fold Cross-Validation Paired t-Test (Dietterich, 1998)

Corrects for overlapping fold correlations:

Run 2-fold cross-validation 5 times (total 10 folds).
Compute variance strictly within each 2-fold replication to avoid cross-replication correlation bugs.

3. Bootstrap Resampling Confidence Intervals (Gold Standard)

No model re-training required!

Given saved test predictions $\hat{y}_A$ and $\hat{y}_B$ for $N$ test samples.
For $b = 1 \dots 1000$ $b = 1 \dots 1000$ :
- Sample $N$ indices with replacement from test set.
- Compute $\Delta^{(b)} = \text{Metric}_A^{(b)} - \text{Metric}_B^{(b)}$ .
Sort 1,000 $\Delta^{(b)}$ values. 95% CI is $[\Delta_{(25)}, \Delta_{(975)}]$ .

Say this out loud

"To prove Model A is statistically better than Model B, we construct confidence intervals around metric differences Δ = Metric_A - Metric_B. Standard paired t-tests over 10-fold CV fail due to training set overlap correlations. We use McNemar's test for paired classification outputs, or Bootstrap Resampling over saved test predictions to construct 95% confidence intervals around Δ metric gains."

Follow-ups to expect

What is the Bonferroni Correction for multi-model comparison? If comparing 10 candidate models against a baseline, running 10 hypothesis tests at $\alpha = 0.05$ yields a high multi-testing false positive rate ( $1 - 0.95^{10} \approx 40\%$ ). Bonferroni divides significance threshold by number of tests: $\alpha_{\text{new}} = \alpha / K$ .
How large does a test set need to be to detect a 0.5% AUC lift? Depends on variance. Using power calculations for paired proportions, detecting a 0.5% difference with 80% power typically requires $N > 20,000$ paired test samples.

Check yourself

Question 1 of 3

Why does running a standard Paired Student's t-test across 10 folds of standard 10-fold cross-validation yield an artificially inflated Type I error rate?

More in Metrics & Evaluation

See all →

Precision, Recall & F14 min Why Accuracy Lies4 min ROC-AUC vs PR-AUC4 min