The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains

Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur: (1) the evaluation matrix is sparse and (2) items vary substantially in difficulty. Through controlled simulation experiments across four domains – NLP (GLUE), clinical drug trials, autonomous vehicle safety, and cybersecurity – we show that Spearman rank correlation $\rho$ between simple-average rankings and ground-truth rankings degrades from $\rho =1.000$ at 100% coverage to $\rho =0.809$ at 67% coverage with high difficulty heterogeneity (mean over 20 seeds). A standard two-parameter logistic (2PL) Item Response Theory (IRT) model maintains $\rho \ge 0.996$ across all conditions. A 150-condition grid sweep over sparsity $S\in [0,0.70]$ and difficulty gap $D\in [0.5,5.0]$ confirms that ranking error forms a failure surface with a strong $S\times D$ interaction (${\gamma }_3=+0.20$, $t=13.05$), while IRT maintains $\rho \ge 0.993$ throughout. We discuss implications for Physical AI benchmarking, where evaluation matrices are often incomplete and difficulty gaps are extreme.