Ranking Reasoning LLMs under Test-Time Scaling

Test-time scaling evaluates reasoning LLMs by sampling multiple outputs per prompt, but ranking models in this regime remains underexplored. We formalize dense benchmark ranking under test-time scaling and introduce Scorio, a library that implements statistical ranking methods such as paired-comparison models, item response theory (IRT) models, voting rules, and graph- and spectral-based methods. Across $20$ reasoning models on four Olympiad-style math benchmarks (AIME’24, AIME’25, HMMT’25, and BrUMO’25; up to $N=80$ trials), most full-trial rankings agree closely with the Bayesian gold standard ${\mathrm{B}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{s}}_{\mathcal{U}}\mathrm{@}80$ (mean Kendall’s ${\tau }_b=0.93$–$0.95$), and $19$–$34$ methods recover exactly the same ordering. In the single-trial regime, the best methods reach ${\tau }_b\approx 0.86$. Using greedy decoding as an empirical prior (${\mathrm{B}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{s}}_{{\mathbf{R}}_0}\mathrm{@}N$) reduces variance at $N=1$ by $16$–$52\mathrm{\%}$, but can bias rankings when greedy and stochastic sampling disagree. These results identify reliable ranking methods for both high- and low-budget test-time scaling. We release Scorio as an open-source library at https://github.com/mohsenhariri/scorio.