When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise. Because such judges prefer verbose or well-formatted answers, the natural fix is to add bias covariates to a Bradley–Terry model and estimate the bias away. We show this cannot work as advertised: the quality/bias split is \emph{not identified} by pairwise comparisons, and the failure is exact – across $48$ real judge-pools the profile likelihood over the coefficient is flat to $\mathbf{0.0000}$ $\mathbf{\text{nats}}$, and scaling the comparisons $26\times$ buys none. A ``debiased’’ score is selected by the prior, not recovered from data. Our contribution is accordingly not a better estimator but a characterization of \emph{when prior-based correction is justified}, plus designs that supply the missing information when it is not. The assumption the prior encodes – quality is a priori uncorrelated with the covariate – pays only while $\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}(\theta ,x)$ stays below a crossing point (configuration-dependent, $0.22$–$0.60$), which is what makes the same model help on LLMBar and hurt on SummEval and Nectar. We give two escapes: a $\mathbf{\text{trusted-anchor}}\mathbf{\text{gate}}$ that decides per (judge, covariate, task) (no false enables in $\mathrm{6,000}$ decisions at $K\ge 10$ anchors, a rate our sample bounds at $\le 6\mathrm{\%}$), and a $\mathbf{\text{paired}}\mathbf{\text{rendering}}\mathbf{\text{design}}$. Across fifteen real LLM judges bias is heterogeneous and capability-dependent: correction improves \topk{} recall by $0.20$–$0.32$ on five biased-but-competent cheap judges and is a no-op on frontier ones (Spearman $\rho =-0.84$ between competence and gain over the $14$ competent judges, $p<{10}^{-3}$), concentrating the benefit where at-scale evaluation happens.