POOL: Propagated Uncertainty Over Lookalikes

Black-box large language models need confidence scores that can separate likely-correct from likely-incorrect outputs, enabling systems to prioritize human review, route uncertain cases to stronger models, or choose abstention thresholds on development data. Yet existing confidence estimators face a cost-quality trade-off: verbal confidence is cheap but is often overconfident, while sampling-based uncertainty is more informative but scales linearly with the number of samples per query. We propose $\text{textsc}POOL$ (\emph{Propagated Uncertainty Over Lookalikes}),a cost-efficient framework that addresses this trade-off taking inspiration from group-testing.$\text{textsc}POOL$ clusters query stems with overlaps, evaluates a base estimator on representative medoids, softly propagates confidence scores to nearby queries, and selectively evaluates high-disagreement cases. We instantiate this framework with $\text{textsc}Hy\mathrm{@}$$p$, a hybrid estimator that combines verbal confidence with spectral answer diversity computed from the negative von Neumann entropy of sampled answer embeddings.Across six domains from three datasets and five black-box LLMs, $\text{textsc}Hy\mathrm{@}$5 achieves higher average AUROC than verbal confidence and $\text{textsc}Vn\mathrm{@}$10 sampling while using half as many samples as $\text{textsc}Vn\mathrm{@}$10. $\text{textsc}POOL$-$\text{textsc}Hy\mathrm{@}$5 retains 93.5–97.9% of its AUROC while saving 19.3–39.3% of generations. On paraphrase-dense workloads, generation savings rise to 73-76%, showing that semantic redundancy can be leveraged to lower confidence-estimation costs.