Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarks

Progress in colonoscopy polyp segmentation is routinely reported through leaderboard comparisons on a small set of public benchmarks. We argue that this apparent progress is difficult to verify: a systematic audit of $\mathbf{\text{27}}\mathbf{\text{papers}}$ published between 2015 and 2026 reveals three structural problems in how the community evaluates models. $\mathbf{\text{First}}$, 25 of 27 papers $\mathit{\text{omit}}\mathit{\text{the}}\mathit{\text{Hausdorff}}\mathit{\text{distance}}$. Hausdorff distance is a boundary-accuracy metric with direct clinical relevance for detecting flat or small polyps, and is a standard in radiotherapy segmentation. $\mathbf{\text{Second}}$, at least five $\mathit{\text{incompatible}}\mathit{\text{train/test}}\mathit{\text{split}}\mathit{\text{protocols}}$ co-exist across papers reporting results on the same two datasets (Kvasir-SEG and CVC-ClinicDB), making published Dice scores non-comparable even when they appear in the same leaderboard column. $\mathbf{\text{Third}}$, 26 of 27 papers make $\mathit{\text{performance}}\mathit{\text{claims}}\mathit{\text{without}}\mathit{\text{any}}\mathit{\text{statistical}}\mathit{\text{significance}}\mathit{\text{test}}$. Strikingly, four papers published \emph{after} the Metrics Reloaded framework~\cite{metricsreloaded2024} (Maier-Hein et al., $\mathit{\text{Nature}}\mathit{\text{Methods}}$ 2024) perpetuate these same problems, suggesting that general-purpose metric guidance has not yet reached the colonoscopy sub-community. To show these problems are not merely cosmetic, we re-evaluate five representative models under three controlled protocols with a single uniform scorer, and find that the reported metric conceals large boundary and recall failures, that the ``best’’ model changes with the metric, and that near-tied rankings reverse across random splits. We propose a five-point $\mathbf{\text{Polyp}}\mathbf{\text{Segmentation}}\mathbf{\text{Reporting}}\mathbf{\text{Checklist}}$~(PSRC) as a lightweight, domain-adapted corrective.