SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: $\mathbf{\text{reward}}\mathbf{\text{hacking}}$, enabled by leakage of gold solutions or hidden evaluation information, and $\mathbf{\text{task}}\mathbf{\text{quality}}\mathbf{\text{issues}}$, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents’ true coding ability. We present $\mathbf{\text{SWE-Bench}}\mathbf{\text{Pro}}\mathbf{\text{Verified}}$, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines $\mathbf{\text{anti-hacking}}$ safeguards that eliminate major leakage channels without disrupting normal agent functionality, with $\mathbf{\text{task}}\mathbf{\text{refinement}}$ that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.