Adversarial Training for Process Reward Models

Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs ($\mathtt{\text{APRM}}$), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$), while $R$ concurrently learns to detect them. This interaction yields progressively harder negatives for $R$, improving its robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, $\mathtt{\text{APRM}}$ improves solver accuracy by $+3.4$ percentage points (pp) over the strongest PRM baseline. $\mathtt{\text{APRM}}$ achieves gains of $+5.3$ pp on out-of-distribution tasks.