Align and Filter: Improving Performance in Asynchronous On-Policy RL

Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: $\mathit{\text{policy}}\mathit{\text{lag}}$, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose $\mathit{\text{total}}\mathit{\text{Variation-based}}\mathit{\text{Advantage}}\mathit{\text{aligned}}\mathit{\text{Constrained}}\mathit{\text{policy}}\mathit{\text{Optimization}}\mathit{\text{(}}\mathit{\text{methodacronym}}\mathit{\text{)}}$ as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.