Learning to Reason under Off-Policy Guidance

Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~($\mathit{\text{RLVR}}$). However, existing $\mathit{\text{RLVR}}$ approaches are inherently ``on-policy’', limiting learning to a model’s own outputs and failing to acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce $\mathbf{\text{LUFFY}}$ ($\mathbf{\text{L}}$earning to reason $\mathbf{\text{U}}$nder o$\mathbf{\text{FF}}$-polic$\mathbf{\text{Y}}$ guidance), a framework that augments $\mathit{\text{RLVR}}$ with off-policy reasoning traces. LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. Specifically, LUFFY combines the Mixed-Policy GRPO framework, which has a theoretically guaranteed convergence rate, alongside policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. Compared with previous RLVR methods, LUFFY achieves an over $\mathbf{\text{+6.4}}$ average gain across six math benchmarks and an advantage of over $\mathbf{\text{+6.2}}$ points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails. These results provide compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR and demonstrates the great potential of utilizing off-policy guidance in RLVR.