SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin

Recently, enhancing the numerical and logical reasoning capability of Large Language Models (LLMs) has emerged as a research hotspot. Existing methods face several limitations: inference-phase techniques (e.g., Chain of Thoughts) rely on prompt selection and the pretrained knowledge; sentence-level Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) struggle with step-wise mathematical correctness and depend on stronger models distillation or human annotations; while Reinforcement Learning (RL) approaches incur high GPU memory costs and unstable training. To address these, we propose $\mathbf{\text{S}}$elf-training framework integrating $\mathbf{\text{P}}$rocess $\mathbf{\text{P}}$reference learning using $\mathbf{\text{D}}$ynamic value margin (SPPD). SPPD leverages a process-based Markov Decision Process (MDP) and Bellman optimality equation to derive $\mathbf{\text{dynamic}}\mathbf{\text{value}}\mathbf{\text{margin}}$ on step-level preference optimization, which employs tree-based self-sampling on model responses $\mathbf{\text{without}}\mathbf{\text{any}}\mathbf{\text{distillation}}$ from other models. Furthermore, we theoretically prove that SPPD is $\mathbf{\text{equivalent}}\mathbf{\text{to}}\mathbf{\text{on-policy}}\mathbf{\text{policy}}\mathbf{\text{gradient}}\mathbf{\text{methods}}$ under reward constraints. Experiments on 7B-scale models demonstrate superior performance across in-domain and out-domain mathematical benchmarks. We open-source our code at \href{https://anonymous.4open.science/r/SSDPO-D-DCDD}{https://anonymous.4open.science/r/SPPD-DCDD}.