Learning Dynamics of VLM Finetuning

Preference-based finetuning of vision–language models (VLMs) is brittle: trivially wrong negatives inject uninformative gradients that destabilize training. We recast alignment as $\mathbf{\text{learning-dynamics–aware}}\mathbf{\text{optimization}}$ and introduce $\mathbf{\text{Cooling-Weighted}}\mathbf{\text{DPO}}\mathbf{\text{(CW-DPO)}}$, a two-stage recipe that explicitly models and exploits the training trajectory. $\mathbf{\text{Stage}}\mathbf{\text{1}}$ performs supervised finetuning with $\mathbf{\text{gentle}}\mathbf{\text{negatives}}$: $\mathbf{\text{low-weight}}\mathbf{\text{smoothed}}\mathbf{\text{supervision}}$ that regularizes the base policy and curbs overconfidence without explicit penalties. $\mathbf{\text{Stage}}\mathbf{\text{2}}$ applies a DPO objective in which the $\mathbf{\text{negative}}\mathbf{\text{term}}\mathbf{\text{is}}\mathbf{\text{scaled}}\mathbf{\text{by}}\mathbf{\text{a}}\mathbf{\text{cooling}}\mathbf{\text{weight}}$ computed from the model’s $\mathbf{\text{average}}\mathbf{\text{token}}\mathbf{\text{log-probability}}$ on each negative, suppressing uninformative gradients from easy or off-distribution samples while preserving signal from hard negatives. In practice, we emphasize $\mathbf{\text{on-policy}}\mathbf{\text{negatives}}$ and allow $\mathbf{\text{mixed}}\mathbf{\text{negatives}}$ by blending a controllable fraction of dataset negatives to maintain contrast freshness. Throughout, we instrument training with $\mathrm{\Delta }\text{\hspace{0.17em}⁣}\log p$ probes on positives and negatives as first-class signals for early stopping, curriculum design, and failure diagnosis. Across diverse VLM tasks, CW-DPO yields $\mathbf{\text{more}}\mathbf{\text{stable}}\mathbf{\text{optimization}}$, $\mathbf{\text{better}}\mathbf{\text{calibration}}$, and $\mathbf{\text{higher}}\mathbf{\text{pairwise}}\mathbf{\text{win-rates}}$ than SFT-only and vanilla DPO, while $\mathbf{\text{converging}}\mathbf{\text{in}}\mathbf{\text{fewer}}\mathbf{\text{steps}}$. Ablations isolate the $\mathbf{\text{cooling-weight}}\mathbf{\text{mechanism}}$ as the primary driver of these gains and show complementary benefits from mixing on-policy and dataset negatives. Taken together, our results show that $\mathbf{\text{smoothing}}\mathbf{\text{learning}}\mathbf{\text{dynamics}}\mathbf{\text{before}}\mathbf{\text{cooling}}\mathbf{\text{preferences}}$ is a simple, general principle for robust VLM alignment.