DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling

Multimodal reward models (MRMs) play a crucial role in aligning Multimodal Large Language Models (MLLMs) with human preferences. Training a good MRM requires high-quality multimodal preference data. However, existing preference datasets face three key challenges: lack of granularity in preference strength, textual style bias, and unreliable preference signals. Besides, existing open-source multimodal preference datasets suffer from substantial noise, yet there is a lack of effective and scalable curation methods to enhance their quality. To address these limitations, we propose $\mathbf{\text{DT2IT-MRM}}$, which integrates a $\mathbf{\text{D}}$ebiased preference construction pipeline, a novel reformulation of text-to-image ($\mathbf{\text{T2I}}$) preference data, and an $\mathbf{\text{I}}$terative $\mathbf{\text{T}}$raining framework that curates existing multimodal preference datasets for $\mathbf{\text{M}}$ultimodal $\mathbf{\text{R}}$eward $\mathbf{\text{M}}$odeling. Our experimental results show that DT2IT-MRM achieves new $\mathbf{\text{state-of-the-art}}$ overall performance on three major benchmarks: VL-RewardBench, Multimodal RewardBench, and MM-RLHF-RewardBench.