EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuning

In complex embodied long-horizon manipulation tasks, effective task decomposition and execution require synergistic integration of textual logical reasoning and visual-spatial imagination to ensure efficient and accurate operation. Current methods fail to adopt a unified generation framework for multimodal planning, lead to inconsistent in multimodal planning. To address this challenge, we present $\mathbf{\text{EVLP}}\mathbf{\text{(Embodied}}\mathbf{\text{Vision-Language}}\mathbf{\text{Planner)}}$, an innovative multimodal unified generation framework that jointly models linguistic reasoning and visual generation. Our approach achieves multimodal planning for long-horizon tasks through a novel training pipeline incorporating dynamic pretraining and reinforced alignment. Our core innovations consist of three key components: $\mathbf{\text{1)}}\mathbf{\text{Unified}}\mathbf{\text{Multimodal}}\mathbf{\text{Generation}}\mathbf{\text{Framework}}$: For understanding, We integrate semantic information with spatial features to provide comprehensive visual perception. For generation, we directly learn the joint distribution of discrete images for one-step visual synthesis, enabling coordinated language-visual modeling through learnable cross-modal attention mechanisms. $\mathbf{\text{2)}}\mathbf{\text{Dynamic}}\mathbf{\text{Perception}}\mathbf{\text{Pretraining}}$: We propose a bidirectional dynamic alignment strategy employing inverse dynamics tasks and forward dynamics tasks, effectively strengthening multimodal correlations within a unified feature space. $\mathbf{\text{3)}}\mathbf{\text{Reinforced}}\mathbf{\text{Supervised}}\mathbf{\text{Fine-Tuning}}$: While conducting instruction-based fine-tuning in the unified generation space, we construct a reinforce loss to align the spatial logic between textual actions and generated images, enabling the model to acquire spatio-awared multimodal planning capabilities.