XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution

Diffusion-based methods, endowed with a formidable generative prior, have received increasing attention in Image Super-Resolution (ISR) recently. However, as low-resolution (LR) images often undergo severe degradation, it is challenging for ISR models to perceive the semantic and degradation information, resulting in restoration images with incorrect content or unrealistic artifacts. To address these issues, we propose a $\mathit{\text{Cross-modal}}\mathit{\text{Priors}}\mathit{\text{for}}\mathit{\text{Super-Resolution}}\mathit{\text{(XPSR)}}$ framework. Within XPSR, to acquire precise and comprehensive semantic conditions for the diffusion model, cutting-edge Multimodal Large Language Models (MLLMs) are utilized. To facilitate better fusion of cross-modal priors, a $\mathit{\text{Semantic-Fusion}}\mathit{\text{Attention}}$ is raised. To distill semantic-preserved information instead of undesired degradations, a $\mathit{\text{Degradation-Free}}\mathit{\text{Constraint}}$ is attached between LR and its high-resolution (HR) counterpart. Quantitative and qualitative results show that XPSR is capable of generating high-fidelity and high-realism images across synthetic and real-world datasets. Codes are released at \url{https://github.com/qyp2000/XPSR}.