TriAlignGR: Triangular Multitask Alignment with Multimodal Deep Interest Mining for Generative Recommendation

We introduce TriAlignGR, a unified multitask-multimodal framework for generative recommendation that establishes two-stage multimodal semantic propagation: (i) encoding visual semantics directly into SIDs via multimodal embeddings, and (ii) enabling the model to decode these semantics through visual description tasks. Existing Semantic ID (SID) pipelines suffer from two fundamental but underexplored problems: $\mathbf{\text{SID}}\mathbf{\text{Content}}\mathbf{\text{Degradation}}\mathbf{\text{(SCD)}}$, where cascaded encoding and residual quantization discard critical multimodal and interest-level semantics; and $\mathbf{\text{SID}}\mathbf{\text{Semantic}}\mathbf{\text{Opacity}}\mathbf{\text{(SSO)}}$, where models autoregressively generate SID sequences without truly comprehending their underlying meaning, leading to hallucination and poor generalization. Prior work addresses at most text-SID alignment, leaving visual semantics and latent user interests entirely unexploited. TriAlignGR resolves both problems through three tightly integrated components: (1)~$\mathbf{\text{Cross-Modal}}\mathbf{\text{Semantic}}\mathbf{\text{Alignment}}\mathbf{\text{(CMSA)}}$ integrates visual content into SID construction through both VLM-generated textual descriptions and a multimodal embedding model that directly encodes image features alongside text, ensuring that SIDs inherently carry multimodal semantics; (2)~$\mathbf{\text{Multimodal}}\mathbf{\text{Deep}}\mathbf{\text{Interest}}\mathbf{\text{Mining}}\mathbf{\text{(MDIM)}}$ leverages LLM Chain-of-Thought reasoning to extract latent user intents (\eg ``productivity-focused lifestyle’’ from noise-canceling headphones) beyond surface attributes, enriching SID semantics before discretization; and (3)~$\mathbf{\text{Triangular}}\mathbf{\text{Multitask}}\mathbf{\text{(TMT)}}$ jointly trains on eight complementary generation tasks under a single autoregressive loss – including two novel visual-semantic tasks (VisDesc$\to$SID, VisDesc$\to$Title) that map VLM-generated image descriptions to SIDs and titles, completing the SID-Text-Image triangle – without requiring task-specific towers or complex loss weighting.