Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

Scene-text image captioning requires fusing three information streams – visual features, OCR-detected text, and linguistic knowledge – to generate descriptions that faithfully integrate text visible in images. Existing fusion approaches treat text as language-agnostic, which fails for Vietnamese: a tonal language where diacritics alter word meaning, OCR errors are pervasive, and word boundaries are ambiguous. We argue that Vietnamese scene-text captioning demands $\mathit{\text{linguistically}}\mathit{\text{informed}}\mathit{\text{multimodal}}\mathit{\text{fusion}}$, where language-specific structural knowledge is explicitly incorporated into the fusion mechanism. Motivated from these insights, we propose $\mathbf{\text{HSTFG}}$ (Heterogeneous Scene-Text Fusion Graph), a general-purpose graph fusion framework with learned spatial attention bias, and show through topology analysis that cross-modal graph edges are harmful for scene-text fusion. Building on this finding, we design $\mathbf{\text{PhonoSTFG}}$ (Phonological Scene-Text Fusion Graph) which specializes graph-level fusion for Vietnamese linguistic reasoning. To support evaluation, we introduce $\mathbf{\text{ViTextCaps}}$, the first large-scale Vietnamese scene-text captioning dataset ($\mathbf{\text{15,729}}$ images with $\mathbf{\text{74,970}}$ captions), with comprehensive linguistic analysis showing that 52.8% of the vocabulary is at risk of diacritic collision.