OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

Recent advances in omni-modal large language models have enabled remarkable progress in joint vision-audio understanding. However, prevailing architectures rely on modality-specific encoders with a \emph{video-coarse, audio-dense} design – sampling visual frames at 1–2 fps while processing audio waveforms at 25 fps – resulting in systems that perceive video \emph{frame by frame, modality by modality} rather than holistically as humans do. Such a discrepancy leaves models with impoverished cross-modal interaction during encoding and an inability to capture fine-grained visual motion. To bridge this gap, we present $\mathbf{\text{Omni-Encoder,}}\mathbf{\text{a}}\mathbf{\text{unified}}\mathbf{\text{Transformer}}\mathbf{\text{backbone}}\mathbf{\text{designed}}\mathbf{\text{to}}\mathbf{\text{co-embed}}\mathbf{\text{visual}}\mathbf{\text{and}}\mathbf{\text{audio}}\mathbf{\text{signals}}\mathbf{\text{at}}\mathbf{\text{a}}\mathbf{\text{symmetrical}}\mathbf{\text{25}}\mathbf{\text{fps}}$ within a shared latent space. This architecture leverages three core innovations – the Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting – to effectively reconcile the dual challenges of modality disentanglement and computational efficiency. Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5-Omni under the same input token budget to the LLM decoder, Omni-Encoder delivers substantial gains on visual continuous understanding tasks – such as sign language recognition and fine-grained sports action analysis – while maintaining competitive performance on established audio-visual benchmarks such as AVQA and Speaker Identification and Localization. These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception.