Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time

In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me $\mathit{\text{that}}$‘’), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio–visual alignment} required by deictic interaction. To bridge this gap, we introduce $\mathbf{\text{Egocentric}}\mathbf{\text{Co-Speech}}\mathbf{\text{Grounding}}\mathbf{\text{(EcoG)}}$, where grounding is executable only if an agent jointly predicts $\mathit{\text{What}}$, $\mathit{\text{Where}}$, and $\mathit{\text{When}}$. To operationalize this, we present $\mathbf{\text{EcoG-Bench}}$, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of $\mathbf{\text{811}}$ egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a $\mathbf{\text{Progressive}}\mathbf{\text{Cognitive}}\mathbf{\text{Evaluation}}$ protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench ($\mathbf{\text{96.9\%}}$ strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: $\mathbf{\text{17.0\%}}$). Moreover, in a diagnostic ablation, replacing the native video–audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model ($\mathbf{\text{17.0\%}}$$\to$$\mathbf{\text{42.9\%}}$). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech–gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.