Beyond Closed-Pool Video Retrieval: A Benchmark and Agent Framework for Real-World Video Search and Moment Localization

Traditional video retrieval benchmarks focus on matching precise descriptions to closed video pools, failing to reflect real-world searches characterized by fuzzy, multi-dimensional memories on the open web. We present $\mathbf{\text{RVMS-Bench}}$, a comprehensive system for evaluating real-world video memory search. It consists of $\mathbf{\text{1,440}}\mathbf{\text{samples}}$ spanning $\mathbf{\text{20}}\mathbf{\text{diverse}}\mathbf{\text{categories}}$ and $\mathbf{\text{four}}\mathbf{\text{duration}}\mathbf{\text{groups}}$, sourced from $\mathbf{\text{real-world}}\mathbf{\text{open-web}}\mathbf{\text{videos}}$. RVMS-Bench utilizes a hierarchical description framework encompassing $\mathbf{\text{Global}}\mathbf{\text{Impression,}}\mathbf{\text{Key}}\mathbf{\text{Moment,}}\mathbf{\text{Temporal}}\mathbf{\text{Context,}}\mathbf{\text{and}}\mathbf{\text{Auditory}}\mathbf{\text{Memory}}$ to mimic realistic multi-dimensional search cues, with all samples strictly verified via a human-in-the-loop protocol. We further propose $\mathbf{\text{RACLO}}$, an agentic framework that employs abductive reasoning to simulate the human ``Recall-Search-Verify’’ cognitive process, effectively addressing the challenge of searching for videos via fuzzy memories in the real world. Experiments reveal that existing MLLMs still demonstrate insufficient capabilities in real-world Video Retrieval and Moment Localization based on fuzzy memories. We believe this work will facilitate the advancement of video retrieval robustness in real-world unstructured scenarios.