Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation

The recent introduction of $\mathtt{\text{SAM3}}$ has revolutionized Open-Vocabulary Segmentation (OVS) through $\mathit{\text{promptable}}\mathit{\text{concept}}\mathit{\text{segmentation}}$, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift ($\mathit{\text{data}}\mathit{\text{drift}}$) or conditional label distributions evolve ($\mathit{\text{concept}}\mathit{\text{drift}}$) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present $\text{textsc}ConceptBank$, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach ($\mathit{\text{i}}$) anchors target-domain evidence via class-wise visual prototypes, ($\mathit{\text{ii}}$) mines representative supports to suppress outliers under data drift, and ($\mathit{\text{iii}}$) fuses candidate concepts to rectify concept drift. We demonstrate that $\text{textsc}ConceptBank$ effectively adapts $\mathtt{\text{SAM3}}$ to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.