Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models

Vision-Language Models (VLMs) demonstrate impressive zero-shot generalization through large-scale image-text pretraining, yet their performance can drop once the deployment distribution diverges from the training distribution. To address this, Test-Time Adaptation (TTA) methods update models using unlabeled target data. However, existing approaches often ignore two key challenges: prototype degradation in long-tailed distributions and confusion between semantically similar classes. To tackle these issues, we propose $\mathbf{\text{C}}$lass-Aware $\mathbf{\text{P}}$rototype $\mathbf{\text{L}}$earning with $\mathbf{\text{N}}$egative $\mathbf{\text{C}}$ontrast($\mathbf{\text{CPL-NC}}$), a lightweight TTA framework designed specifically for VLMs to enhance generalization under distribution shifts. CPL-NC introduces a $\mathit{\text{Class-Aware}}\mathit{\text{Prototype}}\mathit{\text{Cache}}$ Module that dynamically adjusts per-class capacity based on test-time frequency and activation history, with a rejuvenation mechanism for inactive classes to retain rare-category knowledge. Additionally, a $\mathit{\text{Negative}}\mathit{\text{Contrastive}}\mathit{\text{Learning}}$ Mechanism identifies and constrains hard visual-textual negatives to improve class separability. The framework employs asymmetric optimization, refining only textual prototypes while anchoring on stable visual features. Experiments on 15 benchmarks show that CPL-NC consistently outperforms prior TTA methods across both ResNet-50 and ViT-B/16 backbones.