Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
Researchers have developed a new method called SPA (Semantic-guided Patch-level Alignment) to improve class-incremental learning using CLIP. This approach leverages local, patch-level features within CLIP's encoders, which were previously overlooked in favor of global image embeddings. SPA uses GPT-5 to generate semantic descriptions that guide the selection of discriminative visual patches, which are then aligned with these descriptions using optimal transport. The method also incorporates task-specific projectors and pseudo-feature calibration to combat catastrophic forgetting, achieving state-of-the-art results in experiments. AI
IMPACT Introduces a novel approach to leverage local features in vision-language models for continuous learning, potentially improving model adaptability.