Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations.

ABSTRACT: Fine-tuning pre-trained vision-language models, like CLIP, has yielded success on diverse downstream tasks. However, several pain points persist for this paradigm: (i) directly tuning entire pre-trained models becomes both time-intensive and computationally costly. Additionally, these tuned models tend to become highly specialized, limiting their practicality for real-world deployment; (ii) recent studies indicate that pre-trained vision-language classifiers may overly depend on spurious features - patterns that correlate with the target in training data, but are not related to the true labeling function; and (iii) existing studies on mitigating the reliance on spurious features, largely based on the assumption that we can identify such features, does not provide definitive assurance for r

SUBMITTER: You C 

PROVIDER: S-EPMC11620289 | biostudies-literature | 2024 Jun

REPOSITORIES: biostudies-literature