DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks

CLIP-style contrastive pre-training aligns whole images with whole captions, which produces excellent global semantics but discards much of the spatial precision required by dense prediction tasks such as segmentation. DetailCLIP augments the contrastive objective with patch-level self-distillation and a pixel reconstruction term, together with an attention-based token-removal mechanism that preserves semantically and spatially significant tokens. The resulting encoder retains CLIP’s semantic transfer while substantially improving performance on fine-grained, detail-sensitive downstream tasks.

Paper Code