CLIP-Aware Image Embeddings
Smart image-text matching that knows when CLIP works and when to use alternatives.
MCP Integrations
| MCP | Purpose |
|---|
| Firecrawl | Research latest CLIP alternatives and benchmarks |
| Hugging Face (if configured) | Access model cards and documentation |
Quick Decision Tree
Your task:
├─ Semantic search ("find beach images") → CLIP ✓
├─ Zero-shot classification (broad categories) → CLIP ✓
├─ Counting objects → DETR, Faster R-CNN ✗
├─ Fine-grained ID (celebrities, car models) → Specialized model ✗
├─ Spatial relations ("cat left of dog") → GQA, SWIG ✗
└─ Compositional ("red car AND blue truck") → DCSMs, PC-CLIP ✗
When to Use This Skill
✅ Use for:
- Semantic image search
- Broad category classification
- Image similarity matching
- Zero-shot tasks on new categories
❌ Do NOT use for:
- Counting objects in images
- Fine-grained classification
- Spatial understanding
- Attribute binding
- Negation handling
Installation
pip install transformers pillow torch sentence-transformers --break-system-packages
Validation: Run python scripts/validate_setup.py
Basic Usage
Image Search
from transformers import CLIPProcessor, CLIPModel
from PIL import Image
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
# Embed images
images = [Image.open(f"img{i}.jpg") for i in range(10)]
inputs = processor(images=images, return_tensors="pt")
image_features = model.get_image_features(**inputs)
# Search with text
text_inputs = processor(text=["a beach at sunset"], return_tensors="pt")
text_features = model.get_text_features(**text_inputs)
# Compute similarity
similarity = (image_features @ text_features.T).softmax(dim=0)
Common Anti-Patterns
Anti-Pattern 1: "CLIP for Everything"
❌ Wrong:
# Using CLIP to count cars in an image
prompt = "How many cars are in this image?"
# CLIP cannot count - it will give nonsense results
Why wrong: CLIP's architecture collapses spatial information into a single vector. It literally cannot count.
✓ Right:
from transformers import DetrImageProcessor, DetrForObjectDetection
processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
# Detect objects
results = model(**processor(images=image, return_tensors="pt"))
# Filter for cars and count
car_detections = [d for d in results if d['label'] == 'car']
count = len(car_detections)
How to detect: If query contains "how many", "count", or numeric questions → Use object detection
Anti-Pattern 2: Fine-Grained Classification
❌ Wrong:
# Trying to identify specific celebrities with CLIP
prompts = ["Tom Hanks", "Brad Pitt", "Morgan Freeman"]
# CLIP will perform poorly - not trained for fine-grained face ID
Why wrong: CLIP trained on coarse categories. Fine-grained faces, car models, flower species require specialized models.
✓ Right:
# Use a fine-tuned face recognition model
from transformers import AutoFeatureExtractor, AutoModelForImageClassification
model = AutoModelForImageClassification.from_pretrained(
"microsoft/resnet-50" # Then fine-tune on celebrity dataset
)
# Or use dedicated face recognition: ArcFace, CosFace
How to detect: If query asks to distinguish between similar items in same category → Use specialized model
Anti-Pattern 3: Spatial Understanding
❌ Wrong:
# CLIP cannot understand spatial relationships
prompts = [
"cat to the left of dog",
"cat to the right of dog"
]
# Will give nearly identical scores
Why wrong: CLIP embeddings lose spatial topology. "Left" and "right" are treated as bag-of-words.
✓ Right:
# Use a spatial reasoning model
# Examples: GQA models, Visual Genome models, SWIG
from swig_model import SpatialRelationModel
model = SpatialRelationModel()
result = model.predict_relation(image, "cat", "dog")
# Returns: "left", "right", "above", "below", etc.
How to detect: If query contains directional words (left, right, above, under, next to) → Use spatial model
Anti-Pattern 4: Attribute Binding
❌ Wrong:
prompts = [
"red car and blue truck",
"blue car and red truck"
]
# CLIP often gives similar scores for both
Why wrong: CLIP cannot bind attributes to objects. It sees "red, blue, car, truck" as a bag of concepts.
✓ Right - Use PC-CLIP or DCSMs:
# PC-CLIP: Fine-tuned for pairwise comparisons
from pc_clip import PCCLIPModel
model = PCCLIPModel.from_pretrained("pc-clip-vit-l")
# Or use DCSMs (Dense Cosine Similarity Maps)
How to detect: If query has multiple objects with different attributes → Use compositional model
Evolution Timeline
2021: CLIP Released
- Revolutionary: zero-shot, 400M image-text pairs
- Widely adopted for everything
- Limitations not yet understood
2022-2023: Limitations Discovered
- Cannot count objects
- Poor at fine-grained classification
- Fails spatial reasoning
- Can't bind attributes
2024: Alternatives Emerge
- DCSMs: Preserve patch/token topology
- PC-CLIP: Trained on pairwise comparisons
- SpLiCE: Sparse interpretable embeddings
2025: Current Best Practices
- Use CLIP for what it's good at
- Task-specific models for limitations
- Compositional models for complex queries
LLM Mistake: LLMs trained on 2021-2023 data will suggest CLIP for everything because limitations weren't widely known. This skill corrects that.
Validation Script
Before using CLIP, check if it's appropriate:
python scripts/validate_clip_usage.py \
--query "your query here" \
--check-all
Returns:
- ✅ CLIP is appropriate
- ❌ Use alternative (with suggestion)
Task-Specific Guidance
Image Search (CLIP ✓)
# Good use of CLIP
queries = ["beach", "mountain", "city skyline"]
# Works well for broad semantic concepts
Zero-Shot Classification (CLIP ✓)
# Good: Broad categories
categories = ["indoor", "outdoor", "nature", "urban"]
# CLIP excels at this
Object Counting (CLIP ✗)
# Use object detection instead
from transformers import DetrImageProcessor, DetrForObjectDetection
# See /references/object_detection.md
Fine-Grained Classification (CLIP ✗)
# Use specialized models
# See /references/fine_grained_models.md
Spatial Reasoning (CLIP ✗)
# Use spatial relation models
# See /references/spatial_models.md
Troubleshooting
Issue: CLIP gives unexpected results
Check:
- Is this a counting task? → Use object detection
- Fine-grained classification? → Use specialized model
- Spatial query? → Use spatial model
- Multiple objects with attributes? → Use compositional model
Validation:
python scripts/diagnose_clip_issue.py --image path/to/image --query "your query"
Issue: Low similarity scores
Possible causes:
- Query too specific (CLIP works better with broad concepts)
- Fine-grained task (not CLIP's strength)
- Need to adjust threshold
Solution: Try broader query or use alternative model
Model Selection Guide
| Model | Best For | Avoid For |
|---|
| CLIP ViT-L/14 | Semantic search, broad categories | Counting, fine-grained, spatial |
| DETR | Object detection, counting | Semantic similarity |
| DINOv2 | Fine-grained features | Text-image matching |
| PC-CLIP | Attribute binding, comparisons | General embedding |
| DCSMs | Compositional reasoning | Simple similarity |
Performance Notes
CLIP models:
- ViT-B/32: Fast, lower quality
- ViT-L/14: Balanced (recommended)
- ViT-g-14: Highest quality, slower
Inference time (single image, CPU):
- ViT-B/32: ~100ms
- ViT-L/14: ~300ms
- ViT-g-14: ~1000ms
Further Reading
/references/clip_limitations.md - Detailed analysis of CLIP's failures/references/alternatives.md - When to use what model/references/compositional_reasoning.md - DCSMs and PC-CLIP deep dive/scripts/validate_clip_usage.py - Pre-flight validation tool/scripts/diagnose_clip_issue.py - Debug unexpected results
See CHANGELOG.md for version history.