Neura Market Logo
OptionalMLOpsVersion 1.0.0

CLIP Skill: Zero-Shot Image Classification and Search with Hermes Agent

Zero-shot image classification and image-text search.

Written by Neura Market from the official Hermes Agent documentation for Clip. Commands, paths, and version numbers are reproduced from the source unchanged.

Read the official documentation

OpenAI's CLIP model lets you classify images and search them by natural language without any training data. You give it an image and a list of labels, and it returns the most likely match. This skill is useful when you need to sort images by content, find photos from a description, or flag unsafe material, and you do not have a labeled dataset or the time to train a custom model.

What it does

CLIP learns a shared embedding space for images and text. It encodes both into vectors and measures how close they are. In practice this means you can:

  • Classify an image into categories you define on the fly.
  • Compute how similar an image and a sentence are.
  • Search a collection of images using a text query.
  • Detect content like violence or NSFW material.
  • Answer simple visual questions by treating each answer as a label.
  • Retrieve images from text or text from images.

The model was trained on 400 million image-text pairs and matches ResNet-50 on ImageNet without ever seeing ImageNet training data. It is released under the MIT license.

Before you start

This is an optional skill. Install it on demand inside your Hermes Agent environment. The skill runs on Linux, macOS, and Windows. You need Python with PyTorch installed.

Installation

pip install git+https://github.com/openai/CLIP.git
pip install torch torchvision ftfy regex tqdm

Zero-shot classification

The core workflow: load a model, preprocess an image, tokenize your labels, and compute similarity scores.

import torch
import clip
from PIL import Image

# Load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

# Load image
image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)

# Define possible labels
text = clip.tokenize(["a dog", "a cat", "a bird", "a car"]).to(device)

# Compute similarity
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

    # Cosine similarity
    logits_per_image, logits_per_text = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

# Print results
labels = ["a dog", "a cat", "a bird", "a car"]
for label, prob in zip(labels, probs[0]):
    print(f"{label}: {prob:.2%}")

The output shows a probability for each label. The highest one is the model's best guess. Because this is zero-shot, you can change the labels to anything without retraining.

Available models

CLIP ships several architectures. The Vision Transformer variants are generally recommended.

# Models (sorted by size)
models = [
    "RN50",           # ResNet-50
    "RN101",          # ResNet-101
    "ViT-B/32",       # Vision Transformer (recommended)
    "ViT-B/16",       # Better quality, slower
    "ViT-L/14",       # Best quality, slowest
]

model, preprocess = clip.load("ViT-B/32")
ModelParametersSpeedQuality
RN50102MFastGood
ViT-B/32151MMediumBetter
ViT-L/14428MSlowBest

ViT-B/32 is the default for a reason: it balances inference speed and accuracy for most tasks. Switch to ViT-B/16 when you need better quality and can accept slower runs. ViT-L/14 is for offline batch jobs where quality matters most.

Image-text similarity

To get a raw similarity score between one image and one text string, encode both and compute the cosine similarity.

# Compute embeddings
image_features = model.encode_image(image)
text_features = model.encode_text(text)

# Normalize
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)

# Cosine similarity
similarity = (image_features @ text_features.T).item()
print(f"Similarity: {similarity:.4f}")

A score close to 1 means the image and text are semantically aligned. A score near 0 or negative means they are unrelated.

Semantic image search

Build a search index over a set of images by precomputing their embeddings, then query with text.

# Index images
image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
image_embeddings = []

for img_path in image_paths:
    image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
    with torch.no_grad():
        embedding = model.encode_image(image)
        embedding /= embedding.norm(dim=-1, keepdim=True)
    image_embeddings.append(embedding)

image_embeddings = torch.cat(image_embeddings)

# Search with text query
query = "a sunset over the ocean"
text_input = clip.tokenize([query]).to(device)
with torch.no_grad():
    text_embedding = model.encode_text(text_input)
    text_embedding /= text_embedding.norm(dim=-1, keepdim=True)

# Find most similar images
similarities = (text_embedding @ image_embeddings.T).squeeze(0)
top_k = similarities.topk(3)

for idx, score in zip(top_k.indices, top_k.values):
    print(f"{image_paths[idx]}: {score:.3f}")

The loop normalises each embedding as it goes. This is required for cosine similarity to work correctly.

Content moderation

Define categories that describe content policies and classify an image against them.

# Define categories
categories = [
    "safe for work",
    "not safe for work",
    "violent content",
    "graphic content"
]

text = clip.tokenize(categories).to(device)

# Check image
with torch.no_grad():
    logits_per_image, _ = model(image, text)
    probs = logits_per_image.softmax(dim=-1)

# Get classification
max_idx = probs.argmax().item()
max_prob = probs[0, max_idx].item()

print(f"Category: {categories[max_idx]} ({max_prob:.2%})")

The model returns the category with the highest probability. You can adjust the category list to match your moderation rules.

Batch processing

Processing images and texts one at a time is slow. Stack them into batches for GPU efficiency.

# Process multiple images
images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]
images = torch.stack(images).to(device)

with torch.no_grad():
    image_features = model.encode_image(images)
    image_features /= image_features.norm(dim=-1, keepdim=True)

# Batch text
texts = ["a dog", "a cat", "a bird"]
text_tokens = clip.tokenize(texts).to(device)

with torch.no_grad():
    text_features = model.encode_text(text_tokens)
    text_features /= text_features.norm(dim=-1, keepdim=True)

# Similarity matrix (10 images × 3 texts)
similarities = image_features @ text_features.T
print(similarities.shape)  # (10, 3)

The result is a matrix where each row is an image and each column is a text label. You can find the best match per image with argmax.

Integration with vector databases

For large collections, store precomputed embeddings in a vector database and query with text embeddings.

# Store CLIP embeddings in Chroma/FAISS
import chromadb

client = chromadb.Client()
collection = client.create_collection("image_embeddings")

# Add image embeddings
for img_path, embedding in zip(image_paths, image_embeddings):
    collection.add(
        embeddings=[embedding.cpu().numpy().tolist()],
        metadatas=[{"path": img_path}],
        ids=[img_path]
    )

# Query with text
query = "a sunset"
text_embedding = model.encode_text(clip.tokenize([query]))
results = collection.query(
    query_embeddings=[text_embedding.cpu().numpy().tolist()],
    n_results=5
)

This pattern scales to millions of images. The database handles the nearest-neighbour search; CLIP provides the embeddings.

Best practices

  1. Use ViT-B/32 for most cases - Good balance
  2. Normalize embeddings - Required for cosine similarity
  3. Batch processing - More efficient
  4. Cache embeddings - Expensive to recompute
  5. Use descriptive labels - Better zero-shot performance
  6. GPU recommended - 10-50× faster
  7. Preprocess images - Use provided preprocess function

Performance

OperationCPUGPU (V100)
Image encoding~200ms~20ms
Text encoding~50ms~5ms
Similarity compute<1ms<1ms

These numbers are per item. Batch processing on GPU brings the per-item cost down further.

When not to use it

CLIP is not a general vision model. Use alternatives for specific tasks:

  • BLIP-2: Better captioning
  • LLaVA: Vision-language chat
  • Segment Anything: Image segmentation

Limits and gotchas

  1. Not for fine-grained tasks - Best for broad categories
  2. Requires descriptive text - Vague labels perform poorly
  3. Biased on web data - May have dataset biases
  4. No bounding boxes - Whole image only
  5. Limited spatial understanding - Position/counting weak

Because CLIP sees the whole image at once, it cannot tell you where an object is or how many there are. It also inherits biases from its training data, so test on your own domain before relying on it in production.

Related resources

More MLOps skills

Huggingface AccelerateRun PyTorch training across GPUs with minimal changes.ChromaEmbedding database for RAG and semantic search.Evaluating Llms Harnesslm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).Weights And BiasesW&B: log ML experiments, sweeps, model registry, dashboards.FaissFast vector similarity search at billion scale.Optimizing Attention FlashSpeed up long-sequence transformer training and inference.