Segment Anything Model: Zero-Shot Image Segmentation with SAM
SAM: zero-shot image segmentation via points, boxes, masks.
Written by Neura Market from the official Hermes Agent documentation for Segment Anything Model. Commands, paths, and version numbers are reproduced from the source unchanged.
Read the official documentationMeta AI's Segment Anything Model (SAM) is a zero-shot image segmentation tool that can isolate any object in an image without task-specific training. You prompt it with points, boxes, or masks, and it returns high-quality segmentation masks. This guide is for practitioners building annotation tools, generating training data, or processing domain-specific images like medical scans or satellite imagery. It walks through installation, checkpoint selection, interactive prompting, automatic mask generation, batched inference, ONNX deployment, and common workflows, with exact code for each step.
What it does
SAM turns a prompt into a segmentation mask. The prompt can be a foreground point (click on an object), a background point (click outside an object to exclude it), a bounding box, or a previous mask for iterative refinement. It works on any image domain without fine-tuning, which makes it useful when you do not have labeled data for a specific task. The model was trained on 1.1 billion masks from 11 million images, so it generalizes broadly. You can choose among three model sizes: ViT-B (fastest), ViT-L (medium), and ViT-H (most accurate). The architecture separates image encoding from prompt encoding: the image encoder runs once per image, and the prompt encoder runs per prompt, which makes interactive use efficient.
Before you start
This skill is optional and installed on demand. It is available on Linux, macOS, and Windows. The source lists no additional permissions or platform-specific requirements beyond the Python packages and model checkpoints. You will need a Python environment with pip, and for GPU acceleration, a CUDA-compatible device. The source code examples assume you have the segment-anything package installed and a checkpoint downloaded. For the HuggingFace path, you need the transformers library. The ONNX deployment path requires onnxruntime and the export script from the repository.
Installation
Install the package from GitHub, along with optional dependencies for image handling and COCO mask utilities. Alternatively, use the HuggingFace transformers integration.
# From GitHub
pip install git+https://github.com/facebookresearch/segment-anything.git
# Optional dependencies
pip install opencv-python pycocotools matplotlib
# Or use HuggingFace transformers
pip install transformers
Download checkpoints
Download the checkpoint for the model size you plan to use. The ViT-H checkpoint is 2.4 GB, ViT-L is 1.2 GB, and ViT-B is 375 MB. Choose based on your accuracy and speed needs.
# ViT-H (largest, most accurate) - 2.4GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
# ViT-L (medium) - 1.2GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth
# ViT-B (smallest, fastest) - 375MB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth
Basic usage with SamPredictor
The SamPredictor class handles image embedding and prompt-based prediction. You load a model, create a predictor, set the image (which computes embeddings once), and then predict with point prompts. The multimask_output=True option returns three mask candidates, and you can select the best one by score.
import numpy as np
from segment_anything import sam_model_registry, SamPredictor
# Load model
sam = sam_model_registry["vit_h"](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops/models/segment-anything-model/checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
# Create predictor
predictor = SamPredictor(sam)
# Set image (computes embeddings once)
image = cv2.imread("image.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predictor.set_image(image)
# Predict with point prompts
input_point = np.array([[500, 375]]) # (x, y) coordinates
input_label = np.array([1]) # 1 = foreground, 0 = background
masks, scores, logits = predictor.predict(
point_coords=input_point,
point_labels=input_label,
multimask_output=True # Returns 3 mask options
)
# Select best mask
best_mask = masks[np.argmax(scores)]
HuggingFace Transformers
If you prefer the transformers ecosystem, use SamModel and SamProcessor. The processor handles image preprocessing and post-processing of masks back to original size.
import torch
from PIL import Image
from transformers import SamModel, SamProcessor
# Load model and processor
model = SamModel.from_pretrained("facebook/sam-vit-huge")
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")
model.to("cuda")
# Process image with point prompt
image = Image.open("image.jpg")
input_points = [[[450, 600]]] # Batch of points
inputs = processor(image, input_points=input_points, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
# Generate masks
with torch.no_grad():
outputs = model(**inputs)
# Post-process masks to original size
masks = processor.image_processor.post_process_masks(
outputs.pred_masks.cpu(),
inputs["original_sizes"].cpu(),
inputs["reshaped_input_sizes"].cpu()
)
Core concepts
Model architecture
SAM has three components: an image encoder (ViT) that produces image embeddings once per image, a prompt encoder that handles points and boxes per prompt, and a mask decoder that outputs masks and IoU predictions.
SAM Architecture:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Image Encoder │────▶│ Prompt Encoder │────▶│ Mask Decoder │
│ (ViT) │ │ (Points/Boxes) │ │ (Transformer) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
Image Embeddings Prompt Embeddings Masks + IoU
(computed once) (per prompt) predictions
Model variants
| Model | Checkpoint | Size | Speed | Accuracy |
|---|---|---|---|---|
| ViT-H | vit_h | 2.4 GB | Slowest | Best |
| ViT-L | vit_l | 1.2 GB | Medium | Good |
| ViT-B | vit_b | 375 MB | Fastest | Good |
Prompt types
| Prompt | Description | Use Case |
|---|---|---|
| Point (foreground) | Click on object | Single object selection |
| Point (background) | Click outside object | Exclude regions |
| Bounding box | Rectangle around object | Larger objects |
| Previous mask | Low-res mask input | Iterative refinement |
Interactive segmentation
Point prompts
You can provide one or multiple points. Foreground points (label 1) indicate the object, background points (label 0) indicate what to exclude. When prompts are clear, set multimask_output=False to get a single mask.
# Single foreground point
input_point = np.array([[500, 375]])
input_label = np.array([1])
masks, scores, logits = predictor.predict(
point_coords=input_point,
point_labels=input_label,
multimask_output=True
)
# Multiple points (foreground + background)
input_points = np.array([[500, 375], [600, 400], [450, 300]])
input_labels = np.array([1, 1, 0]) # 2 foreground, 1 background
masks, scores, logits = predictor.predict(
point_coords=input_points,
point_labels=input_labels,
multimask_output=False # Single mask when prompts are clear
)
Box prompts
A bounding box prompt is a single array [x1, y1, x2, y2]. This is often enough for larger objects.
# Bounding box [x1, y1, x2, y2]
input_box = np.array([425, 600, 700, 875])
masks, scores, logits = predictor.predict(
box=input_box,
multimask_output=False
)
Combined prompts
Combine a box with points for precise control. The box narrows the region, and points refine the selection.
# Box + points for precise control
masks, scores, logits = predictor.predict(
point_coords=np.array([[500, 375]]),
point_labels=np.array([1]),
box=np.array([400, 300, 700, 600]),
multimask_output=False
)
Iterative refinement
Start with a point prompt and multimask_output=True to get several candidates. Then feed the best mask back as mask_input along with an additional background point to refine the result.
# Initial prediction
masks, scores, logits = predictor.predict(
point_coords=np.array([[500, 375]]),
point_labels=np.array([1]),
multimask_output=True
)
# Refine with additional point using previous mask
masks, scores, logits = predictor.predict(
point_coords=np.array([[500, 375], [550, 400]]),
point_labels=np.array([1, 0]), # Add background point
mask_input=logits[np.argmax(scores)][None, :, :], # Use best mask
multimask_output=False
)
Automatic mask generation
Basic automatic segmentation
SamAutomaticMaskGenerator segments everything in an image without prompts. It returns a list of masks, each with metadata like bounding box, area, and quality scores.
from segment_anything import SamAutomaticMaskGenerator
# Create generator
mask_generator = SamAutomaticMaskGenerator(sam)
# Generate all masks
masks = mask_generator.generate(image)
# Each mask contains:
# - segmentation: binary mask
# - bbox: [x, y, w, h]
# - area: pixel count
# - predicted_iou: quality score
# - stability_score: robustness score
# - point_coords: generating point
Customized generation
You can tune the generator. points_per_side controls the grid density (more points means more masks), pred_iou_thresh and stability_score_thresh filter low-quality masks, crop_n_layers enables multi-scale crops, and min_mask_region_area removes tiny masks.
mask_generator = SamAutomaticMaskGenerator(
model=sam,
points_per_side=32, # Grid density (more = more masks)
pred_iou_thresh=0.88, # Quality threshold
stability_score_thresh=0.95, # Stability threshold
crop_n_layers=1, # Multi-scale crops
crop_n_points_downscale_factor=2,
min_mask_region_area=100, # Remove tiny masks
)
masks = mask_generator.generate(image)
Filtering masks
After generation, you can sort or filter masks by area, predicted IoU, or stability score.
# Sort by area (largest first)
masks = sorted(masks, key=lambda x: x['area'], reverse=True)
# Filter by predicted IoU
high_quality = [m for m in masks if m['predicted_iou'] > 0.9]
# Filter by stability score
stable_masks = [m for m in masks if m['stability_score'] > 0.95]
Batched inference
Multiple images
Process multiple images in a loop. For each image, set the image and predict. This is straightforward but re-encodes each image.
# Process multiple images efficiently
images = [cv2.imread(f"image_{i}.jpg") for i in range(10)]
all_masks = []
for image in images:
predictor.set_image(image)
masks, _, _ = predictor.predict(
point_coords=np.array([[500, 375]]),
point_labels=np.array([1]),
multimask_output=True
)
all_masks.append(masks)
Multiple prompts per image
For multiple prompts on the same image, set the image once and reuse the embeddings. This is much faster than re-encoding.
# Process multiple prompts efficiently (one image encoding)
predictor.set_image(image)
# Batch of point prompts
points = [
np.array([[100, 100]]),
np.array([[200, 200]]),
np.array([[300, 300]])
]
all_masks = []
for point in points:
masks, scores, _ = predictor.predict(
point_coords=point,
point_labels=np.array([1]),
multimask_output=True
)
all_masks.append(masks[np.argmax(scores)])
ONNX deployment
Export model
Export the model to ONNX for deployment in browsers or edge devices. The export script is in the repository. Use --return-single-mask for faster inference.
python scripts/export_onnx_model.py \
--checkpoint sam_vit_h_4b8939.pth \
--model-type vit_h \
--output sam_onnx.onnx \
--return-single-mask
Use ONNX model
Load the ONNX model with onnxruntime. The image embeddings are computed separately (typically with the PyTorch model), and the ONNX session runs the prompt encoder and mask decoder.
import onnxruntime
# Load ONNX model
ort_session = onnxruntime.InferenceSession("sam_onnx.onnx")
# Run inference (image embeddings computed separately)
masks = ort_session.run(
None,
{
"image_embeddings": image_embeddings,
"point_coords": point_coords,
"point_labels": point_labels,
"mask_input": np.zeros((1, 1, 256, 256), dtype=np.float32),
"has_mask_input": np.array([0], dtype=np.float32),
"orig_im_size": np.array([h, w], dtype=np.float32)
}
)
Common workflows
Workflow 1: Annotation tool
Build an interactive annotation tool with OpenCV. On a left click, predict a mask for the clicked point and display the best one.
import cv2
# Load model
predictor = SamPredictor(sam)
predictor.set_image(image)
def on_click(event, x, y, flags, param):
if event == cv2.EVENT_LBUTTONDOWN:
# Foreground point
masks, scores, _ = predictor.predict(
point_coords=np.array([[x, y]]),
point_labels=np.array([1]),
multimask_output=True
)
# Display best mask
display_mask(masks[np.argmax(scores)])
Workflow 2: Object extraction
Extract an object at a point with a transparent background. The function returns an RGBA image where the alpha channel is the mask.
def extract_object(image, point):
"""Extract object at point with transparent background."""
predictor.set_image(image)
masks, scores, _ = predictor.predict(
point_coords=np.array([point]),
point_labels=np.array([1]),
multimask_output=True
)
best_mask = masks[np.argmax(scores)]
# Create RGBA output
rgba = np.zeros((image.shape[0], image.shape[1], 4), dtype=np.uint8)
rgba[:, :, :3] = image
rgba[:, :, 3] = best_mask * 255
return rgba
Workflow 3: Medical image segmentation
Medical images are often grayscale. Convert to RGB before passing to SAM, then use a box prompt to define the region of interest.
# Process medical images (grayscale to RGB)
medical_image = cv2.imread("scan.png", cv2.IMREAD_GRAYSCALE)
rgb_image = cv2.cvtColor(medical_image, cv2.COLOR_GRAY2RGB)
predictor.set_image(rgb_image)
# Segment region of interest
masks, scores, _ = predictor.predict(
box=np.array([x1, y1, x2, y2]), # ROI bounding box
multimask_output=True
)
Output format
Mask data structure
Each mask from SamAutomaticMaskGenerator is a dictionary with the following keys:
# SamAutomaticMaskGenerator output
{
"segmentation": np.ndarray, # H×W binary mask
"bbox": [x, y, w, h], # Bounding box
"area": int, # Pixel count
"predicted_iou": float, # 0-1 quality score
"stability_score": float, # 0-1 robustness score
"crop_box": [x, y, w, h], # Generation crop region
"point_coords": [[x, y]], # Input point
}
COCO RLE format
For interoperability with COCO tools, encode masks to RLE using pycocotools.
from pycocotools import mask as mask_utils
# Encode mask to RLE
rle = mask_utils.encode(np.asfortranarray(mask.astype(np.uint8)))
rle["counts"] = rle["counts"].decode("utf-8")
# Decode RLE to mask
decoded_mask = mask_utils.decode(rle)
Performance optimization
GPU memory
If you run out of memory, switch to a smaller model like ViT-B. Clear the CUDA cache between large batches.
# Use smaller model for limited VRAM
sam = sam_model_registry["vit_b"](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops/models/segment-anything-model/checkpoint="sam_vit_b_01ec64.pth")
# Process images in batches
# Clear CUDA cache between large batches
torch.cuda.empty_cache()
Speed optimization
Use half precision to speed up inference. Reduce points_per_side for automatic generation. For deployment, export to ONNX with --return-single-mask.
# Use half precision
sam = sam.half()
# Reduce points for automatic generation
mask_generator = SamAutomaticMaskGenerator(
model=sam,
points_per_side=16, # Default is 32
)
# Use ONNX for deployment
# Export with --return-single-mask for faster inference
Common issues
| Issue | Solution |
|---|---|
| Out of memory | Use ViT-B model, reduce image size |
| Slow inference | Use ViT-B, reduce points_per_side |
| Poor mask quality | Try different prompts, use box + points |
| Edge artifacts | Use stability_score filtering |
| Small objects missed | Increase points_per_side |
When not to use it
The source explicitly lists alternatives for specific tasks. If you need real-time object detection with class labels, use YOLO or Detectron2. For semantic or panoptic segmentation with categories, use Mask2Former. For text-prompted segmentation, combine GroundingDINO with SAM. For video segmentation, use SAM 2. SAM is not a classifier; it segments objects but does not tell you what they are.
Limits and gotchas
The source's caveats are practical. The model is large, especially ViT-H at 2.4 GB, so memory can be a constraint. Inference speed varies by model size; ViT-B is fastest but less accurate. Mask quality depends on prompt clarity; ambiguous prompts may produce poor masks. Edge artifacts can appear, and small objects may be missed, but you can mitigate these with the settings in the Common issues table.
References and further reading
The skill ships with two reference documents: Advanced Usage covers batching, fine-tuning, and integration, and Troubleshooting addresses common issues. The upstream resources include the GitHub repository, the research paper, an online demo, the SAM 2 repository for video, and the HuggingFace model page.
-
Advanced Usage - Batching, fine-tuning, integration
-
Troubleshooting - Common issues and solutions
-
GitHub: https://github.com/facebookresearch/segment-anything
-
SAM 2 (Video): https://github.com/facebookresearch/segment-anything-2
-
HuggingFace: https://huggingface.co/facebook/sam-vit-huge