Perceptual Intelligence Lab

Segment Anything

Introduction

overview

Current State of Computer Vision: Existing Vision Foundation Models (e.g., CLIP) focus heavily on “Image-Text” alignment but lack unified models and abundant training data for pixel-level Segmentation.

In this work, our goal is to build a foundation model for image segmentation. That is, we seek to develop a promptable model and pre-train it on a broad dataset using a task that enables powerful generalization. With this model, we aim to solve a range of downstream segmentation problems on new data distributions using prompt engineering. The success of this plan hinges on three components: task, model, and data. To develop them, we address the following questions about image segmentation:

  1. What task will enable zero-shot generalization?
  2. What is the corresponding model architecture?
  3. What data can power this task and model?

zero-shot : Zero-shot means the ability of an AI to complete a task it has never been explicitly trained to do and for which it has seen zero examples.

Segment Anything Task

Given any segmentation prompt, the model must return a valid segmentation mask. The prompt can be points, boxes or masks or text. A valid mask means even if a prompt is ambiguous (e.g., a point on a shirt can be the shirt or the person that is wearning it), the output should be a reasonable mask for at least one of those objects.

task

Pre-training & Zero-shot Transfer Pre-training Method: Simulates a sequence of prompts (random points, boxes) for each training image to predict the Ground Truth mask.

Zero-shot Application: During inference, different prompt designs can solve various downstream tasks (e.g., edge detection, object counting) without the need for task-specific retraining.

Segment Anything Model

Architecture Overview model Design Goals: Support flexible prompts, achieve Amortized Real-time performance (the heavy image encoding happens once, while interactive prompting must be <50ms), and possess Ambiguity Awareness.

Three Components:

  1. Image Encoder: Heavyweight; processes the image once. vit

  2. Prompt Encoder: Lightweight; understands user clicks, boxes, or text.

  3. Mask Decoder: Ultra-lightweight; combines the encoder outputs to generate the mask.

A. Image Encoder

  • Function: Converts the input image into an Image Embedding.
  • Architecture: A ViT (Vision Transformer) pre-trained using MAE (Masked Autoencoder). Specifically uses the ViT-H/16 configuration.
  • Input/Output: Resizes any input to $1024 \times 1024 \times 3$; outputs a $64 \times 64 \times 256$ embedding.
  • Feature: Computationally expensive, but only runs once per image.

B. Prompt Encoder

Sparse Prompts:

  1. Points & Boxes: Uses Positional Encodings summed with learned embeddings (no convolution).
  2. Text: Uses the CLIP Text Encoder.

Dense Prompts (Masks): Processed via convolutions and combined with the Image Embedding using element-wise summation.

C. Mask Decoder

maskencoder

Architecture: A modified Transformer Decoder.

Process:

  • Self-Attention: Prompts “talk” to each other.
  • Cross-Attention (Token-to-Image): Prompts act as Queries to “find” targets in the Image Embedding.
  • Cross-Attention (Image-to-Prompt): Image features act as Queries to update Prompt features with spatial context.

Output & Mask Prediction:

  • Upscaling: The $64 \times 64$ embedding is upscaled to $256 \times 256$.
  • MLP Mapping: The Output Token is converted into a vector acting as a Dynamic Linear Classifier.
  • Dot Product Prediction: The MLP vector (representing “what we are looking for”) is multiplied by the upscaled image features (representing “pixel-level textures”) to determine the final mask.

Resolving Ambiguity

Problem: A single point might refer to multiple valid objects (whole, part, or sub-part).

Solution: The model predicts 3 Masks simultaneously (Whole, Part, Sub-part) along with their IoU scores (confidence). IoU

IoU: In most benchmarks, a prediction is considered “correct” (a True Positive) if its IoU is greater than a certain threshold.(0.5-0.7)

During training, loss is only backpropagated for the mask with the minimum loss.

sam Data Engine & Dataset

  • Data Engine sam uses a “Model-in-the-loop” workflow to build the dataset:
  1. Assisted-manual stage: Human annotators use a version of sam trained on public data. sam suggests masks, and humans refine them.
  2. Semi-automatic stage: To increase diversity, sam automatically labels confident objects, and humans are asked to label the “missing” (less prominent or harder) objects.
  3. Fully automatic stage: Once the model is powerful enough, it is prompted with a $32 \times 32$ grid of points to predict all possible masks. Low-quality or duplicate masks are filtered via IoU scores and NMS.

NMS is an algorithm. It uses that IoU measurement to make a decision: “Because these two boxes overlap by 80%, I will delete the one with the lower score.”

  • Scale: 11 Million images and 1.1 Billion masks (400x more masks than any existing dataset).

  • Quality: High-resolution images; masks are of such high quality they are comparable to (or better than) professional human annotations.

Zero-Shot Transfer Experiments

sam acts as a true Foundation Model: no fine-tuning is required for specific tasks.

Key Tasks Tested: Single Point Segmentation: Outperforms specialized models in human evaluation using just one point.

  • Edge Detection: By prompting with a grid of points, sam produces high-quality edge maps, proving it inherently “understands” boundaries.

  • Object Proposals: Excels at finding rare categories that it was never explicitly trained to recognize.

  • Instance Segmentation: Uses an external detector (e.g., ViTDet) to provide boxes. sam generates masks with smoother, more physically accurate edges than the detector’s native mask head.

  • Text-to-Mask: Though not explicitly trained on text, by using CLIP’s text embeddings, sam can segment objects based on text prompts like “a wheel.”