Segment Anything, or SAM, is a Meta AI research project that introduced a model able to cut out almost any object in an image from a simple prompt such as a click or a box. The 2023 paper by Alexander Kirillov and colleagues paired the model with SA-1B, a dataset of more than 1 billion masks on 11 million images, then the largest segmentation dataset by far. The idea is to treat segmentation the way language models treat text: train one promptable model on a huge amount of data, then reuse it for many tasks without retraining. The paper was published as an arXiv preprint and presented at ICCV 2023, a peer-reviewed computer vision conference.
What segmentation is, and why it was hard to scale
Image segmentation assigns every pixel to something: a person, a car, a patch of road. Older systems were trained for a fixed list of categories, such as the classes in a particular benchmark. To handle a new kind of object, or a new kind of image such as microscope slides or underwater photos, teams had to collect new labeled data. Drawing precise masks by hand is slow, so labeled data stayed scarce compared with the text that language models learn from.
SAM’s authors set out to change this with three linked pieces: a new task, a model and a dataset, built together.
The task: promptable segmentation
The paper defines “promptable segmentation.” Given any prompt that says what to segment, the model must return a valid mask. A prompt can be a set of points, a box, a rough mask, or, in the paper’s early experiments, free text. The paper stresses that prompts can be ambiguous: a point on a shirt could mean the shirt or the whole person. So the model is designed to return several plausible masks for one prompt, along with a score for each, and to accept that no single answer is always right.
The model: three parts
The Segment Anything Model is split into components that match how people use it.
- Image encoder. A large vision transformer, pre-trained with a masked autoencoder method, turns the image into an embedding. This is the heavy step, but it runs once per image. Our guide to the Vision Transformer explains the architecture behind it.
- Prompt encoder. Points and boxes are converted to positional encodings, and masks are handled with a small convolutional network. Text prompts use an existing text encoder.
- Mask decoder. A lightweight transformer combines the image embedding with the prompt and outputs the masks.
The split matters in practice. Because the expensive image embedding is computed once, the decoder can respond to new prompts very quickly. The authors report that it can run in a web browser in roughly 50 milliseconds, which makes interactive tools possible.
The data engine and SA-1B
No dataset of this size existed, so the team built one using the model itself in a “data engine” with three stages. In the first, human annotators used SAM interactively to label masks. In the second, SAM proposed masks automatically for some objects while annotators filled in the rest. In the third, SAM generated masks fully automatically across the images. The model improved as the data grew, and the data grew faster as the model improved.
The result, SA-1B, contains over 1 billion masks on 11 million licensed, privacy-respecting images, per the abstract. The paper notes that the vast majority of masks were produced automatically, which is why the number is so far above earlier datasets.
Zero-shot results
The headline claim is zero-shot transfer: SAM was evaluated on image distributions and tasks it was not trained on. The authors report that its performance is “often competitive with or even superior to prior fully supervised results” across numerous tasks, such as edge detection and generating proposals for objects. We have not re-run these evaluations, so treat the claim as the authors’ own. SAM also does not know what an object is; it returns a mask without a label, so it is usually combined with a classifier or detector when you need names.
Where SAM goes next
Meta later released SAM 2, which it describes as the first unified model for segmenting objects across images and videos. You can select an object with a click, box or mask on any image or video frame, and the model is built for streaming inference so it can run in real time. For models that connect vision and language, see also our explainer on BLIP-2.
Limitations
Segmentation quality depends on the prompt and the scene. Fine structures, such as thin wires or hair, and very crowded scenes can produce imperfect boundaries. The model returns masks, not meaning. Text prompting was presented as exploratory in the original paper. As with any large model, performance on specialized data such as medical imaging may need extra adaptation, so measure it on your own images before relying on it.
What to watch next
- Further extensions of promptable segmentation to video and 3D.
- Independent evaluations of SAM-style models on specialist data such as medical or satellite images.
- Tools that pair SAM masks with detectors or language models to label objects automatically.
- New datasets built with the same model-in-the-loop approach.
- Updates from Meta on SAM 2 and its successors.
Key takeaways
- SAM turns clicks, boxes or masks into object masks, with several candidates for ambiguous prompts.
- It has three parts: a heavy image encoder run once, a prompt encoder and a light mask decoder.
- SA-1B holds over 1 billion masks on 11 million images, built with a three-stage data engine.
- The paper is peer-reviewed (ICCV 2023) and reports strong zero-shot performance.
- SAM returns masks without labels, and edge cases still need checking.
Frequently asked questions
What does Segment Anything do?
It takes a prompt, such as a click or a box on an image, and returns a mask outlining the object. It works on new images without task-specific training.
What is SA-1B?
SA-1B is the dataset released with SAM. It has over 1 billion masks on 11 million licensed, privacy-respecting images.
Does SAM identify what an object is?
No. SAM outlines objects but does not name them. To get labels you combine it with a classifier, detector or vision-language model.
