Overview

CLIP, introduced by OpenAI in January 2021, learned from about 400 million image-caption pairs to match images with text. The ViT-L/14 version, a Vision Transformer, became one of the most widely used image encoders, appearing in text-to-image generators, multimodal models and image search systems. Read our CLIP explainer.

Highlights

  • Shared embedding space for images and text
  • Zero-shot image classification from text labels
  • Widely used as a vision encoder in other models
  • MIT license
  • Open reproductions such as OpenCLIP extend it

Typical uses

Image search by text, zero-shot classification, content filtering and as a component of multimodal systems.

Things to consider

  • Inherits social biases from internet training data.
  • Weak at counting and fine spatial reasoning.
  • Newer image encoders outperform it on many benchmarks.

Specifications are taken from the provider’s official announcement, documentation or model card. Benchmark results depend on evaluation settings; see how AI benchmarks work.