Vector databases became one of the hottest infrastructure categories of the generative AI boom. Their job is simple to state: store embeddings — lists of numbers representing the meaning of text, images or other data — and quickly find the ones most similar to a query. They power semantic search, retrieval-augmented generation, recommendations and deduplication.

How similarity search works

An embedding model turns each item into a vector, often with hundreds or thousands of dimensions. Items with similar meaning end up close together. To answer a query, you embed it and look for its nearest neighbors, using a distance or similarity measure such as cosine similarity or dot product.

Comparing the query with every stored vector (exact search) is fine for thousands of items but too slow for millions. Vector databases therefore use approximate nearest neighbor (ANN) indexes that trade a little accuracy for large speed gains.

Common index types

  • HNSW (Hierarchical Navigable Small World), described by Malkov and Yashunin, builds a layered graph that can be navigated quickly toward close neighbors. It is fast and accurate but memory-hungry, and it is the default in many systems.
  • IVF (inverted file) clusters vectors and searches only the most promising clusters.
  • Quantization compresses vectors to reduce memory, at some cost in precision.
  • Disk-based indexes keep most data on SSD for very large collections.
  • Metadata filtering: combine similarity with filters such as user, date or category — essential for permissions.
  • Hybrid search: combine vector and keyword (BM25) search.
  • Updates and deletes: how quickly new data becomes searchable.
  • Multi-tenancy: isolating data for different customers.
  • Scaling and operations: replication, backups, managed hosting.

The main options

Option Type Good fit
pgvector PostgreSQL extension Teams already using Postgres; moderate scale; joins with relational data
Pinecone Managed service Teams wanting zero operations and fast setup
Weaviate Open source + cloud Hybrid search, built-in modules, schema-based data
Qdrant Open source + cloud Performance, rich filtering, self-hosting in Rust
Milvus Open source + cloud (Zilliz) Very large-scale deployments
Chroma Open source Prototypes and local development

Search engines such as Elasticsearch and OpenSearch, and many general databases, have also added vector search.

How to choose

  1. Start with what you have. If you run PostgreSQL, pgvector may be enough for millions of vectors.
  2. Estimate scale: number of vectors, dimensions, queries per second and update frequency.
  3. List must-have features: filtering, hybrid search, multi-tenancy, data residency.
  4. Decide on operations: managed service or self-hosted.
  5. Benchmark with your data for recall and latency at your scale.

Common mistakes to avoid

  • Choosing by hype instead of scale. Many projects never exceed a few million vectors; a simpler option may suffice.
  • Ignoring recall. Approximate search can miss relevant items; measure recall on your data when tuning index parameters.
  • Mixing embedding models. Vectors from different models are not comparable; re-embed everything when you switch.
  • Forgetting metadata. Store source, date and permissions alongside vectors from the start.

A quick sizing example

Suppose you index 2 million document chunks with an embedding model that produces 1,536-dimensional vectors stored as 32-bit floats.

  • Each vector takes 1,536 × 4 bytes ≈ 6 KB.
  • Two million vectors need about 12 GB for the raw vectors alone.
  • An HNSW index adds graph links on top, and you also store text and metadata.

That fits comfortably on a single well-sized server or a mid-tier managed plan. Quantizing vectors to 8 bits would cut raw vector memory roughly by four, at a small cost in recall. Running this kind of back-of-the-envelope calculation early prevents both overspending on infrastructure and surprise memory limits in production.

Key takeaways

  • Vector databases store embeddings and find nearest neighbors quickly using approximate indexes.
  • HNSW graphs are the most common index, trading memory for speed and accuracy.
  • Filtering, hybrid search and updates matter as much as raw search speed.
  • pgvector suits existing Postgres users; managed services reduce operations; specialized engines handle large scale.
  • Benchmark recall and latency with your own data before committing.

Frequently asked questions

Do I need a vector database for RAG?

Not always. Small projects can use an in-memory index or a vector extension in an existing database. Dedicated systems help with scale and advanced features.

What does “dimensions” mean?

The length of each embedding vector. More dimensions can capture more nuance but use more storage and memory.

Is cosine similarity or dot product better?

Use the measure recommended for your embedding model. For normalized vectors, the two give equivalent rankings.