In retrieval-augmented generation (RAG), documents are split into chunks, embedded and stored; at question time the most relevant chunks are retrieved and passed to the model. How you split documents — chunking — often affects answer quality more than which model you use. Chunks that are too small lose context; chunks that are too large dilute relevance and waste tokens.

Why chunking matters

Retrieval compares the question with each chunk. A good chunk is self-contained (it makes sense alone), focused (about one thing) and complete (it contains the full answer, not half of it). The model then has to use what you give it, and research such as Lost in the Middle found models use information in the middle of long inputs less reliably — another reason to send fewer, better chunks.

The main strategies

1. Fixed-size chunks. Split every N tokens (for example 300–800) with some overlap (10–20%). Simple and predictable, but can cut sentences, tables and arguments in half.

2. Structural chunking. Split along the document’s own structure: headings, sections, paragraphs, list items, slides, pages. Works very well for well-formatted documentation, policies and manuals.

3. Recursive splitting. Try to split on the largest separator (sections), then smaller ones (paragraphs, sentences) only when a piece is still too long. A practical default.

4. Semantic chunking. Use embeddings to detect where the topic changes and split there. Helpful for long unstructured text, at extra processing cost.

5. Hierarchical (parent–child) chunks. Index small chunks for precise matching, but return the larger parent section to the model for context. Combines precision with completeness.

Special content

  • Tables: keep rows with their headers; consider converting to text sentences or storing as structured data.
  • Code: split by function or class, never in the middle of a block.
  • Q&A and FAQs: keep each question with its answer.
  • PDFs: clean headers, footers and page numbers; watch out for columns and hyphenation from extraction.

Add context to every chunk

Prepend useful metadata to the text before embedding: document title, section heading, date, product or version. A chunk that says “It must be submitted within 30 days” is far more retrievable as “Expense policy › Reimbursements: It must be submitted within 30 days.” Store metadata separately too, for filtering.

How to tune

  1. Build a small evaluation set: 50 real questions with the chunks that answer them.
  2. Try two or three strategies and sizes.
  3. Measure retrieval recall at top 3/5 and answer quality.
  4. Inspect failures: is the answer split across chunks, buried in a large chunk, or missing context?

There is no universal best size; typical starting points are 300–800 tokens with 10–20% overlap, adjusted to your documents.

Hybrid keyword plus vector search, reranking, and query rewriting all improve retrieval further. See how RAG works and how to choose an embedding model.

A worked example

Imagine an employee handbook of 80 pages. Fixed 500-token chunks split the “Parental leave” section across three chunks, so the question “How many weeks of paid leave do new fathers get?” retrieves a chunk with the eligibility rules but not the number of weeks.

Switching to structural chunking by heading, with each chunk prefixed by “Employee handbook › Parental leave › …”, keeps the eligibility rules and the duration together in one chunk. Retrieval recall on a test set of 50 HR questions rises noticeably, and answers stop saying “the document does not specify”.

The lesson generalizes: look at where answers live in your documents, and make sure chunk boundaries do not cut through them.

What to watch next

  • Long-context models changing how much retrieval is needed.
  • Better document parsing for PDFs, tables and slides.
  • Automatic chunking tuned by evaluation results.
  • Graph-based retrieval that links related chunks.

Key takeaways

  • Chunking strongly affects RAG answer quality.
  • Good chunks are self-contained, focused and complete.
  • Recursive and structural chunking are strong defaults; hierarchical chunks add context.
  • Prepend titles and headings to chunks and store metadata for filters.
  • Tune size and overlap with a small evaluation set of real questions.

Frequently asked questions

What chunk size should I use?

Start around 300–800 tokens with 10–20% overlap, then tune using real questions from your users.

Do long-context models make chunking unnecessary?

Not for large knowledge bases: retrieval is still cheaper and more precise than sending everything.

How do I handle tables?

Keep headers with rows, or convert tables into short descriptive sentences or structured records.