Chunking and reranking
Chunking splits documents into smaller passages so an AI search can find the right one; reranking then reorders the retrieved passages with a more accurate model.
Reranking can only sort what chunking split cleanly.
Chunking prepares manuals, contracts, policies, or wiki pages for AI search. The system cuts each document into passages, usually a few hundred tokens long, computes an embedding for each, and stores it in a vector database. When someone asks a question, it pulls the most similar passages and hands them to a language model that writes the answer; the setup is called retrieval-augmented generation (RAG). Reranking is a second retrieval step in which a more accurate model scores the results again.
The word comes from psychology: in 1956, George A. Miller described how people group single items into larger units, called chunks, and so hold more in short-term memory.
Chunking explains why two RAG systems built on the same documents can answer very differently. Whatever gets torn apart during splitting can no longer be found as a unit, and whatever search misses, the model can't use.
Chapter 1 of 7
Chunking decides what search can find later.
Chunking runs when the search index is built and whenever a source changes. The system loads a document, strips headers and page numbers, splits the text into chunks, and attaches metadata such as document title, chapter, and version date to each one. An embedding model then turns every chunk into a vector that captures its meaning. At question time, the vector database returns the most similar chunks, a reranker reorders them, and the best ones go to the language model with the question.
In the 2020 paper that made RAG widely known, Patrick Lewis and colleagues split every article of a Wikipedia dump into 100-word chunks and ended up with 21 million searchable units. A single embedding for an entire manual would be too coarse: it averages over every topic in the manual and fits no single question well.
Chapter 2 of 7
Four ways of splitting are common.
The simplest method cuts at a fixed length in characters or tokens. Neighboring chunks share a few sentences so an idea doesn't break at the cut; this is called overlap. For Azure AI Search, Microsoft suggests starting with 512 tokens per chunk and 25 percent overlap.
Recursive chunking is gentler. The text splitters in the LangChain library keep whole paragraphs first and fall back to sentences, then words, only when a paragraph is too long.
Structure-based chunking follows the headings, for example in HTML or Markdown. It suits well-organized manuals, where a chapter usually covers one topic.
Semantic chunking splits where the meaning shifts: an embedding model compares consecutive sentences and cuts at a clear change of topic. Many systems combine methods.
Chapter 3 of 7
The right size is a tradeoff you have to measure.
Small chunks match precisely but lose their context. A sentence like “This value applies only to the older model series” doesn't say which value it means on its own. Large chunks keep the context but blend several topics into one embedding and fill the language model's context window faster.
More text in the context window doesn't mean better answers. In their 2023 study “Lost in the Middle,” Nelson Liu and colleagues showed that language models use information at the start and end of a long context better than information in the middle.
No size works best everywhere. The right size comes from testing with a fixed set of real questions where you know which chunk holds the answer. You measure how often search ranks that chunk among the top results.
Chapter 4 of 7
Reranking sorts the results a second time.
First-pass search is fast because the question and the chunks are turned into vectors separately, so at query time the system only compares. This setup is called a bi-encoder. It misses nuance, since the model never sees question and text together.
A reranker, usually a cross-encoder, reads the question and a chunk together and returns a relevance score for each pair. That is more accurate and much slower. The Sentence Transformers documentation therefore describes a combination: a bi-encoder fetches, say, the 100 most similar hits, and a cross-encoder reorders those 100. Rodrigo Nogueira and Kyunghyun Cho showed the gain with BERT in 2019.
Anthropic reported results from its own tests in 2024: adding explanatory context to each chunk, a keyword search (BM25), and reranking cut the failure rate for retrieving the top 20 chunks from 5.7 to 1.9 percent. Reranking can only sort what chunking split cleanly and the first search found.
Chapter 5 of 7
In production, chunks need provenance, access rights, and upkeep.
Every chunk needs its provenance: document, chapter, version, and location. That lets an assistant cite a source for every answer and makes outdated chunks easy to find.
Access rights belong in the index. If a department may not read a document, search must not return its chunks either. Otherwise excerpts from HR files or contracts reach people who could never open the original.
When a source changes, its chunks have to be split and embedded again, and the old ones removed. Costs come from computing embeddings, from storage, and, with reranking, from every query. Smaller chunks aren't automatically cheaper, because more chunks mean more embeddings.
Scanned PDFs without text recognition, tables without column headers, and outdated instructions can't be rescued by even the best splitting.
Chapter 6 of 7
What good implementation looks like.
It starts with a look at the documents. The vendor can say which document types exist and how each one gets split: running text differently from tables, manuals differently from emails.
Tables and lists stay intact, or every chunk carries the column headers. Chunks from the middle of a document know its title and chapter.
There is a set of test questions with known answer locations, and retrieval quality gets measured before launch and after every change. Those tests show more reliably than any spec sheet whether reranking, semantic chunking, or another costlier method pays off.
Answers cite where they come from, so users can check them. And someone is responsible for loading changed documents and for how fast deleted content disappears from the index.
Chapter 7 of 7
Where the term stops working
Chunking improves retrieval, not the source: outdated documents stay outdated, and it reduces hallucinations only by giving the model better evidence. Chunking is one step inside RAG and semantic search. Embeddings are the numeric form of the chunks, the vector database stores them, and tokenization is how the model splits text.
Frequently asked questions
What is chunking?
Chunking is splitting documents into smaller passages so an AI search can find the right spot. Each chunk is stored as its own embedding. In psychology, the term means grouping information into units that are easier to remember.
What is reranking?
Reranking is a second retrieval step that reorders the results of a fast first search with a more accurate model. The reranker reads the question and a chunk together and scores each pair.
What is the best chunk size for RAG?
No chunk size works best everywhere; it depends on the documents and the questions. For Azure AI Search, Microsoft suggests 512 tokens with 25 percent overlap as a starting point. Tests with real questions show whether a size fits.
What is semantic chunking?
Semantic chunking sets boundaries where the meaning of the text shifts instead of cutting at a fixed length. It costs more and pays off when tests show better retrieval.
What is the difference between chunking and tokenization?
Tokenization breaks text into words and word pieces for the language model; chunking breaks documents into passages for search. Tokenization happens on every model call, chunking when the search index is built.
What does overlap mean in chunking?
Overlap is text shared by the end of one chunk and the start of the next. It keeps an idea from falling apart right at the cut. Too much overlap inflates the index and returns duplicate hits.
Do you still need chunking with large context windows?
Yes, in most cases, because even large context windows can't hold an entire document collection and every extra token costs money and time.
Sources
- Miller: The Magical Number Seven, Plus or Minus Two, Psychological Review 63 (2), 1956
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020, arXiv:2005.11401
- Nogueira, Cho: Passage Re-ranking with BERT, arXiv:1901.04085, 2019
- Liu et al.: Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172, 2023
- Microsoft Learn: Chunk large documents for agentic retrieval and vector search in Azure AI Search, as of June 2026
- LangChain: Text splitter integrations, documentation, as of November 2026
- Sentence Transformers: Cross-Encoders, Retrieve & Re-Rank, documentation, as of November 2026
- Anthropic: Introducing Contextual Retrieval, September 19, 2024