Why Chunking Matters in RAG Pipelines: Beyond Splitting Documents

In retrieval-augmented generation (RAG), many wrong answers that look like model errors trace back to retrieval, and many retrieval errors trace back to how the documents were chunked. The generator can only reason over what it is shown. The retriever can only return what was indexed. The unit that gets indexed, embedded, scored and pasted into the prompt is the chunk, and that unit is chosen before any model sees a query.

Chunking is decided offline, once, before any query is known, and it sets the resolution at which the whole system sees the corpus. If a contract clause is split from the definition it depends on, no reranker will put them back together. If a chunk mixes three topics, its embedding is an average of three meanings and matches none of them well. This article looks at chunking as a lossy compression of documents into retrievable units, with trade-offs that can be measured.

Why documents have to be split at all

There are three separate constraints, and they push in different directions.

1. The embedding model has a hard input limit. Many widely used embedding models are BERT-style encoders with a 512-token window. Text beyond the window is truncated, silently, so a 3,000-token section embedded whole is represented only by its first 512 tokens. Long-context embedding models (8K tokens and more) exist, but a bigger window does not remove the next problem.

2. A single vector has limited capacity. A typical encoder produces a vector by pooling token representations, for example a mean over the final hidden states:

$$\mathbf{e}(c) = \frac{1}{|c|}\sum_{t \in c} \mathbf{h}_t$$

If a chunk $c$ covers several topics, $\mathbf{e}(c)$ lands somewhere between them. Retrieval scores a chunk with cosine similarity,

$$s(q, c) = \frac{\mathbf{e}(q)^\top \mathbf{e}(c)}{\lVert \mathbf{e}(q)\rVert\,\lVert \mathbf{e}(c)\rVert},$$

and a query about one of those topics now competes with chunks that are only about that topic. The mixed chunk loses even though it contains the answer. This is the dilution effect. It is why “just embed the whole document” fails even with long-context encoders: the vector stops being specific to anything.

3. The generator has a context budget, and it uses it unevenly. Even with a 128K-token window, every retrieved token costs latency and money, and adds distractors. Models also use long contexts unevenly. Liu et al.’s “Lost in the Middle” (2023) showed accuracy on multi-document QA dropping when the relevant passage sat in the middle of a long context rather than at the start or end. Retrieving fewer, more precise chunks is usually better than retrieving more, larger ones.

So chunking balances three things: chunks small enough to embed specifically, large enough to stay self-contained, and few enough to fit the prompt well.

Tokens, not characters

Chunk sizes are often configured in characters, but every constraint above is measured in tokens, and the ratio varies a lot. English prose is roughly 4 characters per token with common BPE tokenizers. Code, tables, URLs and non-Latin scripts can be far denser in tokens. A “1,000-character” chunk might be 250 tokens of prose or 600 tokens of JSON.

Two practical consequences:

• Measure with the embedding model’s tokenizer when enforcing the embedding limit, and with the generator’s tokenizer when budgeting the prompt. They are often different tokenizers.
• Budget the prompt explicitly. If the generator has a window $W$, a system prompt of $S$ tokens, a query of $Q$ tokens and a reserved answer length $A$, then retrieving $k$ chunks of size $L$ requires

$$S + Q + k\,L + A \le W.$$

In practice the binding constraint is not $W$ but quality: past a certain $kL$, extra context hurts more than it helps.

The main chunking strategies

Fixed-size chunking

Split every $L$ tokens, with an overlap of $O$ tokens between neighbours. It is fast, predictable and easy to reason about. Its weakness is that it ignores meaning: boundaries fall mid-sentence, mid-table and mid-argument.

Overlap is the standard patch. If an answer spans a boundary, an overlap of $O$ tokens means any span of up to $O$ tokens is guaranteed to appear whole in at least one chunk. The cost is index growth. With stride $L - O$, a document of $N$ tokens produces about

$$n_{\text{chunks}} \approx \frac{N - O}{L - O}$$

chunks, so the index is inflated by a factor of roughly $L/(L-O)$. A 20% overlap costs about 25% more vectors, more embedding compute, and more near-duplicate results that waste slots in the top-$k$.

Recursive chunking

Recursive splitters try a list of separators from coarsest to finest, for example paragraph breaks, then line breaks, then sentence ends, then spaces, and only fall back to a finer separator when a piece is still too large. Then they merge small neighbours back up to the target size. The result respects natural boundaries most of the time while still enforcing a hard maximum.

def recursive_split(text, max_tokens, count_tokens,
                    seps=("\n\n", "\n", ". ", " ")):
    """Split on the coarsest separator that works, then merge up to max_tokens."""
    if count_tokens(text) <= max_tokens:
        return [text]
    if not seps:                      # no separator left: hard cut by tokens
        return hard_split(text, max_tokens)
    sep, finer = seps[0], seps[1:]
    pieces = []
    for part in text.split(sep):
        pieces.extend(recursive_split(part, max_tokens, count_tokens, finer))

    chunks, current = [], ""
    for piece in pieces:              # greedily merge small pieces
        candidate = f"{current}{sep}{piece}" if current else piece
        if count_tokens(candidate) <= max_tokens:
            current = candidate
        else:
            chunks.append(current)
            current = piece
    if current:
        chunks.append(current)
    return chunks

This is the sensible default for unstructured prose. Its limitation is that separators are a proxy for topic boundaries, not a measurement of them.

Semantic chunking

Semantic chunking measures topic shifts directly. Embed each sentence, compute the similarity between adjacent sentences (or between sliding windows of sentences), and cut where the similarity drops sharply:

$$d_i = 1 - \cos\big(\mathbf{e}(s_i), \mathbf{e}(s_{i+1})\big), \qquad \text{cut after } s_i \text{ if } d_i > \tau,$$

with $\tau$ often set to a high percentile of the $d_i$ in the document, so the method adapts to how cohesive each document is.

It produces chunks that are topically coherent, which helps embedding specificity. But it has real costs and failure modes:

• Cost. Every sentence is embedded at indexing time, often several times over with windowing.
• Noisy signal. Sentence embeddings are unstable for short sentences, so thresholds based on single sentences are jumpy. Windowed comparisons help.
• Variable sizes. A long, cohesive section becomes a huge chunk unless a max-size fallback is added, which brings back the fixed-size problems.
• Coherence is not completeness. A topically pure chunk can still depend on a definition three sections earlier.

In published comparisons, semantic chunking is not consistently better than a well-tuned recursive splitter. It is a tool for corpora with long, unstructured text and frequent topic shifts, such as transcripts and meeting notes, not a universal upgrade.

Structure-aware chunking

Most real documents already declare their structure: Markdown and HTML headings, PDF layout, code syntax, table rows, legal clause numbering. Structure-aware chunking uses it.

• Headings. Split by section, and prepend the heading path to each chunk (“Billing > Refunds > Partial refunds”). This gives an isolated chunk its missing context for free.
• Tables. Never split a table mid-row. Keep the header row with every chunk of a large table, or convert each row to a sentence that repeats the column names.
• Code. Split on function or class boundaries using a parser (for example tree-sitter), not on blank lines. Keep imports and the class signature with each method.
• PDFs. Layout extraction (columns, headers, footers, captions) matters more than the splitter. Repeated page headers and footers inside chunks are a common, invisible source of noise.

For structured corpora such as documentation, contracts, codebases and knowledge bases, structure-aware chunking is usually the biggest single improvement available. The document’s author has already done the semantic segmentation.

Figure 1. The same document under three strategies. Fixed-size cuts ignore structure, recursive splitting respects paragraphs, and structure-aware splitting follows the author’s sections.

Chunk size as a precision–recall trade-off

Chunk size is best understood through what retrieval metrics measure. For a query $q$ with a set of relevant chunks $R_q$ and a retrieved top-$k$ set $T_k$:

$$\text{Recall@}k = \frac{|R_q \cap T_k|}{|R_q|}, \qquad \text{Precision@}k = \frac{|R_q \cap T_k|}{k}.$$

These change meaning as chunk size changes, which is why chunk-level metrics are misleading across configurations (more on that below). The underlying effects are:

	Small chunks (≈100–200 tokens)	Large chunks (≈800–1,500 tokens)
Embedding specificity	High: one idea per vector	Low: diluted across topics
Self-containment	Low: missing referents, “it”, “this clause”	High
Token precision of context	High	Low: many irrelevant tokens per hit
Chance the full answer is in one chunk	Low for multi-sentence answers	High
Index size and embedding cost	High	Low
Best for	Factoid lookup, keyword-like queries	Explanatory, multi-step, summarization queries

There is no universally right size. A common sweet spot for prose with modern embedders is 256–512 tokens with 10–20% overlap, but that is a starting point to measure from, not a rule. The right size depends on the query distribution: “What is the refund window?” wants a small chunk, “How does the refund policy treat partial shipments?” wants a larger one.

This tension, where small chunks are best for matching and large chunks are best for reading, motivates the most useful pattern in chunking.

Decoupling what you match from what you read

Parent-child (small-to-big) retrieval

Index small child chunks for matching, but return their larger parent to the generator.

1. Split each document into parents (for example a section or 1,000–2,000 tokens).
2. Split each parent into children (for example 100–300 tokens), and store a parent_id on each.
3. At query time, retrieve the top children, map them to parents, deduplicate, and send the parents to the LLM.

def retrieve_parents(query, child_index, parents, k_children=20, k_parents=4):
    hits = child_index.search(query, k=k_children)       # small, specific vectors
    scores = {}
    for hit in hits:                                      # aggregate by parent
        pid = hit.metadata["parent_id"]
        scores[pid] = max(scores.get(pid, 0.0), hit.score)
    best = sorted(scores, key=scores.get, reverse=True)[:k_parents]
    return [parents[pid] for pid in best]                # large, readable context

Aggregating by max rewards the single best match. Summing or counting child hits per parent instead rewards parents that match in several places, which suits broad questions. A variant, sentence-window retrieval, indexes single sentences and returns the sentence plus $\pm w$ neighbours.

The trade-off is context cost. Each hit now brings in a large block, so $k$ must drop, and a parent can contain a lot of irrelevant text around the matching child.

Figure 2. Parent-child retrieval. Small child chunks give precise matches; their enclosing parent sections give the generator enough context to answer.

Contextual retrieval

The deeper problem with small chunks is that they lose their referents. A chunk that says “Revenue grew 3% over the previous quarter” is useless if it does not say which company and which quarter. Anthropic’s contextual retrieval technique (2024) fixes this at indexing time: for each chunk, an LLM is given the whole document and the chunk, and writes a short (roughly 50–100 token) context that situates the chunk. That context is prepended before embedding and before BM25 indexing.

CONTEXT_PROMPT = """<document>
{document}
</document>
Here is a chunk from the document:
<chunk>
{chunk}
</chunk>
Write one or two sentences that situate this chunk within the overall document,
to improve search retrieval of the chunk. Answer only with the context."""

def contextualize(document, chunks, llm):
    out = []
    for chunk in chunks:
        context = llm(CONTEXT_PROMPT.format(document=document, chunk=chunk))
        out.append(f"{context}\n\n{chunk}")
    return out

Anthropic reported that contextual embeddings cut the top-20 retrieval failure rate by 35%, adding contextual BM25 brought that to 49%, and adding a reranker brought it to 67%, on their evaluation sets. The idea transfers: retrieval should not depend on information that was cut away by the splitter.

The cost is one LLM call per chunk over the full document. Prompt caching makes this much cheaper, because the document prefix is identical for every chunk of that document, but on large corpora it is still the most expensive indexing step. It also needs to be re-run when documents change.

Late chunking

A different approach to the same problem is late chunking (Jina AI, 2024). Instead of embedding each chunk in isolation, run a long-context embedding model over the whole document once, then mean-pool the token embeddings within each chunk’s span. Every token representation has already attended to the full document, so “it” in chunk 7 carries information about what “it” refers to. It needs a long-context embedder and access to token-level outputs, but avoids the LLM cost of contextual retrieval.

Proposition-level indexing

At the other extreme, Chen et al. (“Dense X Retrieval”, 2023) used an LLM to rewrite passages into atomic, self-contained propositions (“The Leaning Tower of Pisa leans about 4 degrees”), and found that indexing at that granularity improved retrieval for many queries. It gives very precise vectors and very short context, at the cost of an LLM pass over the corpus and a risk of the rewriting step introducing errors.

Chunking and generation quality

Retrieval metrics are a proxy. What matters is the answer, and chunking affects generation in ways retrieval metrics do not see:

• Truncated evidence produces confident partial answers. If a list of five conditions is split across two chunks and only one is retrieved, the model will often answer as if the three conditions it sees are the complete list.
• Orphaned references produce hallucinated referents. A chunk that says “the limit above does not apply” without the limit invites the model to invent one.
• Redundancy wastes the budget. Heavy overlap and near-duplicate chunks fill the top-$k$ with the same content, crowding out the second piece of evidence a multi-hop question needs.
• Noise lowers faithfulness. Large chunks carry distractors, and models do get distracted by plausible but irrelevant text in context.
• Order and position matter. Given the “lost in the middle” effect, put the most relevant chunks first and keep $k$ modest. Reranking helps here as much as it helps precision.

Chunking also interacts with hybrid search. BM25 is less hurt by long chunks than dense retrieval is, because term matching does not average meaning away. But BM25 cannot recover from missing referents either. Combining BM25 and dense retrieval with reciprocal rank fusion, then reranking with a cross-encoder, is robust to a wider range of chunk sizes than either retriever alone.

Evaluating a chunking configuration

Chunking choices should be tested, not argued about. The difficulty is that the obvious metric, chunk-level recall@k, is not comparable across configurations. With large chunks, “the right chunk” is easy to hit, because it contains a lot. With small chunks, the answer may be spread over three “right” chunks. Recall@5 of 0.9 with 1,500-token chunks and recall@5 of 0.7 with 200-token chunks do not mean the first is better.

A better protocol evaluates at the level of the evidence itself:

1. Build a labelled set. Collect real queries if possible, and mark the span of source text that answers each one (character offsets in the original document), independent of any chunking. Synthetic questions generated by an LLM from sampled passages are a reasonable supplement, but they are biased toward questions that a single passage answers and toward the passage’s own wording.

2. Measure token-level recall and precision of the retrieved context. For a query with evidence tokens $E$ and retrieved tokens $C$:

$$\text{recall}_{\text{tok}} = \frac{|E \cap C|}{|E|}, \qquad \text{precision}_{\text{tok}} = \frac{|E \cap C|}{|C|}.$$

These are comparable across chunk sizes, because they count tokens rather than chunks. Recall says whether the generator could answer, and precision says how much noise it has to read to do so. Chroma’s 2024 chunking evaluation used this kind of token-level metric (with an IoU variant) for exactly this reason.

def token_metrics(evidence_spans, retrieved_spans):
    """Spans are (doc_id, start, end) character offsets in the source documents."""
    def to_set(spans):
        return {(d, i) for d, s, e in spans for i in range(s, e)}
    E, C = to_set(evidence_spans), to_set(retrieved_spans)
    inter = len(E & C)
    recall = inter / len(E) if E else 0.0
    precision = inter / len(C) if C else 0.0
    return recall, precision

(Character offsets are a simple stand-in for tokens. Mapping both to tokenizer offsets gives exact token counts.)

3. Hold the context budget fixed. Compare configurations at the same number of retrieved tokens, not the same $k$. Otherwise large chunks win recall simply by showing the model more text.

4. Add ranking-aware metrics. MRR and nDCG capture whether the first useful result appears early, which matters because of position effects in generation.

5. Evaluate end to end on a subset. For a few hundred queries, generate answers and grade them for correctness and faithfulness to the retrieved context (with human review, or an LLM judge spot-checked by humans). This catches the failure modes above that retrieval metrics miss.

6. Slice the results. Average scores hide the interesting part. Break results down by query type (factoid vs. explanatory vs. multi-hop), by document type (prose, tables, code) and by document length. Chunking that is best for one slice is often worst for another, and the right answer is sometimes different chunking per document type.

Mistakes to avoid

• Assuming long context windows remove the need to chunk. A bigger window raises the limit on how much can be sent, not the quality of what is sent. Retrieval still needs specific units to match, and long prompts still cost latency.
• Treating semantic chunking as an automatic upgrade. It improves topic coherence, not necessarily retrieval quality, and it costs more. Structure-aware or recursive splitting often matches it.
• Adding overlap for safety. Overlap protects against boundary splits but fills the top-$k$ with near-duplicates and grows the index. Beyond 10–20% it rarely pays off.
• Using one chunk size for every document type. API references, legal text, chat logs and tables have different natural units.
• Treating chunking as preprocessing. It sets the resolution of the whole system, and it is one of the cheapest places to improve quality because no model has to be trained.

How production systems tend to do it

A configuration that holds up well in practice looks like this:

1. Parse well first. Clean layout extraction, removed boilerplate, preserved headings and tables.
2. Split by structure, then recursively within sections to a token budget measured with the embedder’s tokenizer (often 200–500 tokens, modest overlap).
3. Attach context to every chunk. At minimum the document title and heading path, and for high-value corpora a generated contextual summary.
4. Keep parent links so the generator can receive the enclosing section when the matched chunk is too small to be useful alone.
5. Index for hybrid search (BM25 plus dense vectors), fuse, and rerank with a cross-encoder before choosing the final few chunks.
6. Version the chunking config alongside the index, and re-run a fixed evaluation set whenever it changes.