Use a decision-tree approach to choose a chunking strategy for your LLM applications.

Meta description: Learn how to choose a chunking strategy for retrieval-augmented generation using a six-question decision tree, strategy tradeoffs, and failure signals.

Tags: chunking strategy, retrieval-augmented generation, document structure, vector search, embedding model

Featured image: abstract, friendly illustration style showing documents being divided into connected blocks, with no real people, no logos, and no text

Why Chunking Boundaries Matter

This guide is for builders of retrieval-augmented generation systems. It explains how to choose a chunking strategy by looking at document structure, retrieval needs, cost, and failure signals instead of defaulting to a familiar method.

When documents are larger than an embedding model or LLM context window can handle, they must be split into smaller chunks. Chunk size and overlap may look like simple configuration settings, but they directly affect retrieval. Bad boundaries can cause the system to miss the passage that answers a query, or to return a chunk that is technically relevant but lacks the surrounding context needed to make it useful.

A chunking strategy defines where a document is split. When the boundaries preserve the right context, embeddings, vector search, and generation have a stronger basis to work with. When the boundaries break important context, reranking and prompt tuning cannot fully recover what was lost during splitting. No single chunking method works well for every document, so the choice depends on the structure of the document and the retrieval task.

Six Questions That Narrow the Choice

Each chunking strategy is based on a specific assumption about the document. If the strategy fits the document, its added complexity can be worthwhile. If it does not fit, it adds overhead without improving retrieval. For example, semantic chunking is unnecessary for a short FAQ entry, while fixed-size chunking can split related clauses in a contract. These mismatches can create retrieval problems before generation even begins.

Use the following six questions to map document and retrieval conditions to a strategy.

Question 1: Does the Document Need Chunking at All?

Short, self-contained content may not need chunking. FAQ entries, product descriptions, or single config snippets often fit comfortably inside one embedding call and cover exactly one topic.

If the document sits well within the model’s context window and does not mix subjects, embed it whole and skip chunking. Splitting content that is already coherent only adds retrieval overhead. If the document is too long, or covers more than one topic, continue to Question 2.

Question 2: Does the Document Have a Clear Internal Structure?

Some documents already contain natural boundaries that reflect how their content is organized. Markdown headings, HTML elements, PDF sections and tables, and code functions or classes can all provide meaningful chunk boundaries.

If the document has a reliable structure that maps to logical content units, use document-based chunking. The document is parsed according to its format, keeping sections, paragraphs, tables, functions, or other structural elements together. If the structure is inconsistent, incomplete, or does not provide useful boundaries, continue to Question 3.

Question 3: Do Chunks Need Context From Elsewhere in the Document?

Some documents carry dependencies that span the whole text. A legal contract may define a term in section one and use it in section eight. A technical spec may have terminology that only makes sense next to an earlier paragraph.

If retrieval needs to preserve that document-wide relationship, use late chunking. Embed the full document with a long-context model first, then derive chunk-level embeddings from that result, so each chunk still carries a trace of the whole document. If chunks can stand on their own without losing meaning, continue to Question 4.

Question 4: Does Information Density Vary, or Do You Need Multiple Levels of Granularity?

This question covers two different conditions, each pointing to a different strategy.

If information density varies across sections, use adaptive chunking to dynamically adjust parameters such as chunk size and overlap to match the content. Dense, information-rich sections can use smaller, more granular chunks, while broader or less dense sections can use larger chunks.

If the same corpus needs to support both broad questions, such as what a document covers, and specific questions, such as exact syntax, use hierarchical chunking, with coarse parent chunks and finer-grained child chunks.

If neither condition applies, continue to Question 5.

Question 5: Does Retrieval Quality Matter More Than Cost and Latency?

This is the cost and quality tradeoff. If retrieval quality is the priority and the budget supports a model call per document, use LLM-based chunking to identify logical breakpoints or atomic propositions.

If quality still matters but per-document model calls are too costly, and topic shifts are unpredictable, use semantic chunking based on sentence-embedding distance. It is less costly than an LLM pass while still accounting for meaning.

If speed and cost are the priority, continue to Question 6.

Question 6: Is This One Document Type, or a Heterogeneous Collection?

If the corpus mixes document types where the right strategy differs from one document to the next, use agentic chunking. For example, a collection might contain structured reports, narrative transcripts, and code. An agent can inspect each document and select the appropriate approach.

If the collection is reasonably uniform, recursive chunking is a practical default. It splits content at paragraph, sentence, and then word boundaries until the chunks fit the target size, while preserving natural structure without requiring per-document analysis.

If speed matters more than chunk quality, fixed-size chunking is a reasonable fallback for quick prototypes.

How the Strategies Compare

The decision tree usually lands on one of these strategies, though real systems often blend more than one.

Document-based chunking works when the document has clear structure, such as headings, tables, or code units. Its value is that it preserves logical boundaries already present in the source format.

Late chunking is useful when chunks need context from elsewhere in the document. It requires a long-context model and embedding the full document first, so it is more complex than simple splitting.

Adaptive chunking helps when information density varies. It adjusts chunk size and overlap, but that flexibility can add implementation complexity.

Hierarchical chunking is useful when the corpus must support broad and specific retrieval at the same time. It keeps parent and child chunks, which can help with different query types but may require more retrieval logic.

LLM-based chunking is the choice when retrieval quality is the priority and a model call per document is acceptable. It can identify logical breakpoints or atomic propositions, but it is more costly than simpler methods.

Semantic chunking based on sentence-embedding distance is a middle option when quality matters but per-document LLM calls are too costly. It accounts for meaning while being less expensive than an LLM pass.

Agentic chunking fits heterogeneous collections. It can select a different approach for each document, but it requires an agent to inspect documents and decide how to split them.

Recursive chunking is a practical default for reasonably uniform collections. It preserves natural structure by splitting at paragraph, sentence, and word boundaries, without requiring per-document analysis.

Fixed-size chunking is a quick fallback when speed matters more than chunk quality. It can be useful for prototypes, but it may split related content when the document has structure or dependencies.

Signals That a Chunking Strategy Needs Review

A strategy that looks right on paper can still misbehave when it meets actual documents. Watch for these signals.

If retrieval misses the passage that answers a query, the boundaries may be splitting the relevant content across chunks or excluding it from the returned chunk.

If the system returns a chunk that is technically relevant but lacks surrounding context, the boundaries may be preserving keywords while breaking the information needed to understand them.

If short, coherent documents are being split unnecessarily, the strategy may be adding retrieval overhead without improving quality.

If structured documents such as contracts are being split by fixed size, related clauses may be separated, causing retrieval problems that originate in preprocessing rather than generation.

If semantic chunking is used for simple content such as a short FAQ entry, the added complexity may not be justified.

If retrieval quality remains weak after reranking and prompt tuning, the issue may be that important context was already lost during chunking.

What To Do Next

Choosing a chunking strategy starts with understanding the structure of your documents and the retrieval task they need to support. Document structure, context dependencies, information density, retrieval requirements, and cost all shape the choice.

Use the six questions to identify a suitable starting point, then validate it with your own queries and retrieval results. Start with the simplest strategy that fits your needs, and introduce more advanced approaches only when they address a specific retrieval challenge.

The best results come from matching chunk boundaries to the information you need to retrieve, while keeping added complexity proportional to the value it provides.