Chunking Is the Whole Ballgame in RAG

When a retrieval system returns irrelevant results, the instinct is to blame the embedding model and go shopping for a better one. Occasionally that is the problem. Far more often the retrieval was doomed earlier: the text was cut into pieces that no longer contain a complete idea, and no embedding model can encode meaning that was destroyed before it arrived.

The default is wrong for almost everything

Nearly every RAG tutorial starts here:

const chunks = splitter.split(text, { chunkSize: 1000, chunkOverlap: 200 });

A thousand characters, chosen because it is a round number. The document gets cut at whatever character happens to be at position 1000, which is the middle of a sentence, or between a heading and the paragraph it introduces, or through the centre of a table.

Consider what that produces. A chunk ending “...the maximum retention period is” and the next beginning “90 days, after which records are purged.” Neither chunk answers “how long is data retained?” The first has the question's vocabulary and not the answer; the second has the answer and nothing to match the question against. Both embed to something vague. The information exists in your index and is unreachable.

What splitting on length actually breaks

• Resolution of pronouns and references. A paragraph beginning “It also supports...” is meaningless once separated from the sentence naming what it is. The chunk embeds as generic text about supporting something.
• Headings. The heading carries the topic; the body carries the detail. Split them apart and the body loses its subject while the heading becomes a near-empty chunk that matches everything weakly.
• Tables and lists. A table cut in half yields rows with no column headers — the numbers survive and their meaning does not.
• Sense of scope. A chunk from the middle of a document gives no signal about whether it describes the current version, a deprecated one, or an example of what not to do.

Overlap is the usual patch, and it is a weak one. It increases the chance that a boundary-straddling fact appears intact somewhere, at the cost of duplicating content across chunks — which then compete with each other in results, filling the context window with near-identical text and crowding out genuinely different material.

Split on structure, because documents already have it

The insight that fixes most of this: documents are not undifferentiated text. They have sections, headings, paragraphs, list items and code blocks — an author-provided structure describing which ideas belong together. Character splitting throws that away and then tries to approximate it with a number.

The approach that works, in order:

• Parse the document into its natural units. Markdown into headings and sections, HTML into semantic elements, PDFs into whatever structure survived — which is a real preprocessing problem, not an afterthought.
• Make a section the candidate chunk. One section is usually one idea, which is exactly what you want to retrieve.
• Split further only when a section exceeds your budget, and split at paragraph boundaries rather than mid-sentence.
• Merge sections that are too small. A three-word heading with one sentence beneath it is not worth its own vector. Combine adjacent short sections until they carry enough signal.

The size targets stop being arbitrary once structure leads. Aim for a range rather than a value — perhaps 200 to 800 tokens — and let the document decide where inside it each chunk lands.

Chunking code, where the default fails hardest

Source code is the case that makes the argument unavoidable. Splitting a file every thousand characters produces chunks containing the last third of one function and the first half of the next. Neither is a thing that exists. A developer asking “how does authentication work?” needs a function, whole, with its signature.

The right unit is the syntax tree. Parse the file, walk the AST, and emit chunks at declaration boundaries — a function, a class, a method — so every chunk is a complete, self-contained construct:

// walk the parse tree and emit whole declarations
function chunksFrom(tree, source) {
  const out = [];
  for (const node of tree.rootNode.namedChildren) {
    if (!DECLARATIONS.has(node.type)) continue;      // function, class, method
    out.push({
      text: source.slice(node.startIndex, node.endIndex),
      symbol: nameOf(node),
      kind: node.type,
      startLine: node.startPosition.row + 1,
    });
  }
  return out;
}

A parser generator with grammars for many languages makes this practical across a polyglot repository rather than being a per-language project. And the payoff goes past retrieval quality: because each chunk is a named declaration, you get structured metadata for free — the symbol, its kind, its file, its line range. That metadata is what lets you filter, cite a precise location, and detect later that a specific function changed.

The same principle generalises. Whatever your corpus is, find the unit the domain already considers whole — a function, a clause, a ticket, a transcript turn — and make that the chunk.

A chunk needs to carry its context

Even a well-bounded chunk is retrieved alone, stripped of everything around it. The reader — a model — sees only what the chunk contains, so the chunk has to be self-describing.

The cheapest high-value technique is prefixing each chunk with its position in the document hierarchy before embedding:

Billing Guide > Refunds > Partial refunds

A partial refund returns part of the captured amount...

Now the vector encodes that this text is about refunds in billing, and a query about “partial refund policy” matches strongly even though the body never repeats those words. This one change routinely does more for retrieval quality than switching embedding models.

Two related habits worth adopting. Store rich metadata alongside every chunk — source, section path, timestamp, version, permissions — because filtering on metadata is often more precise than similarity alone. And keep a pointer back to the full document so that once a chunk is retrieved you can expand around it, giving the model the surrounding paragraphs rather than the fragment.

What to actually tune, and in what order

The temptation is to tune chunk size first because it is a number in a config file. It is roughly the least important variable. In descending order of impact:

• Chunk boundaries. Structural rather than character-based. Largest single improvement available.
• Context prefixing. Section path and title into the embedded text.
• Metadata and filtering. Narrowing the candidate set before similarity does more than reranking a bad candidate set after.
• Chunk size. Within a sensible range, matters much less than the three above.
• Embedding model. Real differences exist, and they are smaller than the differences above.

And measure with retrieval metrics rather than by reading final answers. Build a set of questions with the chunks that should be retrieved for each, then track how often the right chunk appears at all and how highly it ranks. Those two numbers tell you whether a change helped; judging the generated answer conflates retrieval quality with the model's ability to write around gaps.

The short version

Retrieval can only ever return what chunking preserved. Cut on the structure the document already has — sections, paragraphs, declarations — rather than on a character count, and use the syntax tree when the corpus is code. Prefix each chunk with its place in the hierarchy so it is self-describing, and store metadata so you can filter before you rank. Then measure retrieval directly, because a good answer from a bad retrieval is luck.