Choose RAG chunk sizes with an evidence-focused trial

Define the unit of success

Azure documents several chunking strategies rather than one universally optimal size. For this trial, label the smallest source span that supports each expected answer. Include definitions with exceptions, procedures with prerequisites, and paragraphs whose meaning depends on a heading. A relevant document is not enough: the returned chunk must contain the actual qualifying evidence. Keep page and character offsets so a reviewer can distinguish a retrieval failure from a split that removed necessary context.

Evidence: Chunk large documents for RAG and vector search [1]

Build a boundary-sensitive fixture

Create a synthetic support handbook with a standard cancellation rule followed by an exception for annual contracts. Place one test question about the general rule and another about the exception. Generate three chunked versions with different sizes, preserving the same raw text and permissions. Add a question whose evidence already fits inside a short paragraph. That control prevents the evaluation from rewarding larger chunks merely because the deliberately difficult example crosses a boundary.

Compare retrieval and generation separately

Use the same embedding model, query set, candidate depth, and total context-token allowance. A fixed number of chunks would give large chunks a larger information budget. First score whether each required evidence span survives into the final context. Then generate answers and check whether they state the correct exception. Record unused context, duplicate text, and request latency separately. If the evidence is present but the answer is wrong, investigate generation rather than declaring the chunk size defective.

Make a reversible indexing decision

Choose the smallest representation that preserves the required evidence on the held-out boundary cases without unacceptable context waste. This is a proposed decision rule, not a numerical industry threshold. Retain both indexes until you can replay important failures against either version. Review cases where the winning average hides the loss of a rare exception. Record the exact splitter configuration, tokenizer, document hash, and model settings with the decision so the next reindex can reproduce the comparison.

Check the denominator when boundaries change

Changing chunk size also changes how many relevant chunks appear to exist. Do not compare a denominator of ten small relevant chunks with three large relevant chunks and describe the difference as improved corpus recall. Define the required evidence against original document spans or answer requirements, then map each representation back to those stable labels. For a policy exception spanning two small chunks, either return both necessary spans or one larger chunk containing both. Keep the scoring unit unchanged across those possibilities. Otherwise the evaluation may reward a representation simply because its labels became easier to satisfy.

Common questions