Measure duplicate passages in a RAG context window

Distinguish copies from conflicting versions

Azure’s chunking guidance includes overlapping text, which can legitimately produce similar neighboring chunks. Separate those overlaps from copied documents and revised policies. Two passages differing only by a date, quantity, or negation may be crucial alternatives rather than duplicates. Label exact copies using stable source spans where possible, and manually inspect near-duplicate groups in the initial fixture. A text-similarity threshold is a candidate detector, not evidence that two passages are interchangeable.

Evidence: Chunk large documents for RAG and vector search [1]

Construct a context-capacity example

Author a catalog question requiring a product specification and a regional compatibility note. Return three copies of the specification, one unrelated passage, and the compatibility note at the next rank. Under a four-slot context policy, repeated specifications displace the note. The acceptance rule requires both distinct evidence groups. A deduplicated policy should preserve the useful specification once and allow the note to enter, rather than simply reducing the number of returned results without refilling the context.

Calculate an interpretable redundancy measure

Define a group ID for passages carrying the same required evidence. For an illustrative four-result context containing three copies of one fact and one distinct fact, there are two distinct evidence groups across four slots. Report that occupancy directly rather than claiming two groups imply fifty-percent answer quality. Also record repeated token count when source offsets permit it. Slot redundancy and token redundancy answer different questions because chunks can have very different lengths.

Test deduplication before adopting it

Replay the same ranked candidates with and without the proposed deduplication step. Keep the final context allowance fixed and record which new passages enter. Include an exception differing by one word and two policy revisions with different effective dates. The rule must preserve these distinctions or provide a deliberate version-selection policy. An answer that becomes shorter or cites fewer sources is not sufficient evidence of improvement; check whether the required facts and qualifiers survive.

Trace repetition to its origin

If overlap causes most duplication, adjust context assembly or grouping rather than deleting useful source chunks indiscriminately. If copied source files dominate, repair document identity during ingestion. If one large source overwhelms search results, consider a tested diversity constraint and check that it does not exclude necessary evidence from the same document. Preserve duplicate groups and displaced evidence in failure records so the next engineer can choose an intervention aimed at the real cause. Also test two independently authoritative sources that state the same fact. Collapsing their text may be reasonable for context length, but preserve both provenances when independent corroboration is part of the user’s evidence requirement.

Common questions