Test whether context order changes a RAG answer

Distinguish placement from relevance

Lost in the Middle reports evidence-position effects in the models and tasks it studied. It does not establish the same effect size for your application. Define a paired test that changes only passage order. Preserve passage text, separators, source IDs, question, system instructions, and generation settings. If the assembly code also truncates passages or renumbers citations, the experiment changes more than order and cannot isolate placement sensitivity cleanly.

Evidence: Lost in the Middle: How Language Models Use Long Contexts [1]

Prepare a controlled evidence set

Author a fictional billing question with one current applicable policy, several irrelevant passages, and no ambiguity about the expected answer. Put the supporting policy first, in the middle, and last. Then create a separate conflicting-policy case where source dates determine applicability. Do not mix those outcomes: choosing an obsolete policy is an evidence-resolution problem as well as a placement issue. Each variant should preserve the same source identities so citations can be compared without positional renaming.

Score paired changes and repeat unstable runs

For each arrangement, record accepted answer, unsupported claim, citation target, and context length. If generation is nondeterministic, repeat runs using an agreed protocol and retain all outputs rather than selecting the cleanest response. Compare whether the required evidence was cited and used correctly, not merely whether an expected phrase appeared. Check the serialized prompt to confirm that the supporting passage survived token limits in every variant. A missing passage is a truncation failure, not evidence of attention to position.

Use ordering as a tested policy

If a particular assembly policy is more reliable on held-out cases, keep the comparison as a regression test and record the tested context lengths and question types. Avoid treating a beginning-or-end heuristic as a universal solution. A smaller evidence set, clearer source metadata, or better contradiction handling may be a more durable fix. Preserve a case that failed in the middle and a control that stayed stable so future changes can distinguish a real repair from a changed test fixture.

Keep ordering tests independent of source authority

Random shuffling is useful for placement sensitivity, but an application may intentionally order sources by authority, date, or relevance. Test those policies separately from arbitrary permutations. A model that chooses the first passage can look stable under one production policy and fail when an exception is inserted later. Include a case where the most authoritative applicable source is neither first nor longest. The expected answer follows applicability and authority, not position. Save the authority labels outside the prompt when they are evaluation ground truth, while passing only metadata genuinely available to the deployed application.

Common questions