Evaluate context compression for lost evidence

Define information that must survive

The Lost in the Middle paper studies sensitivity to evidence placement in long contexts. Compression addresses a different intervention, so do not treat that paper as proof that a compressor improves answers. Label the source claims, exceptions, and provenance your questions require. Mark content that can be removed without affecting the decision. Keep these labels outside the compressor input so the trial tests whether the transformation preserves evidence rather than whether it can copy a supplied answer key.

Evidence: Lost in the Middle: How Language Models Use Long Contexts [1]

Create a missing-qualifier challenge

Author two passages about a fictional discount. One announces a twenty-percent reduction; the other limits it to new annual subscriptions. Ask whether an existing monthly account qualifies. A compressed context that retains the attractive percentage but drops the qualification must fail, even if its answer is fluent. Add a control question about the percentage alone. That distinction shows whether the compressor preserves the specific information needed by different questions rather than always copying every sentence.

Compare evidence and generated decisions

Save original passages, compressed output, retained source IDs, and token counts. Check claim support before generating answers, then compare the original-context and compressed-context answers under identical generation settings. Include an unsupported question for which both versions should abstain. A compressor may accidentally turn uncertainty into apparent certainty by removing contradictory or missing-information cues. Record compression latency and model calls as well as generation savings; a shorter final prompt does not necessarily mean less total work.

Use a task-specific retention gate

Accept a compression policy when it preserves all chosen critical qualifiers and provenance on held-out cases while delivering the required operational benefit. If source links cannot be mapped back to original passages, treat that as a separate auditability failure. Inspect cases that improve after removing distractors and cases that regress after losing a condition. Keep an uncompressed fallback for high-risk or disputed questions rather than making compression irreversible before the evidence supports that choice.

Inspect whether compression invents connective claims

A compressor can preserve two source facts yet introduce an unsupported relationship between them. For example, a product discount and a customer eligibility rule may appear in different contexts; combining them into “all eligible customers receive this discount” adds a claim. Label those connective statements during review. Require each to be supported by the original passages, not merely plausible. Compare extracts with generated summaries when the task allows it. The point is to find the least complex transformation that preserves the evidence needed for the decision, rather than rewarding fluent compression on appearance. A source attribution removed during compression should be treated as a provenance defect even if the remaining sentence is supported.

Common questions