Test follow-up questions against the correct conversation context

Define the conversation evidence boundary

LangChain documents short-term memory as thread-scoped state. A follow-up evaluation should identify the actual messages and state available when a query is resolved. Do not provide an evaluator with later clarifications that the application never saw. Label the intended entity, requested property, and relevant prior constraints. A standalone rewrite is useful only when it preserves those facts. Otherwise it can make a vague question searchable by silently choosing an unsupported interpretation.

Evidence: Short-term memory [1]

Author a correction-sensitive exchange

Create a fictional conversation in which the user first asks about Plan A, then corrects the request to Plan B, and finally asks “Does it include weekend support?” The expected retrieval query concerns Plan B. Add a second exchange discussing both plans without a clear final referent; that case should request clarification. Include a topic switch to an unrelated product so the resolver cannot pass by always carrying the first named entity through the session.

Score the resolved request before search

Record the original follow-up, relevant history, resolved entity, constraints, and generated retrieval query. Judge those fields independently of search results. An excellent Plan-A passage does not rescue an incorrect Plan-A interpretation of the corrected request. Then test whether retrieval finds the applicable Plan-B evidence. Keeping those stages separate makes it possible to repair conversation resolution without changing the retriever or to repair retrieval without blaming a correct resolution.

Test truncated and summarized history

The application may trim messages or replace them with a summary. Replay the fixture with the actual retained state and check whether the correction survives. A summary that preserves the initial plan but drops the correction creates an upstream evidence loss. Add a case where old context should expire rather than remain a permanent assumption. Record summary revision and message boundaries so an answer failure can be traced to the representation the resolver actually received.

Choose a policy for genuine ambiguity

Define when the assistant should ask a concise clarification instead of guessing. Score excessive clarification separately on unambiguous cases so the resolver remains useful. Preserve user corrections as high-priority state changes and test that they override earlier assumptions within the authorized scope. Keep the resolved query with the final cited evidence in failure records. The next reviewer should be able to see whether the wrong answer arose from an incorrect antecedent, stale history, or an irrelevant search result. Include a follow-up that explicitly repeats the entity name despite conflicting earlier context. That control should override the resolver’s prior assumption and demonstrates whether a memory-summary shortcut is improperly stronger than the user’s current words.

Common questions