Evaluate multi-turn assistants across the whole conversation

Recent research focuses on interaction quality

A May 2026 preprint proposes a framework centered on transparency, consistency, and refinement in human-AI interaction. An August 2026 survey discusses persistent memory and cross-turn grounding as continuing challenges. These sources broaden evaluation beyond single responses but do not establish one standard score for every conversational product.

Evidence: Evaluating Multi-turn Human-AI Interaction [1]Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges [2]

Write the conversation state explicitly

Represent the user’s goal, active constraints, unresolved questions, and permitted actions after each meaningful turn. When the user changes a requirement, record which earlier condition is superseded and which remains in force. This gives reviewers an observable target without requiring access to the model’s internal reasoning.

Distinguish memory that should persist from information that should not. An assistant may need to remember a formatting preference within a session while keeping another customer’s details isolated. Evaluation should reflect the actual product’s session and identity boundaries.

Test realistic conversation transitions

Rubrex recommends scenarios where the user clarifies a vague request, corrects a factual assumption, changes the desired format, interrupts an operation, or returns to an earlier goal. Include cases where asking one question is helpful and cases where repeated clarification adds unnecessary friction.

Use a fixed transcript when you need a controlled comparison, and a carefully specified simulator when branching behavior matters. Inspect whether the simulator gives the assistant extra hints or accepts a premature completion. The simulator is part of the measurement system and can introduce its own bias.

Illustrative example: one constraint changes, another survives

A user requests a summary under 200 words for a technical audience, then asks for a nontechnical version. The audience changes; the length constraint does not automatically disappear. A final response that is clear but 500 words long fails the maintained requirement.

Score the final artifact and the transition: did the assistant apply the correction without reintroducing an earlier rejected assumption? Add a case where the user explicitly removes the length limit. This prevents the evaluation from rewarding rigid retention of every prior instruction.

Report outcomes and interaction costs

Track task completion, constraint preservation, correction uptake, unsupported assumptions, and avoidable clarification turns. Use examples to explain why a longer conversation was necessary or wasteful. More turns are not inherently worse; a well-placed clarification can prevent an incorrect action.

• Define the active task after each user correction.
• Review session boundaries and identity isolation.
• Keep simulated-user policies consistent across candidates.
• Distinguish useful clarification from repeated questioning.

Common questions