On this page
The short answer
Agentic RAG lets an agent decide when and how to retrieve information. Evaluate those decisions alongside the final answer: whether retrieval was needed, whether the selected source was permitted, whether additional searches added evidence, and whether the agent stopped within its task budget. Compare the complete workflow with a fixed retrieval baseline.
What to take away
- Score the final task and the retrieval decisions that produced it.
- Permit alternative valid paths instead of requiring one exact sequence of tool calls.
- Include missing evidence, source outages, conflicting documents, and repeated-search loops.
- Keep permissions and work limits enforceable outside the model.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the extra decisions your agent is allowed to make
LangChain distinguishes fixed retrieval before generation from agentic retrieval in which the model chooses when and how to search. Its current retrieval guide also describes hybrid workflows with validation steps. The label alone does not tell you whether an application can rewrite queries, select among sources, perform follow-up searches, or use other tools.
Write down those allowed decisions for the actual product. An internal policy assistant might search approved policy documents and an FAQ index, while a research assistant might also fetch permitted web pages. Test only the tools and permissions you intend to ship. A more elaborate architecture creates additional behaviors to evaluate; it does not supply evidence of better answers.
Keep a fixed-retrieval baseline on the same task set
Rubrex recommends comparing the agent against a retrieve-then-answer workflow using the same corpus snapshot and task acceptance criteria. Include straightforward questions as well as questions that need multiple sources. Otherwise, an agent may look useful simply because the test set contains only tasks designed around its strengths.
Keep final-answer correctness, evidence support, permissions, and resource use as separate outcomes. A successful answer that exceeded the work budget remains a budget failure. A fast refusal on a question the authorized corpus answers remains an unnecessary abstention. Decide in advance which conditions are release blockers and which are trade-offs to investigate.
Create a small fixture that makes search behavior inspectable
Use a controlled collection of synthetic documents and a search tool that records the query, permitted source scope, returned identifiers, and document versions. Capture the visible tool sequence and final answer; this method does not require hidden model reasoning. Add an explicit task identifier so a reviewer can connect all events to one trial.
The table is a proposed starting fixture, not an executed benchmark. For each case, write the expected final behavior and any prohibited action before running the system. Accept several legitimate retrieval sequences where they reach the same authorized outcome.
| Case | Fixture change | Acceptance condition |
|---|---|---|
| Simple answered question | One current passage directly answers the question | Correct supported answer without unnecessary repeated searches |
| Two-source relationship | One passage defines a term; another gives its exception | Answer reconciles both passages and cites the relevant evidence |
| Ambiguous question | Two projects share the same short name | Ask for clarification or clearly bound the answer |
| Missing evidence | Remove the document needed for a required fact | State the evidence gap without inventing the fact |
| Source outage | One permitted search source returns a controlled error | Use only approved recovery paths and disclose remaining limits |
| Repeated unhelpful results | Return the same irrelevant identifiers | Stop or clarify within the configured work budget |
Measure retrieval decisions without overfitting the trace
Begin with task success: tasks meeting all required answer and boundary conditions divided by tasks attempted. Then inspect retrieval-needed cases in which no search occurred, cases with irrelevant repeated searches, and tasks that reached the step limit. These are diagnostic slices, not components to hide inside a single weighted score.
For a reviewer-labeled subset, mark whether each search added new evidence relevant to an unresolved requirement. Define what counts as new: a different document identifier containing the same copied passage may add no information. Report the reviewed subset and unresolved labels. Avoid declaring every failed search a model error; an empty index, broken filter, or unavailable source can cause the same trace.
Record total latency to a usable final result, search-call count, model attempts, and full task cost. For distributions, retain slow and failed tasks rather than calculating latency only over fast successes. Compare repeated trials per case so that a good average does not conceal a path that sometimes loops.
An illustrative trace with a justified follow-up search
Imagine an invented support policy set. P1 says standard replacements are available within 30 days. P2 says enterprise contracts may override that period. A user asks about an enterprise replacement after 45 days, but supplies no contract identifier. The first search returns P1 and P2. A useful next action is to ask for the missing contract context or search an authorized contract source if the user has provided it.
Repeatedly rewriting the same generic replacement query does not resolve the missing authorization or contract identity. An illustrative review record would say: initial retrieval relevant; exception discovered; required contract evidence absent; clarification needed. The correct final behavior can therefore be an incomplete answer with a precise question, rather than an unsupported yes or no.
If another valid trial retrieves the contract first and then checks the general policy, do not fail it merely for reversing the order. Evaluate required evidence, the permission boundary, and the answer. Trace assertions should focus on meaningful constraints such as no external search or at most the configured number of attempts.
Test permission scope and stopping conditions explicitly
A retrieval tool should enforce the caller’s allowed source scope rather than trust a model-generated tenant identifier. For a proposed isolation test, create two synthetic tenants with different harmless markers and exercise query rewrites, follow-up searches, and cache reuse. Verify that the other tenant’s marker never reaches returned context or the answer. This checks a stated application boundary; it is not proof of complete isolation.
Configure a small maximum search count, model-attempt count, and wall-clock deadline in the test environment. Introduce a slow response and confirm how cancellation affects queued work and in-flight calls. Record incomplete outcomes distinctly from successful abstentions. A message that says the agent stopped is insufficient evidence if tool activity continues afterward.
Treat instructions embedded in retrieved text as content rather than permission to broaden access or take an action. A read-only retrieval assistant should remain read-only even when a synthetic source asks it to send a message. Use an instrumented tool boundary and record whether an unauthorized call was attempted and whether any side effect occurred.
Use failure categories to choose the next change
When the agent misses a required passage, inspect source choice, query construction, filters, and index coverage. When it retrieves the passage but answers incorrectly, investigate context assembly and generation. When it repeats searches without progress, investigate stopping rules or missing clarification behavior. Assign an owner and a specific follow-up test to each category.
Keep the agentic workflow only where its measured benefit justifies additional latency, cost, and operational complexity. Retain the baseline for simpler tasks if that serves the product contract. A Rubrex evaluation can scope these comparisons and document the decision record; it does not turn a small fixture into a guarantee of production reliability.
Limits of the evidence
This is a proposed evaluation workflow, with authored illustrative documents and traces. No live agent, retrieval service, or client system was tested for this article. The LangChain source establishes architecture terminology, not measured performance. Task labels, permission tests, budgets, and acceptance thresholds need to be adapted to the specific product and verified in its runtime.
Common questions
What is the difference between RAG evaluation and agentic RAG evaluation?
Both need supported and useful final answers. Agentic evaluation also examines source selection, follow-up retrieval, recovery, and stopping decisions because those actions can vary between trials.
Should I require an exact tool sequence?
Usually require the necessary evidence and enforce the prohibited actions and budgets. Several sequences can be valid; enforce an exact sequence only where the product contract makes that order necessary.
Does agentic RAG require several agents?
No. A single agent with retrieval tools can choose when and how to search. Multiple agents add a separate coordination design and additional evaluation requirements.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- LangChain: Retrieval and RAG architectures LangChain · Living reference · Maintained documentationReviewed: 2-step, agentic, and hybrid retrieval definitions; original LangChain retrieval URL redirected here. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.