On this page
The short answer
Watch for three kinds of change: what users ask, how well the system responds, and how you score its work. Keep these signals separate. Collect only the data you need to investigate a problem. Use reviewed failures to fill gaps in your test set.
What to take away
- A change in user requests does not always mean the model got worse.
- A new scoring rule can change scores even when the answers stay the same.
- Decide why you need each piece of data before collecting it.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
What the research and documentation cover
ProxyDrift, an August 2026 preprint, studies how to compare live traffic with test sets using structured summaries of interactions. These summaries stand in for direct access to the raw interactions. LangSmith’s documentation describes scoring live runs and adding examples to offline test sets. These are different approaches. Neither proves that a monitoring setup protects privacy or covers every kind of user request.
Evidence: Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations [1]LangSmith Evaluation [2]
Watch three kinds of change
Input drift means the requests have changed. Users may ask different tasks, use new languages, or send longer documents. Outcome drift means results have changed, such as more errors or slower answers. Measurement drift means the scoring process has changed. A new judge, scoring guide, or sampling rule can move the numbers. Record these changes separately.
A higher failure rate may mean the incoming tasks are harder. It does not always mean the model is weaker. An unchanged average can also hide a problem if easier requests balance out harder ones. Compare groups that matter to your product, such as language or task type. Show the number of examples behind each score.
Keep a focused monitoring record
Rubrex recommends starting with the decision each alert should support. To investigate a tool timeout, you may need the tool name, time, error type, and software version. You may not need the full user message. To check whether an answer is true, you may need the source document. Agree who can access that data and when it must be deleted.
Labels and number-based representations, such as embeddings, may still contain sensitive information. Check whether they reveal private details or can be linked to a person. Keep live user data out of public examples. Use fictional examples or remove sensitive details when you explain a failure. Data that has been transformed is not automatically safe to share.
Example: scanned documents cause new failures
Suppose a purchasing assistant starts receiving scanned documents after a new integration launches. The model has not changed, but the system now struggles to extract the text. The old test set uses clean digital text. It still passes, even as real users see more failures.
First, confirm where extraction fails using examples you are allowed to review. Add scanned documents to a separate test group, with checked reference text. Then test the extraction step as well as the final answer. A prompt change may not fix missing source text. Check whether the new tests catch the real problem before relying on them as a warning signal.
Give every alert a next step
Name an owner for each alert. Explain why its threshold matters and what the owner should check first. Review false alarms as well as incidents the alert missed. When a real failure becomes a test case, record where it came from, what should happen, and which test-set version contains it. Note changes in traffic and scoring rules when comparing old and new results.
- Record the application, test-set, judge, and sampling versions.
- Compare quality within each task group as well as overall.
- Review a broad sample alongside cases that trigger alerts.
- Check access and deletion rules for raw data and derived labels.
Make the dashboard useful during an incident
Monitoring shows how selected signals change over time. Observability helps you trace a signal back to a request, system setting, or dependency. Evaluation checks whether the result met the user’s needs. Rubrex recommends connecting these records with a run ID. Keep only the data needed for the investigation, under the product’s agreed data rules.
Try following one alert from start to finish. Can the team find the affected cases and the software version? Can someone turn the failure into a test that another engineer can rerun? Show sample counts, missing results, serious failures, and score definitions beside each trend. After a fix, rerun the test and check whether the result improved. A dashboard is useful when it helps someone decide what to do next.
Limits of the evidence
A summary of an interaction can leave out useful details. It is not a privacy guarantee. Check your monitoring plan against the product’s data rules and real failure cases. The cited preprint describes one approach; Rubrex has not reproduced its deployment or results in this article.
Common questions
Can logs alone tell whether answers are correct?
Usually not. Logs can show events and errors. To decide whether an answer is correct, you also need trusted source material, human review, or a scoring method you have checked.
Should the offline dataset change whenever traffic changes?
First check whether the change leaves a gap in the tests or changes a decision. Save a new version when you update the test set, so earlier results remain understandable.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations arXiv · 2026 · PreprintReviewed: abstract and publication record. Accessed September 25, 2026.
- LangSmith Evaluation LangChain · Living reference · Maintained documentationReviewed: evaluation overview. Accessed September 25, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.