On this page
The short answer
Evaluate memory by testing both appropriate recall and information that should no longer influence an answer. Use synthetic preferences with known owners, versions, and expiry rules. Separate conversation state from durable memory, and verify correction and deletion paths rather than treating every remembered detail as a quality improvement.
What to take away
- Score useful recall and harmful persistence separately.
- Attach owner and provenance to remembered facts.
- Test correction, expiry, and cross-user separation.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose facts worth remembering
LangGraph documents stores for cross-thread information and checkpoints for thread state. Rubrex recommends deciding which product facts deserve durable memory before measuring recall. A temporary shipping address, a stated style preference, and an inferred personal trait have different ownership and retention implications.
Create a synthetic assistant with an explicit user preference for concise reports and a one-task delivery address. The preference may persist under the chosen product rules; the address should expire with the task. Record when each fact was supplied, who owns it, and whether it was stated or inferred.
Evidence: LangGraph: Persistence [1]
Build a sequence rather than isolated questions
Start a new conversation and ask for a report to test appropriate preference recall. Then correct the preference to detailed reports and repeat the task. In a separate branch, delete the preference and confirm that the assistant no longer treats the old value as an instruction.
Add a second synthetic user with an opposite preference and a matching project name. Neither user’s facts should contaminate the other. Include a quoted document that states someone else’s preference; the assistant should not silently store it as the current user’s durable instruction merely because it appeared in context.
Measure recall with provenance and abstention
Count correct applicable recalls, stale recalls, wrong-owner recalls, and unjustified stored inferences. A memory hit rate alone can reward remembering everything, including facts the product intended to forget. Inspect the retrieved memory record and its application in the answer so failures can be attributed to storage, retrieval, or use.
Test a missing or contradictory memory record. The useful response may be a clarification or a neutral default, not a confident assertion about the user. Keep a control where the current request overrides a durable preference; memory should support the user’s present intention rather than trap them in an old setting.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Corrected preference | Old and new memory versions | New preference applies |
| Task address expired | Expiry and retrieval record | No durable reuse |
| Other user | Namespace and returned memory | No cross-user recall |
Retain memory only when it changes useful behavior
Accept the feature when the concise-to-detailed correction changes the report as intended, deleted or expired values stop influencing new tasks, and ownership boundaries hold. Remove fields that add little value but repeatedly create stale assumptions. Store the minimum evidence needed to explain why a memory applies.
Review memory tests after changing summarization, embeddings, or namespace rules. A successful synthetic sequence does not establish that every old memory has been removed from every derived system. The release record should name the surfaces actually tested and the recovery path for a user who reports an incorrect remembered fact.
Limits of the evidence
The preferences and identities are invented. This is a proposed memory evaluation, not a privacy certification or an executed deletion audit. Retention and deletion behavior must be verified across the actual storage, cache, and derived-data surfaces.
Common questions
Is more recall always better?
No. Stale, unauthorized, or wrongly inferred recall can reduce usefulness.
Should memory override a current request?
Usually the explicit current task should govern; define any exceptions in the product contract.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- LangGraph: Persistence LangChain · Living reference · Maintained documentationReviewed: thread-scoped checkpoints versus cross-thread stores; in-memory restart limitation; original durable-execution URL redirects here. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.