On this page
The short answer
Evaluate redaction at every destination where sensitive content may leave the application. Use synthetic markers in prompts, tool output, metadata, and errors, then inspect exported payloads and retained records. A clean dashboard view does not establish that another exporter, raw payload, or artifact received the same transformation.
What to take away
- Test metadata and errors as well as visible text.
- Inspect each configured exporter independently.
- Preserve safe identifiers needed for investigation.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Map transformations before testing patterns
Langfuse documents separate behavior for export-stage masking and legacy SDK hooks. Rubrex recommends mapping where each field is created, transformed, and transmitted. A function that changes one SDK attribute may not affect raw spans from another instrumentor or a different exporter.
Use a synthetic retrieval task with fake contact values in input, a tool result, metadata, and an exception. Assign a distinct marker to each field. Define which fields may be retained, which must be removed or transformed, and which stable identifiers must remain for troubleshooting.
Evidence: Langfuse: Masking sensitive LLM data [1]
Capture what leaves each destination
Inspect payloads at authorized test destinations rather than relying only on the tool’s display. Run two exporters if the intended deployment uses them. A safe representation in one backend does not establish that the other received transformed data.
Add nested metadata, repeated values, and a field whose type changes between a string and a structured object. These cases test the transformation contract without requiring real sensitive information. Include a normal safe field as a control so a redactor that drops the entire trace cannot pass by destroying all useful evidence.
Test failure behavior and semantic usefulness
Make the redaction function fail in an isolated trial and observe whether the batch is dropped, transmitted, retried, or surfaced as an operational error under the actual implementation. The accepted behavior should be explicit and verified; do not infer it from a successful normal case.
Give a reviewer the sanitized record and ask whether they can still identify the task, failing step, and relevant version. Report marker exposure and evidence loss separately. A redactor may remove exact values while leaving a sensitive meaning in a generated summary, so include a bounded semantic review where the task requires it.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Nested metadata marker | Exporter payloads | Restricted value transformed |
| Second exporter | Independent captured payload | No assumed shared masking |
| Redactor error | Batch decision and alert | Declared failure policy observed |
Accept a verified collection design
Release the fixture when prohibited markers are absent from all inspected destinations, the failure path follows the declared policy, and safe diagnostic evidence remains useful. Prefer reducing unnecessary collection when redaction is unreliable rather than expanding a pattern list indefinitely.
Retain the field map, exporter configuration, synthetic payloads, and reviewer decisions. Re-run after SDK upgrades or added exporters. The acceptance record should state which data classes and destinations were tested; it should not claim that every sensitive value or uninspected downstream copy has been removed.
Limits of the evidence
All sensitive-looking values are invented. This proposed redaction test is not a privacy audit or certification. SDK hooks, exporter copies, and failure behavior vary by version, and a passing marker set does not prove complete detection.
Common questions
Is the dashboard the only surface to inspect?
No. Inspect authorized payload and storage surfaces for every configured destination.
Can exact matching establish full privacy?
It checks known markers; semantic disclosure and uninspected data paths need separate review.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Langfuse: Masking sensitive LLM data Langfuse · Living reference · Maintained documentationReviewed: export-stage versus legacy Python masking and scope across exporters; not an implementation audit. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.