On this page
The short answer
Use a unique harmless canary to trace whether information crosses a boundary it should not cross. Place it on a controlled source surface, define permitted destinations, and inspect the actual data path. A missing canary proves little if the system never encountered it, so include an exposure-confirming control.
What to take away
- Use distinctive invented values with no real authority.
- Document permitted and prohibited destinations.
- Verify the canary entered the intended source path.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define one data-flow question
Langfuse documents masking limits at export boundaries. Rubrex recommends using a canary to test one concrete flow rather than labeling the whole application secret-safe. Examples include a tool error reaching telemetry, a private document reaching another user, or an attachment appearing in a generated artifact.
For a synthetic export assistant, place a unique invented marker in a restricted metadata field. The permitted destination is a local restricted diagnostic record; the prohibited destinations are the user-visible summary and a general analytics exporter. Record this matrix before running the fixture.
Evidence: Langfuse: Masking sensitive LLM data [1]
Confirm exposure without creating a real secret
Create an instrumented source stub that confirms the application received the marker. A test can falsely pass because the relevant tool was never called or a parser dropped the entire record. Include a normal neighboring field whose value should survive so a blank-output implementation is not rewarded.
Use separate canaries for prompt content, tool output, metadata, and error messages. Their distinction identifies where exposure occurred. Never assign a canary real account access, route it to an uncontrolled collection endpoint, or copy actual credentials into the fixture merely to make the test realistic.
Inspect transformations and every declared destination
Search exact marker values in the captured payloads, then inspect transformations such as truncation, encoding, or summary output where relevant. A distinctive value can be split across fields or removed from one channel while remaining in another. Keep the inspection limited to destinations the evaluator is authorized to observe.
Record source receipt, application processing, exporter payload, stored result, and final artifact. Separate missing instrumentation from a confirmed blocked flow. Report each canary path individually so a safe analytics export does not conceal disclosure in an error tracker or downloadable file.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Source exposure | Stub receipt | Marker actually encountered |
| Analytics export | Captured exporter payload | Restricted canary absent |
| Safe neighbor field | Final report field | Useful evidence preserved |
Treat findings as data-path defects
Accept the export fixture when the source control confirms exposure, necessary safe fields remain, and prohibited destinations contain no marker or restricted meaning under the defined review. Fix the specific collector, serializer, or consumer that crossed the boundary. Broadly suppressing every output may conceal the defect while breaking the task.
Rotate synthetic values between runs and retain the source-to-destination map. Re-run when adding exporters or artifact formats. A successful canary test can show that one known value did not appear in inspected outputs; it does not establish that all sensitive data classes are detected or that inaccessible downstream stores are clean.
Limits of the evidence
The canary matrix is proposed and no real secret was used. Exact matching is limited to known values and inspected channels. This is a diagnostic method, not a comprehensive data-loss prevention assessment or certification.
Common questions
Is a canary a real credential?
It should be an invented value without authority, selected only for traceability.
Can a missing marker prove redaction?
Only after confirming exposure and inspecting the declared destinations; missing instrumentation is inconclusive.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Langfuse: Masking sensitive LLM data Langfuse · Living reference · Maintained documentationReviewed: export-stage versus legacy Python masking and scope across exporters; not an implementation audit. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.