Privacy-Safe LLM Evaluation Datasets for AI Workflows

An LLM evaluation dataset becomes a privacy problem when a team copies a useful production failure into a fixture and forgets how many systems receive it. The prompt may be cleaned, while retrieved passages, tool results, grader inputs, traces, exports, and screenshots still contain customer details or secrets. That second store can outlive the production record it came from.

Use a fixture pipeline instead of relying on one redaction pass. Inventory every derivative, remove fields before sampling, replace necessary sensitive values with relationship-preserving synthetic values, and test privacy and task fidelity together. This guide gives AI, data, and platform engineers a concrete design for building representative evaluations without carrying raw production data through the test stack.

Why sanitized production examples still leak

Useful evaluations need representative cases. The OpenAI evaluation guide recommends task-specific evaluations built from production data, historical logs, expert examples, and edge cases. Anthropic's evaluation guidance likewise starts with clear success criteria and representative test cases. Real failures expose conditions that invented happy paths miss.

Copying a production trace creates several records. A typical evaluation run may persist:

• the original fixture in a repository or dataset service;
• a rendered prompt assembled from several fixture fields;
• retrieved documents used as fixed context;
• fake or replayed tool responses;
• the candidate model's answer and action trace;
• a grader prompt containing the input and answer;
• grader reasoning, annotations, and reviewer comments;
• CI logs, experiment dashboards, exports, and backups.

Cleaning only the user message leaves the rest exposed. A support case without a customer's name may still contain a unique order number, delivery address, account notes, or a tool response that makes the person identifiable. NIST's de-identification guidance frames the problem around whether information can still be linked to an individual, not whether obvious names were deleted. The ICO introduction to anonymisation also distinguishes anonymisation from pseudonymisation and treats identification risk as contextual.

A fixture with reversible tokens is pseudonymized, not anonymous. A fixture with synthetic names can still identify someone if the unusual narrative, exact timestamp, and order value remain unchanged. A privacy-safe fixture has residual risk that fits its permitted use and controls, regardless of whether a scanner found zero email addresses.

Define the evaluation purpose before sampling

Start with one workflow and one release decision. Do not build a central dump of interesting traces and decide how to use them later. A narrow purpose tells you which fields the evaluation actually needs.

Suppose a support workflow must classify a request, retrieve an order, and decide whether a refund requires approval. The release test may need the issue type, order state, refund value band, customer message structure, permitted tools, and required approval result. It probably does not need the customer's real name, email, exact address, account creation date, or original order identifier.

Write a field contract with four decisions for every source field:

1. Drop fields that are irrelevant to the assertion.
2. Generalize values when a category is sufficient, such as replacing an exact age with an age band.
3. Synthesize values when relationships or formats matter, such as consistent fake identifiers.
4. Restrict raw values that cannot be transformed without destroying the case.

The restricted category should be rare. It belongs in a tightly controlled diagnostic tier, not in the portable suite used by developer laptops, hosted graders, or ordinary CI jobs. If a case needs the original payload only to reproduce a parser defect, keep that raw diagnostic artifact under a short retention period and derive a portable fixture once the failure is understood.

Request-time PII filtering protects one model call. An evaluation fixture is repeatedly copied, edited, reviewed, and replayed, so its policy must cover the complete lifecycle.

Map every derivative before choosing tools

Draw the evaluation data flow from source incident to deletion. Include the fixture builder, repository, object store, evaluator, model provider, trace backend, annotation interface, CI runner, export job, and backup system. For each boundary, record:

• which fields enter and leave;
• whether content is stored and for how long;
• who and which service identity can read it;
• whether another tenant shares the system;
• where the data is processed and stored;
• how a deletion request reaches that copy;
• whether logs capture request or response bodies by default.

Do not assume the hosted evaluation tool sees only the final prompt. A grader often receives the source input, expected behavior, candidate response, retrieved context, and grading rubric in one request. A tracing integration may capture all of those again. Disable body capture where it is not required, or send only the portable fixture tier to that service.

Give each derived case a stable internal identifier and a source reference that does not reveal the production record. Keep the production lookup in a separate restricted mapping with its own access policy and expiry. The portable dataset needs to know that a fixture came from a confirmed incident and which transformation policy produced it. It does not need the customer's database key.

Build relationship-preserving synthetic fixtures

Blind masking often destroys the condition the test must preserve. Replacing every identifier with [REDACTED] prevents the workflow from distinguishing two people, validating an order format, or checking whether a tool used the current request's identifier instead of a stale one.

Use typed synthetic values generated within each fixture. Equal source values should map to equal synthetic values inside the case. Different source values should remain different. Preserve only the properties required by the test, such as format, ordering, category, or cross-field equality.

fixture_id: refund-approval-014
purpose: test-refund-approval-boundary
source_class: confirmed-production-escape
transformation_policy: support-eval-v3
input:
  customer_name: "Priya Shah"
  customer_email: "[email protected]"
  order_id: "ORD-730184"
  message: "Refund ORD-730184. The item arrived damaged."
context:
  refund_amount_gbp: 750
  approval_limit_gbp: 500
expected:
  required_tool: get_order
  forbidden_tool: issue_refund
  requires_approval: true
privacy:
  portable: true
  raw_source_retained: false
  sensitive_classes: [person, email, account_identifier]

These values are fictional. The reserved .test domain prevents accidental delivery, and the order identifier matches the application's accepted shape without retaining the real value. The repeated synthetic order ID lets the test detect a stale or mismatched tool argument.

For free text, rewrite the minimum span needed to remove identifying detail while keeping the failure mechanism. If the original defect depended on a misspelled street name confusing an entity detector, create another plausible misspelling with the same token pattern. If the defect depended on two people sharing a surname, retain that relationship with fictional names. Record which properties were deliberately preserved so a later editor does not simplify away the reason the fixture exists.

Treat detectors as inputs to policy

Automated scanners help find sensitive spans, but they do not decide whether a case is safe. Detection errors vary by language, domain, and data shape. In Presidio issue 1498, users reported a URL recognizer flagging ordinary content in code snippets. That author report is a useful reminder that a detector can both miss protected content and remove material the task needs.

Run several checks during fixture creation:

• deterministic scans for credentials, payment patterns, emails, phone numbers, and identifiers controlled by your application;
• entity recognition configured for the fixture's language and domain;
• exact comparison against sensitive source fields before the raw record leaves the restricted builder;
• rarity checks for timestamps, locations, free-text phrases, and unusual value combinations;
• a human review for high-risk or hard-to-transform cases.

Fail closed when a required scanner is unavailable. Do not let the builder silently emit a portable fixture because one detection service timed out. Store detection classes and policy decisions, but avoid writing raw findings into ordinary logs.

Treat a scanner pass as one input to the release decision. The release gate must also test task fidelity.

Separate restricted and portable fixture tiers

Use two explicit storage tiers instead of applying one vague "test data" label.

The restricted diagnostic tier holds the minimum raw evidence needed to investigate a new failure. Limit access to named responders, encrypt the store, log reads, block third-party graders, and enforce short retention. A case leaves this tier when engineers have reproduced the mechanism and created a portable replacement.

The portable evaluation tier contains transformed fixtures approved for normal development and CI. It may be version controlled if policy allows, but it still needs owners, review dates, and deletion handling. Synthetic values do not make the surrounding business narrative harmless by default.

Promote a case only when all of these statements are true:

• every field has a documented purpose;
• raw identifiers and secrets are absent;
• preserved relationships are listed;
• the transformed case still reproduces the failure;
• grader and trace payloads have been inspected;
• the fixture passes privacy scans and human review required by its risk class;
• retention and deletion owners are assigned.

Do not copy a restricted trace into a ticket to ask for promotion approval. The approval record should reference the internal case identifier and transformation report.

Test privacy and evaluation fidelity together

A fixture pipeline needs two independent gates. A privacy gate checks that protected source values and disallowed identifying combinations are absent from every emitted derivative. A fidelity gate checks that the transformed fixture still exercises the behavior the team intends to measure.

For the support example, privacy checks should search the fixture, rendered prompt, tool responses, candidate trace, grader prompt, grader output, and CI logs. Fidelity checks should confirm that the workflow reads ORD-730184, compares 750 with the 500 limit, requests approval, and does not call issue_refund.

Run the accepted baseline and a known-bad candidate against the transformed case. If both pass, the transformation removed the useful signal or the assertion is weak. If both fail, the rewrite may have changed the task. The existing Fire in Belly regression-testing guide explains how fixed cases, deterministic assertions, trajectory checks, and release thresholds work together. The privacy-specific addition is to validate the emitted data at every evaluation boundary, not only the fixture file.

Add adversarial privacy tests too. Seed fake credentials in a retrieved passage and confirm the fixture builder rejects or replaces them. Put distinct synthetic customer IDs in the current input and prior conversation, then confirm the workflow uses the current one. Include multilingual names, OCR errors, code snippets, and values that look like identifiers but are ordinary domain terms. These tests measure detector errors without exposing a real person.

Handle retention, correction, and deletion

Every fixture needs an owner, purpose, source class, transformation-policy version, creation date, review date, and retention rule. Cases based on confirmed failures may remain useful for years, but usefulness does not cancel a deletion obligation or a policy change.

Maintain a protected lineage mapping from the production source to every restricted and portable derivative. When a source record must be deleted, the workflow should locate fixtures, trace payloads, annotation records, exports, and backup expiry schedules. If a portable fixture is demonstrably independent of the person after transformation, policy may permit retention. Record that decision instead of assuming it.

Rebuild fixtures when transformation rules improve. A new detector or risk model can reveal that old cases retained rare phrases or quasi-identifiers. Scan the full portable corpus under the new policy, quarantine failures, and rerun baseline evaluations after repair. Privacy changes can alter test semantics, so they need the same review as any other dataset revision.

Verify the complete pipeline

Before opening the portable dataset to ordinary CI, run a controlled drill:

1. Select one restricted production failure and create a transformed fixture.
2. Inspect every persisted request and response in the evaluator and tracing stack.
3. Search all derivatives for the original sensitive values and known variants.
4. Prove the transformed case still catches the original workflow defect.
5. Revoke the builder's source access and confirm portable evaluation still works.
6. Trigger deletion by source reference and verify every mapped derivative is removed or scheduled for documented backup expiry.
7. Review the evidence with someone who did not build the transformer.

Stop the rollout if a grader receives raw context, a trace backend retains prompts unexpectedly, a detector failure emits a case, or the transformed fixture no longer distinguishes the good build from the bad one.

Build the first safe dataset

Choose one internal workflow and five confirmed failures. Keep raw examples in a restricted diagnostic store. For each case, write the purpose and field contract, create relationship-preserving synthetic values, and inspect the exact payloads sent to models, graders, and tracing systems. Add privacy and fidelity checks to the same CI job.

Do not scale the corpus until one deletion drill removes every derivative and one known-bad workflow still fails the transformed suite. That gives the team an LLM evaluation dataset that can learn from production without becoming an unmanaged copy of production.

References

1. OpenAI evaluation guide: supports task-specific evaluations, representative data sources, graders, and continuous evaluation.
2. Anthropic evaluation guidance: supports defining success criteria and representative test cases before selecting evaluation methods.
3. NIST de-identification guidance: supports assessing whether transformed information can still be linked to an individual.
4. ICO introduction to anonymisation: supports the distinction among anonymisation, pseudonymisation, and contextual identification risk.
5. Presidio issue 1498: supplies a practitioner report about false-positive detection in code-like content.
6. Fire in Belly AI workflow regression testing: supplies the first-party baseline for representative fixtures, trajectory checks, and release gates.

---