Skip to content
Outrich Data

How to read a de-identification benchmark before you trust a score

Published 2026-11-17 · Updated 2026-11-17 · 6 min read · By the Outrich Data team

A micro1 research paper dated September 24, 2026 reports de-identification scores on a benchmark its own authors built. The scores are useful for comparing methods, but they do not show that your files are safe to license. A company still has to test its own sample and approve each package in writing.

This page is general information, not legal advice. Whether your company can license a given dataset depends on your contracts, your privacy notices and the law where you and your customers are. Review it with your own counsel before signing anything.

What was published

On September 24, 2026, micro1 published a research post on PII transformation for enterprise datasets. It describes a system, which the authors call flow-transform 1.0, that replaces real identities with consistent synthetic ones across a whole dataset. The authors argue that anonymization should be treated as a transformation of the whole corpus, not as a hunt for individual spans of text.

The post also introduces two things the authors made themselves: a Transformation Quality Index, and a synthetic tabular test set they call the Enterprise De-Identification Bench. We read the page on November 17, 2026. Every figure below is the authors' own, as stated on that page, and we have not checked or reproduced any of it.

We cover this because a company that licenses data will hear scores like these in sales conversations. Knowing what a score does and does not tell you is part of approving a dataset. Our guide on how to anonymize company data covers the steps themselves. This post covers how to judge the claims made about them.

The scores as reported

On the authors' own enterprise benchmark, the paper compares three systems. The first is an open baseline, NVIDIA's NeMo Anonymizer. The second is flow-transform without an extra review step. The third is flow-transform with an optional agentic review layer.

Measure on the authors' enterprise benchNeMo baselineFlow-transformWith agentic review
Transformation Quality Index (higher is better)74.984.188.9
Detection F1 (higher is better)0.6780.7580.829
Decoy transformation rate (lower is better)72.3%65.2%40.0%

Scores on outside test sets

The paper also reports results on test sets built by others. These carry more weight than a benchmark the authors designed, because the authors did not pick the questions.

Test setFlow-transformComparison reported
PrivacyBench F196.00%94.5% for Tonic Textual
PrivacyBench combined transformation accuracy95.46%92.0% for Tonic Textual combined with a general model
Nemotron-PII F188.33%87.84% for NeMo Anonymizer
AI4Privacy F183.74%73.08% for NeMo Anonymizer

What the paper says about its own limits

The authors are direct about several limits, and it is worth reading them before reading the scores. The main benchmark is synthetic and covers tabular data only. Two of its measures, Fidelity and Coverage, score 1.0 for every system, so they do not separate one system from another. The index uses weights the authors chose, and the authors note that small differences in it need a sensitivity analysis.

The paper also acknowledges gaps with complex file formats, multimodal data and ambiguous identity linkage. The results are reported by the vendor and have not been checked by an independent party, as far as the page shows.

None of this is a criticism. It is how a first paper on a new benchmark usually reads. It does mean the headline numbers answer a narrow question: how well does a method do on this set of test records. That is different from how well it will do on your support tickets.

Five questions to put to any score

These apply to any de-identification claim, from any vendor or buyer, including ours. Take them to a call before you approve a package.

  • Who built the test set. A set built by the vendor is a fair way to compare versions of its own method. A set built by others is a fairer way to compare vendors.
  • What kind of data was in it. Tabular rows with clean columns are far easier than free text, where a name can sit inside a sentence, a signature block or a pasted screenshot. Enterprise data is mostly free text.
  • What the metric counts. Detection counts how many identifiers were found. Transformation asks whether each one was replaced correctly and consistently. A record can be detected and still be replaced badly.
  • What happens to the misses. Every method misses some identifiers. The useful figure for a company is how many were left in, and what a reviewer does about them.
  • Whether anyone outside the vendor has checked. Independent testing is rare for this kind of claim. Where it is missing, say so in your own file and test a sample yourself.

Why consistent replacement matters, and where it can backfire

The idea at the center of the paper is a sound one. If a customer named in one thread is replaced with a different fake name in the next, a model trained on the data learns a broken picture of the account. Consistent replacement keeps the structure of the work intact while removing the real identity.

Consistency has a cost that a company should understand. A stable pseudonym is still a link across records. If someone can match the pseudonymized records to an outside source, for example by a rare job title, an unusual transaction amount or a distinctive date, the link can lead back to the person. That is why de-identification and anonymization are different standards, a point we set out in de-identification vs anonymization. The free re-identification risk checker tests a structured file for exactly this kind of exposure.

Which standard applies depends on the law and on the contract. HIPAA, the CCPA and the GDPR each define the term differently. This is general information, not legal advice, and counsel should confirm the standard for your data.

What a company should do with a sample, whatever the score

A benchmark is a reason to shortlist a method. It is not a substitute for looking at your own data. The work below takes a few hours and tells you more than any published figure.

  • Pull a sample of a few hundred records from each system, chosen at random and not chosen for being tidy.
  • Run the free PII scrubber over it in your browser and read what it flags and what it leaves alone. Nothing is uploaded.
  • Have a person read the output line by line. Look for names inside sentences, account numbers in free text, email signatures and quoted replies.
  • Count what remains and write the count down. A documented rate of misses is more honest than a claim of none.
  • Repeat on a second sample after any fix, so you can show that the fix worked.
  • Keep the results with the package. A data card has a place to state the method used and the identifier classes covered.

Where this fits in a data sale

At Outrich Data, personal and client-identifying details are removed before any dataset is shown to a buyer, and you review and approve the packaged dataset in writing before delivery. The method is documented, and delivery is verified against the checksum manifest before escrowed funds are released. See how it works and escrow for data transactions.

That approval step is the point of this post. A score from any vendor is an input to your decision. The decision, and the signature, stay with the company that owns the data and the counsel who advises it.

Common questions

Does a high benchmark score mean my data is anonymous?
No. A score measures performance on a test set. It does not show that your records are safe to license. Anonymous is a legal standard that depends on the law and the contract, so test your own sample and have counsel confirm.
What is a decoy transformation rate?
The paper reports it as a measure where lower is better. In plain terms it counts text that looked like an identifier but should have been left alone, and that was changed anyway. Read the paper's own definition before comparing it with other metrics.
Why do outside test sets matter more?
The people who built the method did not choose the questions. A result on a set built by others is harder to tune for, so it is a fairer way to compare one vendor with another.
Should I rely on a vendor's own benchmark?
Use it to compare versions of that vendor's own method, and to learn how it fails. For a decision about your data, rely on a sample from your own systems that a person has read.
Does Outrich Data publish a benchmark?
No. We do not publish de-identification scores. We document the method for each dataset, let you review the package before any buyer sees it, and ask for your written approval.

Find out if your data qualifies.

Eleven questions, about five minutes, no call required. You get an answer on screen and a written follow-up.

esc