Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ jobs:
run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v8.jsonl --config regex+known --min-recall 0.90
- name: Blind benchmark v9 (regex+known must stay at or above 90% recall)
run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v9.jsonl --config regex+known --min-recall 0.90
- name: Blind benchmark v10 (regex+known must stay at or above 90% recall)
run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v10.jsonl --config regex+known --min-recall 0.90

e2e:
runs-on: ubuntu-latest
Expand Down
30 changes: 15 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ docker compose -f deploy/docker-compose.yml up --build
| `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex + local model |
| `EMPLOYER` | "I work at the Tim Hortons on Barton" | local model |

Names are found from context in English, French, Spanish, Polish and Hebrew without a model: self-introductions (also lowercase, as in call transcripts), relatives with or without "my", chart labels, sign-offs and signature blocks, email headers, and full names next to a date of birth or chart number. Names with no cue word need the local model or your records. Dates are also matched in French, Spanish, Polish and Hebrew, written or spoken ("le 12 mai 1990", "el catorce de febrero de mil novecientos noventa y uno", "trzeciego lipca", "5 בנובמבר"), and spoken digits in those languages. Numbers read out loud in calls are rebuilt before matching: grouped and compound number words in all five languages ("eight hundred, five five five, oh one four two", "dziewięćset pięć, pięćset pięćdziesiąt pięć..."), spelled letters and the phonetic alphabet ("T three A, four B seven", "november sierra hotel four zero zero..."), so spoken phone numbers, postal codes, health cards, member and donor codes, and street numbers ("eighteen forty two west cermak road") are masked. Canadian and US cities and towns over 15,000 people (from [GeoNames](https://www.geonames.org/), CC BY 4.0) are masked after a location cue ("I live in Sudbury", "j'habite à Laval", "mieszkam w Toronto"); names that are also common words, such as Normal or Mobile, must be capitalized. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`. Machine-shaped text is read by its field names: JSON keys (`"last_name"`, `"patientName"`, `"zip"`), `key: value` dumps (`preferred_name: Ana`, `Nom : Tremblay`), CSV columns under a header row (`last,first,dob,member_id`), and HL7 v2 PID, NK1, GT1 and IN1 segments; each value must still look like what its field says, so `Phone: press 1` or `Last name: TBD` are left alone.
Names are found from context in English, French, Spanish, Polish and Hebrew without a model: self-introductions (also lowercase, as in call transcripts), relatives with or without "my", chart labels, sign-offs and signature blocks, email headers, and full names next to a date of birth or chart number. Names with no cue word need the local model or your records. Dates are also matched in French, Spanish, Polish and Hebrew, written or spoken ("le 12 mai 1990", "el catorce de febrero de mil novecientos noventa y uno", "trzeciego lipca", "5 בנובמבר"), and spoken digits in those languages. Numbers read out loud in calls are rebuilt before matching: grouped and compound number words in all five languages ("eight hundred, five five five, oh one four two", "dziewięćset pięć, pięćset pięćdziesiąt pięć..."), spelled letters and the phonetic alphabet ("T three A, four B seven", "november sierra hotel four zero zero..."), so spoken phone numbers, postal codes, health cards, member and donor codes, and street numbers ("eighteen forty two west cermak road") are masked. Canadian and US cities and towns over 15,000 people (from [GeoNames](https://www.geonames.org/), CC BY 4.0) are masked after a location cue ("I live in Sudbury", "j'habite à Laval", "mieszkam w Toronto"); names that are also common words, such as Normal or Mobile, must be capitalized. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`. Machine-shaped text is read by its field names: JSON keys (`"last_name"`, `"patientName"`, `"zip"`), `key: value` dumps (`preferred_name: Ana`, `Nom : Tremblay`), CSV columns under a header row (`last,first,dob,member_id`), and HL7 v2 PID, NK1, GT1 and IN1 segments; each value must still look like what its field says, so `Phone: press 1` or `Last name: TBD` are left alone. Emails written to dodge filters ("ana [at] example [dot] org") or spoken in five languages, and social handles and usernames (as `ID`), are masked. A clinic address on the allow list also matches its common spellings (Ave and Avenue, Suite and #), and a name found once is masked on its later mentions in the same message. Invisible characters (zero-width spaces, bidi marks) are ignored when matching.

### Optional local model

Expand All @@ -154,31 +154,31 @@ Presidio is also supported: `pip install phiproxy[presidio]` and add `PresidioDe

## Benchmark

`benchmark/` holds ten synthetic **patient communications** sets: SMS, web chat, call transcripts and voicemails, emails, referral letters, insurance letters and intake text, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented.
`benchmark/` holds eleven synthetic **patient communications** sets: SMS, web chat, call transcripts and voicemails, emails, referral letters, insurance letters and intake text, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented.

- **v0** (288 messages, English): template-based, written by the detector authors.
- **blind v1 to v9** (420, 400, 360, 400, 400, 400, 400, 400 and 400 messages): English, French, Spanish, Polish, Hebrew and mixed, each written by an AI model (Opus, Sonnet, Fable, then Opus, Fable, Opus, Opus, and four Sonnet authors for v8 and for v9, each set with a new brief) that never saw the detector code, each with about 30% of messages containing no identifiers but full of lookalikes (drug doses, embryo grades, billing codes). v3 is US-heavy and mentions people without cue words ("Can Omar come in with me?"). v4 comes from a British Columbia clinic with international patients, and has spoken and corrected numbers, scanned faxes, internal staff notes, the clinic's own staff roster, and patients who share a staff member's name. v5 comes from a New Jersey clinic, with conversations with the clinic's AI assistant, names from many backgrounds and US insurance paperwork. v6 comes from an Arizona clinic with a Winnipeg office: a quarter calls, first names with no cue word, staff handoffs and internal chat, cross-border surrogacy; its author generated messages from templates, so it is less varied than the others. v7, written by hand, comes from a Houston clinic with a Montreal office: email threads, intake forms, male factor and LGBTQ+ families, specialty-pharmacy calls. v8, written by hand by four authors, comes from a Miami clinic with a Toronto office: phone typing and autocorrect, pasted screenshots, group texts on shared phones, relatives over 89, military and student coverage. v9, also written by four authors, comes from a Denver clinic with a Calgary office: record-system dumps, CSV, JSON and HL7 text, long legal and clinical documents, names that look like words or places, and unusual identifier formats. A second annotator, also blind to the code, corrected the labels.
- **blind v1 to v10** (420, 400, 360, 400, 400, 400, 400, 400, 400 and 400 messages): English, French, Spanish, Polish, Hebrew and mixed, each written by an AI model (Opus, Sonnet, Fable, then Opus, Fable, Opus, Opus, and four Sonnet authors for v8, v9 and v10, each set with a new brief) that never saw the detector code, each with about 30% of messages containing no identifiers but full of lookalikes (drug doses, embryo grades, billing codes). v3 is US-heavy and mentions people without cue words ("Can Omar come in with me?"). v4 comes from a British Columbia clinic with international patients, and has spoken and corrected numbers, scanned faxes, internal staff notes, the clinic's own staff roster, and patients who share a staff member's name. v5 comes from a New Jersey clinic, with conversations with the clinic's AI assistant, names from many backgrounds and US insurance paperwork. v6 comes from an Arizona clinic with a Winnipeg office: a quarter calls, first names with no cue word, staff handoffs and internal chat, cross-border surrogacy; its author generated messages from templates, so it is less varied than the others. v7, written by hand, comes from a Houston clinic with a Montreal office: email threads, intake forms, male factor and LGBTQ+ families, specialty-pharmacy calls. v8, written by hand by four authors, comes from a Miami clinic with a Toronto office: phone typing and autocorrect, pasted screenshots, group texts on shared phones, relatives over 89, military and student coverage. v9, also written by four authors, comes from a Denver clinic with a Calgary office: record-system dumps, CSV, JSON and HL7 text, long legal and clinical documents, names that look like words or places, and unusual identifier formats. v10, also written by four authors, comes from a Minneapolis clinic with a Winnipeg office: calls through phone interpreters, numbers and emails spoken as words, and identifiers people half-hide or obfuscate ("ana [at] example [dot] org", "SSN ends in 4417", social handles). A second annotator, also blind to the code, corrected the labels.

Each blind set was scored once with the detectors frozen, then used to find gaps. So each set's first score is a test of the fixes made on the sets before it:

| recall, first score on unseen text | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 |
|---|---|---|---|---|---|---|---|---|---|
| regex | 36.0% | 57.3% | 58.4% | 66.1% | 73.7% | 80.2% | 75.7% | 77.0% | 73.7% |
| regex + known entities | 54.1% | 80.2% | 68.3% | 73.8% | 80.9% | 88.8% | 84.9% | 87.2% | 84.4% |
| regex + local model | 77.3% | 87.9% | 84.2% | 86.5% | 91.2% | 96.8% | 93.1% | 93.4% | 91.6% |
| regex + known + local model | 79.2% | 92.3% | 88.1% | 88.4% | 93.1% | 97.5% | 94.7% | 95.4% | **93.4%** |
| recall, first score on unseen text | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 |
|---|---|---|---|---|---|---|---|---|---|---|
| regex | 36.0% | 57.3% | 58.4% | 66.1% | 73.7% | 80.2% | 75.7% | 77.0% | 73.7% | 70.0% |
| regex + known entities | 54.1% | 80.2% | 68.3% | 73.8% | 80.9% | 88.8% | 84.9% | 87.2% | 84.4% | 81.3% |
| regex + local model | 77.3% | 87.9% | 84.2% | 86.5% | 91.2% | 96.8% | 93.1% | 93.4% | 91.6% | 86.6% |
| regex + known + local model | 79.2% | 92.3% | 88.1% | 88.4% | 93.1% | 97.5% | 94.7% | 95.4% | 93.4% | **89.7%** |

| first score, regex + known + local model | v7 | v8 | v9 |
| first score, regex + known + local model | v8 | v9 | v10 |
|---|---|---|---|
| by channel | SMS 100%, chat 95%, email 93%, notes 96%, calls 95% | SMS 96%, chat 97%, email 96%, notes 94%, calls 94% | SMS 95%, chat 93%, email 94%, notes 93%, calls 92% |
| by language | en 95%, fr 95%, es 94%, pl 95%, he 87%, mixed 100% | en 96%, fr 97%, es 96%, pl 92%, he 84%, mixed 97% | en 94%, fr 95%, es 95%, pl 92%, he 85%, mixed 93% |
| precision (default / clinic details allowed / plus staff list) | 66.6% / 72.5% / 78.4% | 74.9% / 86.1% / 88.3% | 73.0% / 82.2% / 85.2% |
| messages without identifiers where something was masked | 40% / 34% / 32% | 54% / 30% / 30% | 60% / 37% / 36% |
| by channel | SMS 96%, chat 97%, email 96%, notes 94%, calls 94% | SMS 95%, chat 93%, email 94%, notes 93%, calls 92% | SMS 93%, chat 91%, email 89%, notes 93%, calls 86% |
| by language | en 96%, fr 97%, es 96%, pl 92%, he 84%, mixed 97% | en 94%, fr 95%, es 95%, pl 92%, he 85%, mixed 93% | en 90%, fr 95%, es 93%, pl 87%, he 81%, mixed 85% |
| precision (default / clinic details allowed / plus staff list) | 74.9% / 86.1% / 88.3% | 73.0% / 82.2% / 85.2% | 72.4% / 80.8% / 83.8% |
| messages without identifiers where something was masked | 54% / 30% / 30% | 60% / 37% / 36% | 68% / 47% / 45% |

**How to read this honestly:**
- Expect real traffic to score lower: these sets are AI-written, not written by clinic staff. v0 overstated recall by about 30 points.
- Without the local model, names not in your records are the main gap. On v3, where names often come with no cue, the pattern rules alone found 43% of names.
- After the gaps found on each set were fixed, the current code scores 95% to 99% with the local model on the nine blind sets. Those numbers are optimistic, which is why the table above shows first scores. v4 showed how much: the code that scored 95% to 97% on v1 to v3 scored 88% on v4. After the Hebrew, call and date work, the next unseen set (v5) scored 93%, v6 then scored 97.5%, but it was built from templates and is probably easier; v7, written by hand, scored 94.7%, v8 95.4%, and v9 93.4%: its brief aimed at text the rules had never seen (JSON, CSV and HL7 dumps, court and case numbers, Alberta health numbers, nicknames). Each set's brief differs, so expect roughly 90% to 95% on new text of this kind, and less on real traffic.
- After the gaps found on each set were fixed, the current code scores 96% to 99% with the local model on the ten blind sets. Those numbers are optimistic, which is why the table above shows first scores. v4 showed how much: the code that scored 95% to 97% on v1 to v3 scored 88% on v4. After the Hebrew, call and date work, the next unseen set (v5) scored 93%, v6 then scored 97.5%, but it was built from templates and is probably easier; v7, written by hand, scored 94.7%, v8 95.4%, v9 93.4%, and v10 89.7%: its brief aimed at text the rules had never seen (JSON, CSV and HL7 dumps, court and case numbers, Alberta health numbers, nicknames), and v10's went after spoken, obfuscated and half-hidden forms (emails written "ana [at] example [dot] org" or read out loud, numbers read one digit at a time, "SSN ends in 4417", social handles). Each set's brief differs, so expect roughly 90% to 95% on new text of this kind, and less on real traffic.
- Over-masking is real: v6, full of staff talking about staff, had 55% of its clean messages masked with the clinic and staff lists at first; the staff-list cues added after it bring that to 34%. Most of it is the clinic's own contact details (pass them in `Policy(allow=...)`), staff names (pass them in `Policy(staff=...)`), and holiday dates, which stay masked on purpose.
- Details, label rules, the corrections and full tables are in [`benchmark/README.md`](https://github.com/DecisionTreeTechnology/phiproxy/blob/main/benchmark/README.md).

Expand Down
Loading
Loading