diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d2c9b4f..4e75697 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,6 +47,8 @@ jobs: run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v11.jsonl --config regex+known --min-recall 0.90 - name: Blind benchmark v12 (regex+known must stay at or above 90% recall) run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v12.jsonl --config regex+known --min-recall 0.90 + - name: Blind benchmark v13 (regex+known must stay at or above 90% recall) + run: python benchmark/score.py --data benchmark/data/patient_comms_blind_v13.jsonl --config regex+known --min-recall 0.90 e2e: runs-on: ubuntu-latest diff --git a/README.md b/README.md index 74126a4..724836b 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,9 @@ When the reply comes back, phiproxy swaps the stand-ins for the real values, so ## What makes it different - **Your own records as a detector.** Pass the identifiers you already hold for the patient (name, DOB, phone, health card). Exact and one-typo matching against them catches lowercase names and misspellings that pattern matching misses. On the blind benchmark below this lifts recall from 57% to 80%. -- **Optional local model.** `phiproxy[gliner]` adds a small named-entity model that runs on your own machine and finds names, cities and workplaces that no rule or record covers ("my husband ravi", "I live in Sudbury"). With it, recall on the newest blind set, scored once before any fix, was 93%, at about 50 ms per message on CPU. -- **Your clinic's own details pass.** `Policy(allow=["Harborview Reproductive Centre", "613-555-0100", "@clinic.example.org"])` lets the clinic's name, phone numbers, address and email domain reach the model unmasked, so answers keep them. Anything that matches the patient's own record is still masked. -- **Your staff pass, patients who share their names do not.** `Policy(staff=["Jocelyn Paré", "Will Barnes"])`: a staff member's full name passes anywhere; a first name or surname alone passes only next to a staff cue ("Jocelyn in billing", "AGENT: ... this is Will", "call Hsu at ext. 2240", "Sig: T. Kowalski", "Attendees: Hsu, Reyes", "Dr. Paré and Will", "it's Grace at Harborview Fertility"). "Will is picking me up" stays masked, because a patient's husband can be called Will too. +- **Optional local model.** `phiproxy[gliner]` adds a small named-entity model that runs on your own machine and finds names, cities and workplaces that no rule or record covers ("my husband ravi", "I live in Sudbury"). With it, recall on the newest blind set, scored once before any fix, was 96%, at about 50 ms per message on CPU. +- **Your clinic's own details pass.** `Policy(allow=["Harborview Reproductive Centre", "613-555-0100", "@clinic.example.org"])` lets the clinic's name, phone numbers, address and email domain reach the model unmasked, so answers keep them. Anything that matches the patient's own record is still masked. Links are narrower: a link written without "https" passes only as the bare clinic address (with or without "www" or a part such as "portal") or a short generic page ("/faq", "/contact"). A patient's subdomain, a profile path or a record number in the path is masked, and an allow entry with a path (a clinic's social page) does not open the whole site. +- **Your staff pass, patients who share their names do not.** `Policy(staff=["Jocelyn Paré", "Will Barnes"])`: a staff member's full name passes anywhere; a first name or surname alone passes only next to a staff cue ("Jocelyn in billing", "AGENT: ... this is Will", "call Hsu at ext. 2240", "Sig: T. Kowalski", "Attendees: Hsu, Reyes", "Dr. Paré and Will", "it's Grace at Harborview Fertility"). "Will is picking me up" stays masked, because a patient's husband can be called Will too. A full name that is not on the list does not let the same first name pass elsewhere, and a repeated name passes only where a staff name would pass on its own. - **Stable tokens per conversation.** The same value gets the same token on every turn of a session, so the model can follow who is who and provider prompt caching keeps working. Tokens are sequential, not derived from the value (see 45 CFR 164.514(c)). - **Fail closed.** If masking errors, or a request holds media the policy does not allow, the request is rejected and nothing is sent. Clients cannot switch the guardrail off through request metadata. - **Clinic-specific identifiers.** Canadian health card numbers (OHIP, RAMQ), SIN, donor and embryo IDs, spoken numbers and dates from call transcripts. @@ -133,7 +133,7 @@ docker compose -f deploy/docker-compose.yml up --build | `PROVIDER` | "Dr. Patel" (passes by default, as clinician names are not patient identifiers; enable with `Policy(disabled_types=set())`) | regex + local model | | `EMPLOYER` | "I work at the Tim Hortons on Barton" | local model | -Names are found from context in English, French, Spanish, Polish and Hebrew without a model: self-introductions (also lowercase, as in call transcripts), relatives with or without "my", chart labels, sign-offs and signature blocks, email headers, and full names next to a date of birth or chart number. Names with no cue word need the local model or your records. Dates are also matched in French, Spanish, Polish and Hebrew, written or spoken ("le 12 mai 1990", "el catorce de febrero de mil novecientos noventa y uno", "trzeciego lipca", "5 בנובמבר"), and spoken digits in those languages. Numbers read out loud in calls are rebuilt before matching: grouped and compound number words in all five languages ("eight hundred, five five five, oh one four two", "dziewięćset pięć, pięćset pięćdziesiąt pięć..."), spelled letters and the phonetic alphabet ("T three A, four B seven", "november sierra hotel four zero zero..."), so spoken phone numbers, postal codes, health cards, member and donor codes, and street numbers ("eighteen forty two west cermak road") are masked. Canadian and US cities and towns over 15,000 people (from [GeoNames](https://www.geonames.org/), CC BY 4.0) are masked after a location cue ("I live in Sudbury", "j'habite à Laval", "mieszkam w Toronto"); names that are also common words, such as Normal or Mobile, must be capitalized. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`. Machine-shaped text is read by its field names: JSON keys (`"last_name"`, `"patientName"`, `"zip"`), `key: value` dumps (`preferred_name: Ana`, `Nom : Tremblay`), CSV columns under a header row (`last,first,dob,member_id`), and HL7 v2 PID, NK1, GT1 and IN1 segments; each value must still look like what its field says, so `Phone: press 1` or `Last name: TBD` are left alone. Emails written to dodge filters ("ana [at] example [dot] org") or spoken in five languages, and social handles and usernames (as `ID`), are masked. A clinic address on the allow list also matches its common spellings (Ave and Avenue, Suite and #), and a name found once is masked on its later mentions in the same message. Phone numbers given in pieces across lines or turns ("area code 404", then "555", then "0182"), a patient's initials after a patient cue ("pt KJ"), payment handles and bank numbers (Venmo, Cash App, transit and routing numbers), and rural routes, GPS coordinates, counties and reserves are masked too. So are travel booking codes ("booking ref K7QX2M") and e-ticket numbers, passport numbers and foreign national IDs (Aadhaar, CURP and others), ages over 89 in family histories ("grandmother, 91", "d. 2011 at 93"), names spelled with Spanish or French letter names ("erre o ese a"), and foreign addresses and postcodes (UK, India, the Philippines, Mexico). Invisible characters (zero-width spaces, bidi marks) are ignored when matching. +Names are found from context in English, French, Spanish, Polish and Hebrew without a model: self-introductions (also lowercase, as in call transcripts), relatives with or without "my", chart labels, sign-offs and signature blocks, email headers, and full names next to a date of birth or chart number. Names with no cue word need the local model or your records. Dates are also matched in French, Spanish, Polish and Hebrew, written or spoken ("le 12 mai 1990", "el catorce de febrero de mil novecientos noventa y uno", "trzeciego lipca", "5 בנובמבר"), and spoken digits in those languages. Numbers read out loud in calls are rebuilt before matching: grouped and compound number words in all five languages ("eight hundred, five five five, oh one four two", "dziewięćset pięć, pięćset pięćdziesiąt pięć..."), spelled letters and the phonetic alphabet ("T three A, four B seven", "november sierra hotel four zero zero..."), so spoken phone numbers, postal codes, health cards, member and donor codes, and street numbers ("eighteen forty two west cermak road") are masked. Canadian and US cities and towns over 15,000 people (from [GeoNames](https://www.geonames.org/), CC BY 4.0) are masked after a location cue ("I live in Sudbury", "j'habite à Laval", "mieszkam w Toronto"); names that are also common words, such as Normal or Mobile, must be capitalized. Code-shaped identifiers without a keyword ("SL-4471920-03", "W123456789", any bare number of 7+ digits) are masked as `ID`. Machine-shaped text is read by its field names: JSON keys (`"last_name"`, `"patientName"`, `"zip"`), `key: value` dumps (`preferred_name: Ana`, `Nom : Tremblay`), CSV columns under a header row (`last,first,dob,member_id`), and HL7 v2 PID, NK1, GT1 and IN1 segments; each value must still look like what its field says, so `Phone: press 1` or `Last name: TBD` are left alone. Emails written to dodge filters ("ana [at] example [dot] org") or spoken in five languages, and social handles and usernames (as `ID`), are masked. A clinic address on the allow list also matches its common spellings (Ave and Avenue, Suite and #), and a name found once is masked on its later mentions in the same message. Phone numbers given in pieces across lines or turns ("area code 404", then "555", then "0182"), a patient's initials after a patient cue ("pt KJ"), payment handles and bank numbers (Venmo, Cash App, transit and routing numbers), and rural routes, GPS coordinates, counties and reserves are masked too. So are travel booking codes ("booking ref K7QX2M") and e-ticket numbers, passport numbers and foreign national IDs (Aadhaar, CURP and others), ages over 89 in family histories ("grandmother, 91", "d. 2011 at 93"), names spelled with Spanish or French letter names ("erre o ese a"), and foreign addresses and postcodes (UK, India, the Philippines, Mexico). Names in document page headers ("LOPEZ, MARIA T" above "Page 1 of 3"), forum usernames and profile links, licence plates and passports, ITIN and last-four answers, and Nepali, Hindi and Lao names are masked too. A link on the clinic's own website passes only as the bare clinic address or a generic page. Invisible characters (zero-width spaces, bidi marks) are ignored when matching. ### Optional local model @@ -154,31 +154,31 @@ Presidio is also supported: `pip install phiproxy[presidio]` and add `PresidioDe ## Benchmark -`benchmark/` holds thirteen synthetic **patient communications** sets: SMS, web chat, call transcripts and voicemails, emails, referral letters, insurance letters and intake text, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented. +`benchmark/` holds fourteen synthetic **patient communications** sets: SMS, web chat, call transcripts and voicemails, emails, referral letters, insurance letters and intake text, the kind of text clinics actually send to LLMs. Most existing de-identification work targets clinical notes instead. All people and numbers are invented. - **v0** (288 messages, English): template-based, written by the detector authors. -- **blind v1 to v12** (420, 400, 360, 400, 400, 400, 400, 400, 400, 400, 400 and 400 messages): English, French, Spanish, Polish, Hebrew and mixed, each written by an AI model (Opus, Sonnet, Fable, then Opus, Fable, Opus, Opus, and four Sonnet authors for v8 to v12, each set with a new brief) that never saw the detector code, each with about 30% of messages containing no identifiers but full of lookalikes (drug doses, embryo grades, billing codes). v3 is US-heavy and mentions people without cue words ("Can Omar come in with me?"). v4 comes from a British Columbia clinic with international patients, and has spoken and corrected numbers, scanned faxes, internal staff notes, the clinic's own staff roster, and patients who share a staff member's name. v5 comes from a New Jersey clinic, with conversations with the clinic's AI assistant, names from many backgrounds and US insurance paperwork. v6 comes from an Arizona clinic with a Winnipeg office: a quarter calls, first names with no cue word, staff handoffs and internal chat, cross-border surrogacy; its author generated messages from templates, so it is less varied than the others. v7, written by hand, comes from a Houston clinic with a Montreal office: email threads, intake forms, male factor and LGBTQ+ families, specialty-pharmacy calls. v8, written by hand by four authors, comes from a Miami clinic with a Toronto office: phone typing and autocorrect, pasted screenshots, group texts on shared phones, relatives over 89, military and student coverage. v9, also written by four authors, comes from a Denver clinic with a Calgary office: record-system dumps, CSV, JSON and HL7 text, long legal and clinical documents, names that look like words or places, and unusual identifier formats. v10, also written by four authors, comes from a Minneapolis clinic with a Winnipeg office: calls through phone interpreters, numbers and emails spoken as words, and identifiers people half-hide or obfuscate ("ana [at] example [dot] org", "SSN ends in 4417", social handles). v11, also written by four authors, comes from an Atlanta clinic with a Halifax office: long threads where a phone number or address comes in pieces and is corrected, staff shorthand with patient initials ("pt KJ 34yo"), rural and hard addresses, and payment and bank details. v12, also written by four authors, comes from a Seattle clinic with a Regina office: family histories with pedigrees and ages over 89, legal donor and surrogacy paperwork, travel documents (passports, flight booking codes, foreign national IDs and addresses), and call transcripts with speech-recognition errors. A second annotator, also blind to the code, corrected the labels. +- **blind v1 to v13** (420, 400, 360, 400, 400, 400, 400, 400, 400, 400, 400, 400 and 400 messages): English, French, Spanish, Polish, Hebrew and mixed, each written by an AI model (Opus, Sonnet, Fable, then Opus, Fable, Opus, Opus, and four Sonnet authors for v8 to v13, each set with a new brief) that never saw the detector code, each with about 30% of messages containing no identifiers but full of lookalikes (drug doses, embryo grades, billing codes). v3 is US-heavy and mentions people without cue words ("Can Omar come in with me?"). v4 comes from a British Columbia clinic with international patients, and has spoken and corrected numbers, scanned faxes, internal staff notes, the clinic's own staff roster, and patients who share a staff member's name. v5 comes from a New Jersey clinic, with conversations with the clinic's AI assistant, names from many backgrounds and US insurance paperwork. v6 comes from an Arizona clinic with a Winnipeg office: a quarter calls, first names with no cue word, staff handoffs and internal chat, cross-border surrogacy; its author generated messages from templates, so it is less varied than the others. v7, written by hand, comes from a Houston clinic with a Montreal office: email threads, intake forms, male factor and LGBTQ+ families, specialty-pharmacy calls. v8, written by hand by four authors, comes from a Miami clinic with a Toronto office: phone typing and autocorrect, pasted screenshots, group texts on shared phones, relatives over 89, military and student coverage. v9, also written by four authors, comes from a Denver clinic with a Calgary office: record-system dumps, CSV, JSON and HL7 text, long legal and clinical documents, names that look like words or places, and unusual identifier formats. v10, also written by four authors, comes from a Minneapolis clinic with a Winnipeg office: calls through phone interpreters, numbers and emails spoken as words, and identifiers people half-hide or obfuscate ("ana [at] example [dot] org", "SSN ends in 4417", social handles). v11, also written by four authors, comes from an Atlanta clinic with a Halifax office: long threads where a phone number or address comes in pieces and is corrected, staff shorthand with patient initials ("pt KJ 34yo"), rural and hard addresses, and payment and bank details. v12, also written by four authors, comes from a Seattle clinic with a Regina office: family histories with pedigrees and ages over 89, legal donor and surrogacy paperwork, travel documents (passports, flight booking codes, foreign national IDs and addresses), and call transcripts with speech-recognition errors. v13, also written by four authors, comes from a Nashville clinic with an Ottawa office: clinical documents with the patient's name repeated in page headers and footers, forum posts and profile links, work, benefits, safety and legal paperwork, and automation output such as templates and keypad logs. A second annotator, also blind to the code, corrected the labels. Each blind set was scored once with the detectors frozen, then used to find gaps. So each set's first score is a test of the fixes made on the sets before it: -| recall, first score on unseen text | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 | v11 | v12 | -|---|---|---|---|---|---|---|---|---|---|---|---|---| -| regex | 36.0% | 57.3% | 58.4% | 66.1% | 73.7% | 80.2% | 75.7% | 77.0% | 73.7% | 70.0% | 69.9% | 75.6% | -| regex + known entities | 54.1% | 80.2% | 68.3% | 73.8% | 80.9% | 88.8% | 84.9% | 87.2% | 84.4% | 81.3% | 75.7% | 85.2% | -| regex + local model | 77.3% | 87.9% | 84.2% | 86.5% | 91.2% | 96.8% | 93.1% | 93.4% | 91.6% | 86.6% | 87.1% | 92.1% | -| regex + known + local model | 79.2% | 92.3% | 88.1% | 88.4% | 93.1% | 97.5% | 94.7% | 95.4% | 93.4% | 89.7% | 88.4% | **93.8%** | +| recall, first score on unseen text | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 | v11 | v12 | v13 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| regex | 36.0% | 57.3% | 58.4% | 66.1% | 73.7% | 80.2% | 75.7% | 77.0% | 73.7% | 70.0% | 69.9% | 75.6% | 82.7% | +| regex + known entities | 54.1% | 80.2% | 68.3% | 73.8% | 80.9% | 88.8% | 84.9% | 87.2% | 84.4% | 81.3% | 75.7% | 85.2% | 87.7% | +| regex + local model | 77.3% | 87.9% | 84.2% | 86.5% | 91.2% | 96.8% | 93.1% | 93.4% | 91.6% | 86.6% | 87.1% | 92.1% | 95.4% | +| regex + known + local model | 79.2% | 92.3% | 88.1% | 88.4% | 93.1% | 97.5% | 94.7% | 95.4% | 93.4% | 89.7% | 88.4% | 93.8% | **95.8%** | -| first score, regex + known + local model | v10 | v11 | v12 | +| first score, regex + known + local model | v11 | v12 | v13 | |---|---|---|---| -| by channel | SMS 93%, chat 91%, email 89%, notes 93%, calls 86% | SMS 90%, chat 82%, email 94%, notes 90%, calls 87% | SMS 97%, chat 94%, email 93%, notes 96%, calls 91% | -| by language | en 90%, fr 95%, es 93%, pl 87%, he 81%, mixed 85% | en 89%, fr 88%, es 88%, pl 87%, he 88%, mixed 87% | en 94%, fr 95%, es 94%, pl 95%, he 90%, mixed 92% | -| precision (default / clinic details allowed / plus staff list) | 72.4% / 80.8% / 83.8% | 72.0% / 80.9% / 84.0% | 77.2% / 85.8% / 88.1% | -| messages without identifiers where something was masked | 68% / 47% / 45% | 61% / 39% / 39% | 57% / 34% / 34% | +| by channel | SMS 90%, chat 82%, email 94%, notes 90%, calls 87% | SMS 97%, chat 94%, email 93%, notes 96%, calls 91% | SMS 94%, chat 96%, email 96%, notes 95%, calls 97% | +| by language | en 89%, fr 88%, es 88%, pl 87%, he 88%, mixed 87% | en 94%, fr 95%, es 94%, pl 95%, he 90%, mixed 92% | en 97%, fr 96%, es 98%, pl 95%, he 87%, mixed 90% | +| precision (default / clinic details allowed / plus staff list) | 72.0% / 80.9% / 84.0% | 77.2% / 85.8% / 88.1% | 75.4% / 85.6% / 87.8% | +| messages without identifiers where something was masked | 61% / 39% / 39% | 57% / 34% / 34% | 64% / 43% / 42% | **How to read this honestly:** - Expect real traffic to score lower: these sets are AI-written, not written by clinic staff. v0 overstated recall by about 30 points. - Without the local model, names not in your records are the main gap. On v3, where names often come with no cue, the pattern rules alone found 43% of names. -- After the gaps found on each set were fixed, the current code scores 96% to 99% with the local model on the twelve blind sets. Those numbers are optimistic, which is why the table above shows first scores. v4 showed how much: the code that scored 95% to 97% on v1 to v3 scored 88% on v4. After the Hebrew, call and date work, the next unseen set (v5) scored 93%, v6 then scored 97.5%, but it was built from templates and is probably easier; v7, written by hand, scored 94.7%, v8 95.4%, v9 93.4%, v10 89.7%, v11 88.4%, and v12 93.8%. v9's brief aimed at text the rules had never seen (JSON, CSV and HL7 dumps, court and case numbers, Alberta health numbers, nicknames), and v10's went after spoken, obfuscated and half-hidden forms (emails written "ana [at] example [dot] org" or read out loud, numbers read one digit at a time, "SSN ends in 4417", social handles). v11 scored lowest of the recent sets because its brief went after numbers given in pieces over long threads ("the area code is 404", then "555", then "0182"), staff shorthand with patient initials, rural and hard addresses (rural routes, GPS coordinates, counties, reserves), and payment and bank details. v12 scored higher because the v10 and v11 fixes carried over to new text, while its brief went after family-history pedigrees (ages over 89, relatives' names and dates), legal donor paperwork, travel documents (passports, booking codes, foreign national IDs) and speech-recognition errors in call transcripts. Each set's brief differs, so expect roughly 88% to 95% on new text of this kind, and less on real traffic. +- After the gaps found on each set were fixed, the current code scores 96.6% to 100% with the local model on the thirteen blind sets (100% on v13, after fixes made with it). Those numbers are optimistic, which is why the table above shows first scores. v4 showed how much: the code that scored 95% to 97% on v1 to v3 scored 88% on v4. After the Hebrew, call and date work, the next unseen set (v5) scored 93%, v6 then scored 97.5%, but it was built from templates and is probably easier; v7, written by hand, scored 94.7%, v8 95.4%, v9 93.4%, v10 89.7%, v11 88.4%, v12 93.8%, and v13 95.8%. v9's brief aimed at text the rules had never seen (JSON, CSV and HL7 dumps, court and case numbers, Alberta health numbers, nicknames), and v10's went after spoken, obfuscated and half-hidden forms (emails written "ana [at] example [dot] org" or read out loud, numbers read one digit at a time, "SSN ends in 4417", social handles). v11 scored lowest of the recent sets because its brief went after numbers given in pieces over long threads ("the area code is 404", then "555", then "0182"), staff shorthand with patient initials, rural and hard addresses (rural routes, GPS coordinates, counties, reserves), and payment and bank details. v12 scored higher because the v10 and v11 fixes carried over to new text, while its brief went after family-history pedigrees (ages over 89, relatives' names and dates), legal donor paperwork, travel documents (passports, booking codes, foreign national IDs) and speech-recognition errors in call transcripts. v13 scored higher again because the earlier fixes carried over, while its brief went after names in document page headers, forum posts and profile links, work and benefits paperwork, safety and legal paperwork, and automation output (templates, keypad logs, message IDs). Each set's brief differs, so expect roughly 88% to 95% on new text of this kind, and less on real traffic. - Over-masking is real: v6, full of staff talking about staff, had 55% of its clean messages masked with the clinic and staff lists at first; the staff-list cues added after it bring that to 34%. Most of it is the clinic's own contact details (pass them in `Policy(allow=...)`), staff names (pass them in `Policy(staff=...)`), and holiday dates, which stay masked on purpose. - Details, label rules, the corrections and full tables are in [`benchmark/README.md`](https://github.com/DecisionTreeTechnology/phiproxy/blob/main/benchmark/README.md). diff --git a/benchmark/README.md b/benchmark/README.md index 053faa8..6c137fb 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -19,38 +19,42 @@ Labelled sets of the kind of text clinics send to LLMs: patient SMS, web chat, p | `patient_comms_blind_v10.jsonl` | 400 (121 with no identifiers) | 1,232 + 224 staff and clinician names | en, fr, es, pl, he, mixed | four blind AI authors (Claude Sonnet), each writing 100 messages by hand from one brief: a Minneapolis clinic with a Winnipeg office, calls through phone interpreters, numbers and emails spoken as words, spelled names ("F as in Foxtrot"), identifiers people half-hide or obfuscate ("SSN ends in 4417", "DOB 03/xx/1988", "ana [at] example [dot] org", social handles, usernames, numbers split across lines, a letter O typed for zero), quoted, forwarded and replied email, attachment names with a patient's name in them, hard date forms (ranges, ordinals in five languages, Hebrew calendar dates, day-month against month-day), names in odd places and forms (possessives, "the Vang-Lees", Hmong clan names, Somali three-part names, Ojibwe and Métis names, nicknames), and who is a provider (outside OBs, pharmacists, patients who are themselves doctors) | | `patient_comms_blind_v11.jsonl` | 400 (137 with no identifiers) | 1,408 + 277 staff and clinician names | en, fr, es, pl, he, mixed | four blind AI authors (Claude Sonnet), each writing 100 messages by hand from one brief: an Atlanta clinic with a Halifax office, long threads with the clinic's AI assistant or staff (10 to 40 turns) where identifiers come far from their cue, are given in pieces over several turns ("the area code is 404", "then 555", "0182"), corrected ("sorry, 0128") and read back, staff shorthand (Teams and Slack handoffs, huddle and triage notes such as "pt KJ 34yo G2P1", "Mrs. O rm 3", "re: AB's FET") where a patient's initials alone are a name and staff initials are a provider, rural and hard addresses (rural routes, counties, First Nation communities, GPS coordinates, directions to a home, military and campus addresses, Mexican colonias), payment handles and bank details (Venmo, Cash App, Interac, transit and routing numbers, cheques), photo and attachment names and devices, and foster care, adoption and caseworkers | | `patient_comms_blind_v12.jsonl` | 400 (116 with no identifiers) | 1,386 + 135 staff and clinician names | en (200), es (60), fr (40), pl (30), he (30), mixed (40) | four blind AI authors (Claude Sonnet), each writing 100 messages by hand from one brief: a Seattle clinic with a Regina office, family history and genetics (genetic counselling letters, pedigrees written out with relatives' names, birth and death dates and ages over 89), third-party reproduction paperwork (carrier and donor agreements in legal style, bar and notary numbers, court parentage orders, donor numbers, cryo-storage and shipping labels), travel and cross-border care (passports, flight booking codes and e-tickets, foreign national IDs such as Aadhaar, PhilSys, NIN and CURP, foreign addresses and postcodes), speech-recognition errors in call transcripts (names heard as words, numbers half in words), form, app and calendar exports and OCR noise, and teenagers preserving fertility before cancer treatment with a parent or guardian; the mixed rows add Tagalog, Punjabi, Somali, Russian, Ukrainian, Japanese, Vietnamese and Cree | +| `patient_comms_blind_v13.jsonl` | 400 (118 with no identifiers) | 1,193 + 170 staff and clinician names | en (200), es (60), fr (40), pl (30), he (30), mixed (40) | four blind AI authors (Claude Sonnet), each writing 100 messages by hand from one brief: a Nashville clinic with an Ottawa office, clinical documents with repeated page headers and footers ("LAST, FIRST M" above "Page 1 of 3"), public posts, reviews, forum threads and direct messages (usernames, profile links), work and benefits paperwork (employee, badge, claim and policy numbers, employers), safety and legal paperwork (address confidentiality programs, protective orders), automation output (templates with placeholders, keypad logs, message IDs) and one-word replies; families from Nashville's Kurdish, Somali, Egyptian Coptic, Mexican, Guatemalan, Laotian and Bhutanese-Nepali communities and Ottawa's Franco-Ontarian, Lebanese, Somali, Inuit, Haitian, Chinese and Filipino communities; the mixed rows use Nepali and Hindi (Devanagari), Lao, Arabic, Somali and other languages next to English or French | -**How the blind sets were made.** Each was written by a separate AI agent that had only a brief: the label rules, channels, languages, realistic mess, and a list of things that look like identifiers but are not. It was told not to read the detector code, the tests or the other sets, and it wrote every message from scratch. The v4 to v12 authors also wrote the clinic's own details (`blind/clinic_v4.json` to `blind/clinic_v12.json`) and a 25- or 26-person staff roster (`blind/staff_v4.json` to `blind/staff_v12.json`) before writing any message, as a clinic would configure them. The v6 author wrote small generator scripts with message templates and filled them with different names, numbers and dates, so v6 is less varied than the other sets: 378 of its 400 messages start differently, and 48 messages (12%) are close pairs. The v7 brief then forbade generator scripts; all 400 v7 messages start differently and none are close pairs, though some French and Spanish emails retell English situations. v8 was written by four authors in parallel (English SMS and chat; English calls, email and notes; Spanish and French; Polish, Hebrew and mixed), after a fifth agent wrote the clinic and staff files; none are close pairs. v9 was made the same way from a new brief; none are close pairs. v10 was made the same way from a new brief, after a fifth agent wrote the clinic and staff files; none are close pairs. v11 was made the same way from a new brief, and the fifth agent also wrote a file of staff initials (`blind/staff_initials_v11.json`), so the staff list can tell a staff member's initials from a patient's; none are close pairs. v12 was made the same way from a new brief (a Seattle clinic with a Regina office; family history, legal donor paperwork, travel documents and speech-recognition errors), with four authors writing 100 messages each after a fifth agent wrote the clinic, staff and staff-initials files (`blind/staff_initials_v12.json`); none are close pairs. The sets are independent of the detector code, but they are AI-written, not written by clinic staff. +**How the blind sets were made.** Each was written by a separate AI agent that had only a brief: the label rules, channels, languages, realistic mess, and a list of things that look like identifiers but are not. It was told not to read the detector code, the tests or the other sets, and it wrote every message from scratch. The v4 to v13 authors also wrote the clinic's own details (`blind/clinic_v4.json` to `blind/clinic_v13.json`) and a 25- or 26-person staff roster (`blind/staff_v4.json` to `blind/staff_v13.json`) before writing any message, as a clinic would configure them. The v6 author wrote small generator scripts with message templates and filled them with different names, numbers and dates, so v6 is less varied than the other sets: 378 of its 400 messages start differently, and 48 messages (12%) are close pairs. The v7 brief then forbade generator scripts; all 400 v7 messages start differently and none are close pairs, though some French and Spanish emails retell English situations. v8 was written by four authors in parallel (English SMS and chat; English calls, email and notes; Spanish and French; Polish, Hebrew and mixed), after a fifth agent wrote the clinic and staff files; none are close pairs. v9 was made the same way from a new brief; none are close pairs. v10 was made the same way from a new brief, after a fifth agent wrote the clinic and staff files; none are close pairs. v11 was made the same way from a new brief, and the fifth agent also wrote a file of staff initials (`blind/staff_initials_v11.json`), so the staff list can tell a staff member's initials from a patient's; none are close pairs. v12 was made the same way from a new brief (a Seattle clinic with a Regina office; family history, legal donor paperwork, travel documents and speech-recognition errors), with four authors writing 100 messages each after a fifth agent wrote the clinic, staff and staff-initials files (`blind/staff_initials_v12.json`); none are close pairs. v13 was made the same way from a new brief (a Nashville clinic with an Ottawa office; page headers and footers in clinical documents, public posts and forum threads, work, benefits, safety and legal paperwork, and automation output), with four authors writing 100 messages each after a fifth agent wrote the clinic, staff and staff-initials files (`blind/staff_initials_v13.json`); none are close pairs. Template placeholders such as "{first_name}" are not labelled. The sets are independent of the detector code, but they are AI-written, not written by clinic staff. -**Second annotator.** A third agent, also without access to code or detector output, reviewed each blind set against the written label rules and listed missing and wrong labels (`blind/audit_v1.jsonl` to `blind/audit_v12.jsonl`; 45, 149, 2, 50, 6, 11, 62, 16, 47, 63, 68 and 70 corrections; v8 to v12 had two independent annotators per 100 messages and an adjudicator that kept only clearly correct findings, mostly unlabelled repeat mentions of names, staff names, spoken dates and numbers read back by the agent). `blind/convert.py` applies them. All scores below use the corrected labels. +**Second annotator.** A third agent, also without access to code or detector output, reviewed each blind set against the written label rules and listed missing and wrong labels (`blind/audit_v1.jsonl` to `blind/audit_v13.jsonl`; 45, 149, 2, 50, 6, 11, 62, 16, 47, 63, 68, 70 and 58 corrections; for v13, 33 add, 7 remove and 18 retype; v8 to v13 had two independent annotators per 100 messages and an adjudicator that kept only clearly correct findings, mostly unlabelled repeat mentions of names, staff names, spoken dates and numbers read back by the agent). `blind/convert.py` applies them. All scores below use the corrected labels. -**Frozen scoring.** Before each blind set was scored, the detectors were committed and not changed: v1 against commit 082c2e1, v2 against ab99dc9, v3 against 1dd13f2, v4 against 1f548b0, v5 against b1e794b, v6 against dc33528 (0.1.0a8), v7 against f0c957d (0.1.0a9), v8 against 2a9104e (0.1.0a10), v9 against 3aaa537 (0.1.0a11), v10 against 8db92f4 (0.1.0a12), v11 against 7fb0f01 (main after v10, not yet released), v12 against 650fe26 (0.1.0a13). Those first scores are the fair ones (`results_blind_*_frozen.json`). After a set was scored it was used to find and fix gaps, so its later scores are optimistic. Each set is the re-test of the fixes made on the sets before it. v4 was also scored against 5762fb9, a spoken-date change committed before v4 was seen: it caught one more identifier (88.5% instead of 88.4%) and masked two more non-identifiers, so the spoken-date forms fixed from v1 to v3 were not the ones v4 used. +**Frozen scoring.** Before each blind set was scored, the detectors were committed and not changed: v1 against commit 082c2e1, v2 against ab99dc9, v3 against 1dd13f2, v4 against 1f548b0, v5 against b1e794b, v6 against dc33528 (0.1.0a8), v7 against f0c957d (0.1.0a9), v8 against 2a9104e (0.1.0a10), v9 against 3aaa537 (0.1.0a11), v10 against 8db92f4 (0.1.0a12), v11 against 7fb0f01 (main after v10, not yet released), v12 against 650fe26 (0.1.0a13), v13 against 59f1f80 (0.1.0a14). Those first scores are the fair ones (`results_blind_*_frozen.json`). After a set was scored it was used to find and fix gaps, so its later scores are optimistic. Each set is the re-test of the fixes made on the sets before it. v4 was also scored against 5762fb9, a spoken-date change committed before v4 was seen: it caught one more identifier (88.5% instead of 88.4%) and masked two more non-identifiers, so the spoken-date forms fixed from v1 to v3 were not the ones v4 used. ## Results (recall) First score of the frozen code on each set it had never seen: -| configuration | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 | v11 | v12 | -|---|---|---|---|---|---|---|---|---|---|---|---|---| -| regex | 36.0% | 57.3% | 58.4% | 66.1% | 73.7% | 80.2% | 75.7% | 77.0% | 73.7% | 70.0% | 69.9% | 75.6% | -| regex + known entities | 54.1% | 80.2% | 68.3% | 73.8% | 80.9% | 88.8% | 84.9% | 87.2% | 84.4% | 81.3% | 75.7% | 85.2% | -| regex + local model | 77.3% | 87.9% | 84.2% | 86.5% | 91.2% | 96.8% | 93.1% | 93.4% | 91.6% | 86.6% | 87.1% | 92.1% | -| regex + known + local model | 79.2% | 92.3% | 88.1% | 88.4% | 93.1% | 97.5% | 94.7% | 95.4% | 93.4% | 89.7% | 88.4% | **93.8%** | +| configuration | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 | v11 | v12 | v13 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| regex | 36.0% | 57.3% | 58.4% | 66.1% | 73.7% | 80.2% | 75.7% | 77.0% | 73.7% | 70.0% | 69.9% | 75.6% | 82.7% | +| regex + known entities | 54.1% | 80.2% | 68.3% | 73.8% | 80.9% | 88.8% | 84.9% | 87.2% | 84.4% | 81.3% | 75.7% | 85.2% | 87.7% | +| regex + local model | 77.3% | 87.9% | 84.2% | 86.5% | 91.2% | 96.8% | 93.1% | 93.4% | 91.6% | 86.6% | 87.1% | 92.1% | 95.4% | +| regex + known + local model | 79.2% | 92.3% | 88.1% | 88.4% | 93.1% | 97.5% | 94.7% | 95.4% | 93.4% | 89.7% | 88.4% | 93.8% | **95.8%** | The current code ("now", optimistic: every set below was used to find gaps): -| configuration | v0 | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 | v11 | v12 | -|---|---|---|---|---|---|---|---|---|---|---|---|---|---| -| regex | 88.0% | 86.7% | 86.9% | 74.4% | 80.6% | 81.6% | 84.5% | 84.1% | 86.4% | 85.5% | 85.8% | 86.4% | 83.1% | -| regex + known entities | 98.0% | 90.9% | 95.9% | 82.7% | 88.2% | 86.8% | 92.6% | 90.5% | 93.7% | 94.3% | 94.5% | 91.3% | 92.2% | -| regex + local model | 98.6% | 96.1% | 97.0% | 94.7% | 95.7% | 95.7% | 98.6% | 96.8% | 98.1% | 98.4% | 97.3% | 98.6% | 97.5% | -| regex + known + local model | 99.1% | 97.1% | 98.7% | 97.3% | 96.7% | 96.4% | 99.2% | 97.1% | 98.9% | 99.3% | 99.2% | 99.3% | 98.9% | +| configuration | v0 | v1 | v2 | v3 | v4 | v5 | v6 | v7 | v8 | v9 | v10 | v11 | v12 | v13 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| regex | 88.0% | 86.7% | 87.0% | 74.5% | 81.0% | 82.0% | 84.8% | 84.4% | 86.6% | 85.6% | 85.8% | 86.4% | 83.2% | 88.3% | +| regex + known entities | 98.0% | 91.2% | 96.0% | 82.9% | 88.3% | 87.2% | 93.2% | 90.7% | 93.7% | 94.5% | 94.5% | 91.3% | 92.2% | 92.7% | +| regex + local model | 98.6% | 96.1% | 97.2% | 94.7% | 96.1% | 95.9% | 98.6% | 97.0% | 98.2% | 98.4% | 97.3% | 98.6% | 97.6% | 99.6% | +| regex + known + local model | 99.1% | 97.1% | 98.8% | 97.3% | 96.9% | 96.6% | 99.2% | 97.4% | 98.9% | 99.4% | 99.2% | 99.3% | 98.9% | 100.0% | - "First" is the score of the frozen code on a set it had never seen. "Now" is the current code, after fixes informed by that set, so it is optimistic. - v3 tested the name rules added after v2 (finding names from self-introductions, relatives, chart labels, sign-offs and signature blocks, without a model). On v3, before those rules, the pattern detector found 8% of names and 43.1% of all identifiers; with them, 43% of names and 58.4% overall, at the same precision. With the local model the gain is smaller (82.1% to 84.2%), because the model already finds most names. - v0 overstates recall, because it was written by the people who wrote the detectors. - On v2 the "known entities" rows are optimistic: for some messages the v2 author filled the patient record from the message itself. - Spoken identifiers: after the v3 first score, numbers read out loud are rebuilt before matching (grouped and compound number words in five languages, spelled letters, the phonetic alphabet). "Now" recall on call transcripts with the local model rose from 91% to 94% (v1), 88% to 94% (v2) and 94% to 96% (v3); without the model, from 82% to 86%, 79% to 86% and 77% to 80%. False positives did not rise on any set. +- v13 is the fair test of 0.1.0a14 (everything fixed through v12): 95.8% with the local model, 87.7% without it, higher than v12 (93.8%), v11 (88.4%) and v10 (89.7%) because the earlier fixes carried over to new text. Its brief went after clinical documents with repeated page headers and footers ("LAST, FIRST M" above "Page 1 of 3"), public posts, reviews and forum threads (usernames, profile links), work and benefits paperwork (employee, badge, claim and policy numbers, employers), safety and legal paperwork (address confidentiality programs, protective orders), and automation output (templates with placeholders, keypad logs, message IDs). By channel: calls 97%, chat 96%, email 96%, notes 95%, SMS 94%. By language: es 98%, en 97%, fr 96%, pl 95%, mixed 90%, he 87%. Weakest types: SSN (67%: an ITIN, a last-four answer), ages over 89 (79%: in brackets and after "lived to"), employers (86%: company and school names, mostly in Hebrew), IDs (87%: forum usernames, passports, licence plates, a Brazilian CPF, employee and claim numbers), URLs (89%: social profile links) and health cards (89%). With the clinic details allowed, recall was 95.6%, precision 85.6% and 43% of clean messages were masked; with the staff list too, 95.6%, 87.8% and 42%. Without the local model (regex + known entities), precision was 83.4% and 47% of clean messages were masked. +- Some v13 label calls were left as the annotators made them: some forum usernames are labelled `NAME` rather than `ID`. Either way the code must mask them, so recall is not affected. +- After the v13 first score: Names: "LAST, FIRST M" names in document page headers and footers (the whole name, with particles, suffixes and up to three given names; the page marker or record field may be on the next line; lab test-name headers and form titles are left alone); names spelled with English letter names ("gee you zed ...") after a spelling cue; first-name lists after a sign-off ("Love, Cody, Josie and Mia"); Nepali and Hindi names in Devanagari after name cues, relative cues and sign-offs; Lao names; a Hebrew surname written after "in Hebrew"; "Posted by: Jane D."; the second name in a couple ("Hannah and Rob") when the first is a found name. Dates and ages: dates with Arabic, Persian and Devanagari digits and Arabic, Persian, Hindi and Nepali month names; Somali dates; a Spanish weekday with a number word ("martes veintisiete"); a date of birth typed with spaces right after a DOB cue in five languages ("DOB 09 30 1988"); a month alone when tied to a person ("in May", "planned for January", "pour juillet") but not in clinic notices ("closed in May"); ages over 89 after "lived to", in brackets ("(lat 91)") and in Hebrew. IDs and links: forum usernames after "posted by" or before "wrote"; licence plates; passports (also a second one in the same sentence, in lowercase); Brazilian CPF; ITIN and tax IDs with any separator, and every number in a list after one cue; last-four answers in four languages (also when they look like a year); employee, badge, participant, claim and dossier numbers; policy, group and member numbers (also Polish and Hebrew); spoken health card digits; ZIP codes after a PO box or apartment; social and link-in-bio profile links without "https"; a personal website after a first-person cue. Employers and addresses: "drives", "delivers" or "volunteers for" a company; Hebrew workplace and school cues; a Hebrew street and house number after a residence cue. Two things that let data through were narrowed during review, because privacy comes first. (1) The clinic's own website: only the bare clinic address (optionally with "www" or a generic part such as "portal") and a short generic page pass; a patient's subdomain, a profile path or a record number in the path is masked, and an allow entry with a path (for example a clinic's social page) no longer opens the whole site. (2) The staff list: a name in a "Cc:" line that is not on the staff list no longer makes the same first name pass elsewhere (a patient's husband named like a staff member stayed visible), and a repeated name passes as staff only where a staff name would pass on its own. Speed: two parts of the code that slowed down sharply on very long text with a repeated name (both present in 0.1.0a14) were rewritten; a 100 KB message of one repeated name went from about 20 s to under 2 s. One round of adversarial review (four reviewers) confirmed 25 problems, a final round (newly masked text, privacy, robustness) 14 more, and a privacy and robustness round 8 more; an independent checker re-ran every fix against the frozen code. All were fixed or are listed in the known limits, and the fixes became tests (2,193 tests). With the clinic details allowed (`results_blind_v13.json`): regex 88.3% recall and 92.1% precision, regex + known entities 92.7% and 92.4%, regex + local model 99.6% and 86.0%, all detectors 100.0% and 86.0%. On v13 with the clinic and staff lists, recall went from 95.6% to 100.0%, precision from 87.8% to 88.1%, and clean messages masked stayed at 42%. The 100% is after fixes made with this set, so it is optimistic: the fair measure is the first score above (95.8%), and new text will score less. Regression check against the frozen 59f1f80 code, on v0 to v13 in four configurations (regex + known and all detectors, each with and without the staff list): no identifier it caught is missed now. With regex + known and the staff list, v1 gained 2 identifiers, v2 1, v3 2, v4 3, v5 5, v6 7, v7 2 and v9 2; with all detectors and the staff list, v2 gained 1, v4 3, v5 3, v7 2 and v9 1. New false positives on older sets: four of the clinics' own links with a page path (".../pay", ".../openhouse", ".../seminario", ".../pagos"), now masked because the clinic-website rule was narrowed; one agent reading back part of a health card number; and, with the staff list, seven staff first names written alone ("Paris" after "Paris Okafor") that are now masked because they repeat a masked name or have no staff cue. v13 with all detectors and the staff list went from 155 to 156 false positives. Mean time is 4.0 ms per message on the 5,180 blind messages (3.3 ms before), worst case 24 ms (22 to 33 ms before); a 47 KB message takes 1.4 s (1.2 s before). The speed was measured on a loaded machine, so compare the ratio, not the absolute numbers. These are "now" numbers. On the thirteen blind sets the current code scores 96.6% to 100% with all detectors. - v12 is the fair test of 0.1.0a13 (everything fixed through v11): 93.8% with the local model, 85.2% without it, higher than v11 (88.4%) and v10 (89.7%) because the v10 and v11 fixes carried over to new text. By channel: SMS 97%, notes 96%, chat 94%, email 93%, calls 91%. By language: fr 95%, pl 95%, en 94%, es 94%, mixed 92%, he 90%. Weakest types: ages over 89 (58%: pedigrees such as "d. June 2011 at 93" and "maternal grandmother, 91"), IDs (65%: flight booking codes, e-ticket numbers, passports, Aadhaar and PhilSys numbers, lawyers' bar numbers, court cause numbers), postal codes (84%: Indian PIN, Philippine, Mexican and UK postcodes), employers (88%: schools in Spanish, French, Polish and Hebrew) and health cards (88%: member IDs in letter-digit groups). With the clinic details allowed, recall was 93.7%, precision 85.8% and 34% of clean messages were masked; with the staff list too, 93.7%, 88.1% and 34%. Without the local model (regex + known entities), precision was 86.4% and 36% of clean messages were masked. - Some v12 label calls were left as the annotators made them: a cryo cane position ("C-14") and a notary office number ("Notaría 114") are labelled `ID`, while the code leaves them visible because they do not identify a person; two hospitals where a child is treated ("Hospital Infantil ...", "Hôpital pour enfants de ...") are labelled `EMPLOYER` although they are not a workplace or school, and the English equivalent is not labelled; speech-recognition mishearings of names ("rose marry", "cove all check", "tao chan") are labelled `NAME`, and the code does not catch them. - After the v12 first score: IDs (only after a cue): travel booking and record-locator codes after a booking cue in five languages and Chinese, Japanese and Korean text, e-ticket numbers, passport numbers after a passport cue, foreign national IDs after their names (Aadhaar, PhilSys, Nigerian NIN, Belgian national number), lawyers' bar numbers and notary commission numbers, court cause numbers, insurance member IDs in letter-digit groups, a scanned PESEL with OCR letters, and NHS and Japanese "My Number" values. Ages and dates: ages over 89 after death and life words ("d. 2011 at 93", "passed away in 2015 at the age of 94", "zmarła mając 92 lata", "alive, 93"), after a relative with an optional name ("maternal grandmother, 91", "grandmother (Anna, 91)"), and Russian and Ukrainian ages and day-and-month dates ("14 грудня"). Names: names spelled out with Spanish and French letter names after a spelling cue, Cyrillic names after cues and vocatives, Japanese names in kanji after a name cue, first names in transcripts with time stamps, and pedigree markers. Addresses: Philippine lot-and-block and barangay addresses (with an address context), UK street types and postcodes, Indian addresses with a 6-digit PIN, Mexican floor and door ("3º B"), reserve and settlement names after a cue, PIN and postcode cues in local formats, and the clinic's own city when a patient says they live there; schools and universities as `EMPLOYER` in Spanish, French, Polish and Hebrew after a work or study cue. Four features were removed during review because each could leave patient data visible: letting URLs on the clinic's own domain pass, skipping phone numbers near the word "NHS", treating "the 4th floor" as not a date, and treating a lab range "3.5-12" as not a date. All four behave as the frozen code did. One round of adversarial review (four reviewers) confirmed 15 problems, a final round (newly masked text, privacy, robustness) confirmed 10 more, and a privacy and robustness round found one more (a rule that ran in quadratic time on long blank runs); a separate verifier re-ran every claim before it counted. All were fixed or are listed in the known limits, and the fixes became tests (1,792 tests). On v12 with the clinic and staff lists, recall went from 93.7% to 98.9%, precision from 88.1% to 89.0%, and clean messages masked stayed at 34%. With the clinic details allowed (`results_blind_v12.json`): regex 83.1% recall and 92.8% precision, regex + known entities 92.2% and 93.3%, regex + local model 97.5% and 86.6%, all detectors 98.9% and 86.7%. By type now (all detectors, clinic details allowed): employers 95%, names 98%, IDs 98%, all others 100%; by channel: email 100%, notes 100%, chat 99%, SMS 99%, calls 96%. The 15 remaining misses (with the staff list) are seven speech-recognition mishearings of names ("rose marry", "cove all check", "tao chan", "rose à Lyne"), three spelled-out names ("en gee", "N G", "D E V"), one Hebrew first name with no cue, the two treating hospitals, and the two label calls above (cane position, notary office). Regression check against the frozen 650fe26 code, on v0 to v12 in four configurations (regex + known and all detectors, each with and without the staff list): no identifier it caught is missed now. With regex + known and the staff list, v3 gained 1 identifier, v4 1, v5 2, v7 2, v9 2 and v11 3; with all detectors and the staff list, v7 gained 1. False positives are the same or lower on every set in all four configurations; v12 with all detectors and the staff list went from 169 to 163, and with regex + known and the staff list from 95 to 87. New false positives on older sets: one Russian vocative on a voicemail sign-off ("Оксана") masked as a name. Mean time is 2.9 ms per message on the 4,780 blind messages (2.7 ms before), worst case 17 ms (16 ms before); a 47 KB message takes 0.7 s (0.6 s before). These are "now" numbers. On the twelve blind sets the current code scores 96.4% to 99.3% with all detectors. @@ -129,6 +133,10 @@ The current code ("now", optimistic: every set below was used to find gaps): | v12 first score, clinic details allowed | 85.8% | 34% | | v12 first score, clinic details and staff list | 88.1% | 34% | | v12 now, clinic details and staff list | 89.0% | 34% | +| v13 first score, default policy | 75.4% | 64% | +| v13 first score, clinic details allowed | 85.6% | 43% | +| v13 first score, clinic details and staff list | 87.8% | 42% | +| v13 now, clinic details and staff list | 88.1% | 42% | v2 and v3 contain many email footers and letterheads with the fictional clinic's own name, phone, fax, address and email. Those are not patient data, so they are not labelled, and phiproxy masks them unless the deployment lists them in `Policy(allow=[...])`. The "clinic details allowed" rows pass `blind/clinic_v2.json` or `blind/clinic_v3.json` with `--allow`. Most of the rest is holiday and clinic-closure dates (masked on purpose: a date is never exempted, because it could be a birthday), staff first names that are also words ("Will", "Grace", "Hope"), staff names without a title, and card last-four digits the label rules did not ask for. @@ -152,9 +160,9 @@ On v6 this took false positives from 282 to 162 and clean messages masked from 5 ## Files - `generate.py`: deterministic generator (fixed seed) for `data/patient_comms_v0.jsonl`. -- `blind/blind_v*.source.jsonl`: the blind sets as written, with inline labels (`[[NAME|jess]]`). `blind/audit_v*.jsonl`: second-annotator corrections. `blind/clinic_v*.json` and `blind/staff_v*.json`: clinic details and staff rosters for `--allow` and `--staff`. `blind/staff_initials_v11.json` and `blind/staff_initials_v12.json`: the initials of the v11 and v12 staff. `blind/convert.py v1|...|v12` builds `data/patient_comms_blind_v*.jsonl`. +- `blind/blind_v*.source.jsonl`: the blind sets as written, with inline labels (`[[NAME|jess]]`). `blind/audit_v*.jsonl`: second-annotator corrections. `blind/clinic_v*.json` and `blind/staff_v*.json`: clinic details and staff rosters for `--allow` and `--staff`. `blind/staff_initials_v11.json` to `blind/staff_initials_v13.json`: the initials of the v11 to v13 staff. `blind/convert.py v1|...|v13` builds `data/patient_comms_blind_v*.jsonl`. - `score.py`: scores every available configuration and prints recall by type, channel and language, precision, and clean-message masking. `--show-misses N`, `--show-false-positives N`, `--allow clinic.json` and `--staff staff.json` help with error analysis. The local-model configurations appear when `phiproxy[gliner]` is installed. -- `results_*.json`: scores for the built-in configurations (v2 to v12 with the clinic details allowed; the frozen first scores with the default policy). +- `results_*.json`: scores for the built-in configurations (v2 to v13 with the clinic details allowed; the frozen first scores with the default policy). ## Record format @@ -180,8 +188,8 @@ On v6 this took false positives from 282 to 162 and clean messages masked from 5 ## Known limits - The blind sets are AI-written. Real messages are messier, and a set written by people who work in clinics is still the goal. -- Call transcripts were the weakest channel (86% on v3's first score, 83% on v4's, then 92% on v5's, 97% on v6's, 86% on v10's, 87% on v11's and 91% on v12's): names and numbers spoken in fragments and corrected mid-number, voice-to-text errors ("pre may" for Priya, "for one six" for 416). Faxed notes were 84% on v3's first score and 87% on v4's. -- Hebrew is still the weakest language (77% on v3's first score, 80% on v4's, 86% on v5's, 87% on v7's, 84% on v8's, 85% on v9's, 81% on v10's, 88% on v11's, 90% on v12's; 99% on v6's template-built set): staff and clinic names transliterated into Hebrew ("קיילה", "בלובוניט") are not on the English lists. +- Call transcripts were the weakest channel (86% on v3's first score, 83% on v4's, then 92% on v5's, 97% on v6's, 86% on v10's, 87% on v11's, 91% on v12's and 97% on v13's): names and numbers spoken in fragments and corrected mid-number, voice-to-text errors ("pre may" for Priya, "for one six" for 416). Faxed notes were 84% on v3's first score and 87% on v4's. +- Hebrew is still the weakest language (77% on v3's first score, 80% on v4's, 86% on v5's, 87% on v7's, 84% on v8's, 85% on v9's, 81% on v10's, 88% on v11's, 90% on v12's, 87% on v13's; 99% on v6's template-built set): staff and clinic names transliterated into Hebrew ("קיילה", "בלובוניט") are not on the English lists. - The local model tags insurers, pharmacies and other clinics as a person's workplace ("Gulf Coast Mutual", "Prairie Rx Specialty"). The sets disagree on whether "Dr. X from " is a workplace, so these are not filtered. - Staff named with no cue at all ("Thank you Priya!!") stay masked, so internal staff traffic over-masks even with the staff list. - Bare days and months tied to a patient ("the twelfth", "until January") are often missed: a bare month is too ambiguous to mask everywhere. @@ -201,11 +209,17 @@ On v6 this took false positives from 282 to 162 and clean messages masked from 5 - A spoken identifier answered more than 3,000 characters after its question on a single line is not tied to the question. - Unaccented Vietnamese spoken digits are masked only after an unaccented phone cue ("so dien thoai"). - Speech-recognition mishearings of names ("rose marry" for Rosemary) are not caught. -- Links on the clinic's own website are masked, and so are clinic holiday closure dates. +- Only the bare clinic address (optionally with "www" or a part such as "portal") and short generic pages ("/faq", "/contact") pass. Other links on the clinic's own website, including a page path that is not a generic section word, are masked, and so are clinic holiday closure dates. - A lab range written "3.5-12" is masked as a date, and "the 4th floor" is masked as a date. - An NHS number or a Japanese "My Number" may be labelled `PHONE` (it is masked either way). - "Father, 100" is read as an age, and the clinic's own city is masked when a patient says they live there. - UK mobile and Mexican landline numbers without a phone cue are not caught. +- The same identifier written two ways ("912/70/5531" and "912.70.5531") in one message gets one placeholder and comes back in the first-seen format (phone numbers already behaved this way). +- A drug or brand name beside a patient's first name may be masked in rare phrasings. Staff first names next to a patient's name ("Dana and Priya came") are masked. +- "volunteers for IVF awareness" masks "IVF", and a long charity name after "volunteers for" is masked only in part. +- A bare subdomain without "https" or a visit cue ("jane.clinic.example") is not detected as a link. +- "badge number: 4821", "aged ninety four" and a generic Hebrew "the municipality" as an employer are not caught. +- "Your appointment is for January" is masked (a month tied to a person). - Over-masking costs answer quality, not privacy, but it is real: see the table above. Contributions are welcome, especially messages written by people who work in clinics (see ../CONTRIBUTING.md). All data must be synthetic. diff --git a/benchmark/blind/audit_v13.jsonl b/benchmark/blind/audit_v13.jsonl new file mode 100644 index 0000000..87b2e95 --- /dev/null +++ b/benchmark/blind/audit_v13.jsonl @@ -0,0 +1,58 @@ +{"id": "v13-en-sms-003", "action": "add", "type": "NAME", "surface": "Kim", "occurrence": 1, "reason": "patient first name alone"} +{"id": "v13-en-sms-021", "action": "add", "type": "NAME", "surface": "Hernández", "occurrence": 1, "reason": "patient surname alone"} +{"id": "v13-en-sms-006", "action": "add", "type": "NAME", "surface": "Marguerite", "occurrence": 1, "reason": "patient first name in greeting"} +{"id": "v13-en-chat-076", "action": "add", "type": "DATE", "surface": "09 30 1988", "occurrence": 1, "reason": "spoken-form date of birth"} +{"id": "v13-en-chat-079", "action": "add", "type": "NAME", "surface": "Hawa", "occurrence": 2, "reason": "repeat mention of patient name"} +{"id": "v13-en-chat-079", "action": "add", "type": "NAME", "surface": "Hawa", "occurrence": 3, "reason": "repeat mention of patient name"} +{"id": "v13-en-sms-008", "action": "add", "type": "NAME", "surface": "Y-O-L-A-N-D-A", "occurrence": 1, "reason": "first name spelled letter by letter"} +{"id": "v13-en-chat-055", "action": "add", "type": "NAME", "surface": "K O W A L C Z Y K", "occurrence": 1, "reason": "surname spelled letter by letter"} +{"id": "v13-en-sms-018", "action": "remove", "type": "ADDRESS", "surface": "Nashville", "occurrence": 1, "reason": "clinic's own city"} +{"id": "v13-en-chat-059", "action": "remove", "type": "ADDRESS", "surface": "Ottawa", "occurrence": 1, "reason": "clinic's own city"} +{"id": "v13-en-chat-095", "action": "remove", "type": "ADDRESS", "surface": "Nashville", "occurrence": 1, "reason": "clinic's own city"} +{"id": "v13-en-chat-100", "action": "remove", "type": "ADDRESS", "surface": "Ottawa", "occurrence": 2, "reason": "clinic's own city"} +{"id": "v13-en-call-105", "action": "add", "type": "NAME", "surface": "Marguerite", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-call-109", "action": "add", "type": "NAME", "surface": "Tamika", "occurrence": 2, "reason": "insurer rep name repeated"} +{"id": "v13-en-call-110", "action": "retype", "from": "NAME", "type": "PROVIDER", "surface": "Jorge", "occurrence": 1, "reason": "pharmacist"} +{"id": "v13-en-call-117", "action": "add", "type": "NAME", "surface": "Helen", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-call-117", "action": "add", "type": "NAME", "surface": "Helen", "occurrence": 3, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-call-121", "action": "add", "type": "NAME", "surface": "Kayla", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-call-121", "action": "add", "type": "NAME", "surface": "Kayla", "occurrence": 3, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-141", "action": "add", "type": "NAME", "surface": "Tessa", "occurrence": 2, "reason": "patient name in signature"} +{"id": "v13-en-email-147", "action": "retype", "from": "NAME", "type": "ID", "surface": "sunflower_mom82", "occurrence": 1, "reason": "username"} +{"id": "v13-en-email-147", "action": "retype", "from": "NAME", "type": "ID", "surface": "TTC_in_TN", "occurrence": 1, "reason": "username"} +{"id": "v13-en-email-145", "action": "add", "type": "NAME", "surface": "Samira", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-145", "action": "add", "type": "NAME", "surface": "Karim", "occurrence": 2, "reason": "signature"} +{"id": "v13-en-email-146", "action": "add", "type": "NAME", "surface": "Tyrell", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-153", "action": "add", "type": "NAME", "surface": "Chantal", "occurrence": 2, "reason": "signature"} +{"id": "v13-en-email-154", "action": "add", "type": "NAME", "surface": "Hannah", "occurrence": 2, "reason": "signature"} +{"id": "v13-en-email-156", "action": "add", "type": "NAME", "surface": "Kayla", "occurrence": 3, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-156", "action": "add", "type": "NAME", "surface": "Kayla", "occurrence": 4, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-161", "action": "add", "type": "NAME", "surface": "Beshoy", "occurrence": 2, "reason": "signature"} +{"id": "v13-en-email-163", "action": "add", "type": "NAME", "surface": "Priya", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-164", "action": "add", "type": "NAME", "surface": "Alma", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-167", "action": "add", "type": "NAME", "surface": "Ivy", "occurrence": 2, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-email-167", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "EMB-5532-1", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-email-167", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "EMB-5532-2", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-email-167", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "EMB-5532-3", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-email-169", "action": "add", "type": "NAME", "surface": "Eduardo", "occurrence": 2, "reason": "signature"} +{"id": "v13-en-email-171", "action": "add", "type": "NAME", "surface": "Jean-Marc", "occurrence": 2, "reason": "signature"} +{"id": "v13-en-email-172", "action": "add", "type": "NAME", "surface": "Bimala", "occurrence": 3, "reason": "unlabelled repeat of patient/relative name"} +{"id": "v13-en-note-176", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "E49913-1", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-note-176", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "E49913-2", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-note-176", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "E49913-3", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-note-176", "action": "retype", "from": "DONOR_ID", "type": "ID", "surface": "E49913-4", "occurrence": 1, "reason": "embryo ID is not donor ID"} +{"id": "v13-en-note-191", "action": "retype", "from": "NAME", "type": "ID", "surface": "nashville_nana_74", "occurrence": 1, "reason": "username"} +{"id": "v13-en-note-191", "action": "retype", "from": "NAME", "type": "ID", "surface": "nashville_nana_74", "occurrence": 2, "reason": "username"} +{"id": "v13-en-note-191", "action": "retype", "from": "NAME", "type": "ID", "surface": "BlueberryBabyDust", "occurrence": 1, "reason": "username"} +{"id": "v13-en-note-193", "action": "retype", "from": "NAME", "type": "ID", "surface": "@kiki_and_the_baby", "occurrence": 1, "reason": "social handle"} +{"id": "v13-en-note-193", "action": "retype", "from": "NAME", "type": "ID", "surface": "@kiki_and_the_baby", "occurrence": 2, "reason": "social handle"} +{"id": "v13-en-note-193", "action": "retype", "from": "NAME", "type": "ID", "surface": "@kiki_and_the_baby", "occurrence": 3, "reason": "social handle"} +{"id": "v13-en-note-193", "action": "retype", "from": "NAME", "type": "ID", "surface": "jasmine.t", "occurrence": 1, "reason": "social handle"} +{"id": "v13-en-note-193", "action": "retype", "from": "NAME", "type": "ID", "surface": "jasmine.t", "occurrence": 2, "reason": "social handle"} +{"id": "v13-en-call-118", "action": "add", "type": "NAME", "surface": "Bree", "occurrence": 2, "reason": "patient first name repeat"} +{"id": "v13-es-chat-019", "action": "remove", "type": "ADDRESS", "surface": "Nashville", "occurrence": 1, "reason": "clinic's own city is excluded by the rules"} +{"id": "v13-es-chat-036", "action": "remove", "type": "ADDRESS", "surface": "Nashville", "occurrence": 1, "reason": "clinic's own city is excluded by the rules"} +{"id": "v13-es-note-053", "action": "remove", "type": "ADDRESS", "surface": "Nashville", "occurrence": 1, "reason": "clinic's own city is excluded by the rules"} +{"id": "v13-pl-sms-002", "action": "add", "type": "NAME", "surface": "Kowalczyk", "occurrence": 1, "reason": "patient surname given as the whole message"} +{"id": "v13-pl-email-021", "action": "add", "type": "NAME", "surface": "Justyna", "occurrence": 2, "reason": "patient first name in signature"} +{"id": "v13-he-chat-010", "action": "add", "type": "PHONE", "surface": "615 555 0142", "occurrence": 1, "reason": "patient phone number"} diff --git a/benchmark/blind/blind_v13.source.jsonl b/benchmark/blind/blind_v13.source.jsonl new file mode 100644 index 0000000..b7405d4 --- /dev/null +++ b/benchmark/blind/blind_v13.source.jsonl @@ -0,0 +1,400 @@ +{"id":"v13-en-sms-001","channel":"sms","lang":"en","text":"Hi [[NAME|Dana]], this is Cumberland Fertility Partners. Your retrieval is confirmed for [[DATE|11/24]] at 7:15 AM. Nothing to eat or drink after midnight. Reply STOP to 72340 to opt out.","known":{"names":["Danielle Ruiz"],"phones":["305-555-0133"],"emails":[],"dates_of_birth":["1988-09-30"],"addresses":[],"identifiers":["MRN 0071245"]}} +{"id":"v13-en-sms-002","channel":"sms","lang":"en","text":"Reminder: your appt is on {date} at {time} with {provider_name}. Reply C to confirm or R to reschedule.","known":{}} +{"id":"v13-en-sms-003","channel":"sms","lang":"en","text":"Kim","known":{"names":["Kimberly Odom"],"phones":["615-555-0164"],"emails":[],"dates_of_birth":["1991-02-08"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-004","channel":"sms","lang":"en","text":"yes [[DATE|3/14/88]]","known":{"names":["Anna Kowalski"],"phones":["615-555-0120"],"emails":[],"dates_of_birth":["1988-03-14"],"addresses":[],"identifiers":["MRN 0042817"]}} +{"id":"v13-en-sms-005","channel":"sms","lang":"en","text":"[[PHONE|615 555 0142]]","known":{"names":["Rhonda Pickett"],"phones":["615-555-0142"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-006","channel":"sms","lang":"en","text":"Hi Marguerite, it's [[PROVIDER|Rojin]] from Cumberland. Your AMH came back at 1.1 and Dr. [[PROVIDER|Barzani]] would like to talk it through. Can you do Friday the [[DATE|20th]]?","known":{"names":["Marguerite Leblanc"],"phones":["613-555-0187"],"emails":[],"dates_of_birth":["1985-06-11"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-007","channel":"sms","lang":"en","text":"CFP: Your lab requisition REQ-88213 is ready at the Suite 410 front desk. Specimen accession ACC-2026-004417. Hours Mon-Fri 7-4. We are closed Thanksgiving Day.","known":{}} +{"id":"v13-en-sms-008","channel":"sms","lang":"en","text":"its [[NAME|Yolanda]], Y-O-L-A-N-D-A, last name [[NAME|B-A-R-R-A-Z-A]]. calling about my bloodwork from yesterday","known":{"names":["Yolanda Barraza"],"phones":["615-555-0111"],"emails":[],"dates_of_birth":["1993-12-02"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-009","channel":"sms","lang":"en","text":"This is [[NAME|Tomás]], [[NAME|Elena]]'s husband. She is awake and asking when Dr. [[PROVIDER|Cortés]] will come by. Her phone is dead.","known":{"names":["Elena Vargas Ruiz"],"phones":["615-555-0158"],"emails":[],"dates_of_birth":["1987-07-19"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-010","channel":"sms","lang":"en","text":"Member ID for the new plan: [[HEALTH_CARD|XQB448120937]] group [[ACCOUNT|7731-0022]]. Name on the card is [[NAME|Priya Raman]]","known":{"names":["Priya Raman"],"phones":["615-555-0139"],"emails":[],"dates_of_birth":["1990-01-25"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-011","channel":"sms","lang":"en","text":"ok thanks!! see you at 9:40. should I do the trigger shot at 10pm tonight like we talked? the vial says lot 4471B exp 2027-03","known":{}} +{"id":"v13-en-sms-012","channel":"sms","lang":"en","text":"Hi, we moved. New address is [[ADDRESS|418 Gallatin Pike S]], [[ADDRESS|Madison]] [[ZIP|37115]]. Please update before you mail the consent forms.","known":{"names":["Lamar Tillman"],"phones":["615-555-0126"],"emails":[],"dates_of_birth":[],"addresses":["77 Dickerson Pike, Nashville, TN 37207"],"identifiers":[]}} +{"id":"v13-en-sms-013","channel":"sms","lang":"en","text":"Bonjour, my OHIP number is [[HEALTH_CARD|4829 317 560 QK]] and my birthday is [[DATE|June 2, 1991]]. Sorry I forgot at the visit.","known":{"names":["Camille Tremblay"],"phones":["613-555-0152"],"emails":[],"dates_of_birth":["1991-06-02"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-014","channel":"sms","lang":"en","text":"the [[DATE|12th]]","known":{}} +{"id":"v13-en-sms-015","channel":"sms","lang":"en","text":"Hi [[NAME|Reem]], your appt on [[DATE|12/03]] at 8:00 AM is confirmed at Cumberland Fertility Partners. Reply C to confirm, R to reschedule.","known":{"names":["Reem Al-Hamad"],"phones":["615-555-0171"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-016","channel":"sms","lang":"en","text":"Sorry wrong number, I'm not [[NAME|Tasha]]. Please stop texting this number.","known":{}} +{"id":"v13-en-sms-017","channel":"sms","lang":"en","text":"Your Ottawa visit is booked for 2:30 PM, Suite 905, 340 Laurier Avenue West. Parking is validated at the front desk. Reply STOP to 72340 to opt out.","known":{}} +{"id":"v13-en-sms-018","channel":"sms","lang":"en","text":"I'm in the Safe at Home program. Mail goes to [[ADDRESS|PO Box 7719]], [[ADDRESS|Nashville]] [[ZIP|37202]] and my participant number is [[ID|SAH-0049281]]. Do not call the house, text only to [[PHONE|615-555-0188]].","known":{"names":["Alison Verhoeven"],"phones":["615-555-0188"],"emails":[],"dates_of_birth":["1989-04-17"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-019","channel":"sms","lang":"en","text":"Dr [[PROVIDER|Whitfeild]]'s office said to text this number about my results. Its [[NAME|Bethany Ostrowski]]","known":{"names":["Bethany Ostrowski"],"phones":["615-555-0192"],"emails":[],"dates_of_birth":["1992-08-05"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-020","channel":"sms","lang":"en","text":"Embryo update: 2 blasts, 4AA and 3BB, both biopsied day 5. PGT-A results in about 10 days. CPT 89258 for the freeze will show on your statement.","known":{}} +{"id":"v13-en-sms-021","channel":"sms","lang":"en","text":"Hernández","known":{"names":["Lucía Hernández"],"phones":["615-555-0135"],"emails":[],"dates_of_birth":["1994-10-21"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-022","channel":"sms","lang":"en","text":"Tell the nurse my mother [[NAME|Bridget]] had breast cancer, she passed at [[AGE|91]]. My sister [[NAME|Maeve]] has endometriosis.","known":{"names":["Siobhan O'Donnell"],"phones":["613-555-0114"],"emails":[],"dates_of_birth":["1986-03-03"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-023","channel":"sms","lang":"en","text":"We are closed Dec 24-26 and Jan 1 for the holidays. For anything urgent call 615-555-0701 and press 6 for the nurse line.","known":{}} +{"id":"v13-en-sms-024","channel":"sms","lang":"en","text":"my email is [[EMAIL|ivy.nakamura88@example.com]] can you resend the portal invite","known":{"names":["Ivy Nakamura"],"phones":["615-555-0149"],"emails":["ivy.nakamura@example.net"],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-025","channel":"sms","lang":"en","text":"Hi Dr. [[PROVIDER|Ghali]] thanks for the call. Husband [[NAME|Mehdi]] says he will drop the sample at 8 on Tuesday the [[DATE|17th]]","known":{"names":["Soraya Karimi"],"phones":["615-555-0107"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-026","channel":"sms","lang":"en","text":"Invoice [[ACCOUNT|INV-2026-118830]] for $1,850.00 is past due. Pay by [[DATE|Nov 30]] at portal.cumberland-fp.example.org or call billing at 615-555-0701.","known":{}} +{"id":"v13-en-sms-027","channel":"sms","lang":"en","text":"lol hubby nearly fainted over the sample cup. will bring it by 8:15, the cup has a lot sticker 22A-7731 on it right?","known":{}} +{"id":"v13-en-sms-028","channel":"sms","lang":"en","text":"Employer form needs my SIN [[SIN|291 735 408]], is it ok to text?? tell me no if not","known":{"names":["Colette Gagnon"],"phones":["613-555-0178"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-029","channel":"sms","lang":"en","text":"I work at [[EMPLOYER|Harpeth Valley Utilities]] and HR says short-term disability starts [[DATE|Dec 1]]. Claim number [[ACCOUNT|STD-5520718]].","known":{"names":["Monique Daugherty"],"phones":["615-555-0103"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-030","channel":"sms","lang":"en","text":"I have hope this cycle works. Is there a grace period on the bill? Also can I wear a rose gold bracelet to retrieval or does all jewelry have to come off?","known":{}} +{"id":"v13-en-sms-031","channel":"sms","lang":"en","text":"Tue 8:30 works for me. Can I bring my sister into the room?","known":{}} +{"id":"v13-en-sms-032","channel":"sms","lang":"en","text":"Hi it's [[NAME|Aminata]]. I live in [[ADDRESS|Vanier]] now, [[ADDRESS|214 Montreal Road Apt 6]] [[POSTAL_CODE|K1L 6C5]]. Can the Ottawa nurse call me after 5?","known":{"names":["Aminata Diallo"],"phones":["613-555-0121"],"emails":[],"dates_of_birth":[],"addresses":["88 Beechwood Ave, Ottawa, ON K1M 1L7"],"identifiers":[]}} +{"id":"v13-en-sms-033","channel":"sms","lang":"en","text":"Best number is [[PHONE|(615) 555-0173]] and best time is after 6","known":{"names":["Delphine Okafor"],"phones":["615-555-0173"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-034","channel":"sms","lang":"en","text":"Our donor ID is [[DONOR_ID|D-48812-C]] and the straws are at Northgate Cryobank. Can you confirm they arrived?","known":{"names":["Marlena Hoyt"],"phones":["615-555-0167"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-035","channel":"sms","lang":"en","text":"Reminder: cycle monitoring is tomorrow at 7:00 AM, Suite 410, Room 3. Please arrive 10 min early. Ticket CFP-20931.","known":{}} +{"id":"v13-en-sms-036","channel":"sms","lang":"en","text":"Hi, [[NAME|Jamal]] here, I'm [[NAME|Keisha Whitaker-Reed]]'s partner. She asked me to cancel her IUI on [[DATE|11/28]] since her mom is in the hospital.","known":{"names":["Keisha Whitaker-Reed"],"phones":["615-555-0182"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-037","channel":"sms","lang":"en","text":"Sent the forms in. Confirmation number from the insurer for the prior auth is [[ACCOUNT|PA-2026-3318842]]","known":{"names":["Noreen Castellano"],"phones":["615-555-0145"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-038","channel":"sms","lang":"en","text":"My last name is spelled [[NAME|gee you zed em ay en]], [[NAME|Guzman]], and first name [[NAME|Marisela]]. You had me down wrong","known":{"names":["Marisela Guzmán"],"phones":["615-555-0156"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-039","channel":"sms","lang":"en","text":"Sorry I missed your call. Can [[PROVIDER|Marcus]] from the lab call me back about my sample?","known":{"names":["Troy Henley"],"phones":["615-555-0138"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-040","channel":"sms","lang":"en","text":"Passport for the records transfer is [[ID|C0482917]], issued in Canada. Husband's is [[ID|P7731884K]]","known":{"names":["Farah Nasser"],"phones":["613-555-0133"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-041","channel":"sms","lang":"en","text":"Thx. 2 follicles at 17 and 19mm, E2 1480, LH 4.1. Trigger tonight 9:30. ICD O09.82 on the visit code, just fyi no worries.","known":{}} +{"id":"v13-en-sms-042","channel":"sms","lang":"en","text":"STOP","known":{}} +{"id":"v13-en-sms-043","channel":"sms","lang":"en","text":"Can you call [[NAME|Ruth Abara]] instead? She is my sister and has my consent. Her cell is [[PHONE|+1 613-555-0166]]","known":{"names":["Esther Abara"],"phones":["613-555-0119"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-044","channel":"sms","lang":"en","text":"Protective order case [[ID|2026-OP-004417]] is in effect, [[NAME|Derek Lindqvist]] must not get any information. Talk only to me.","known":{"names":["Whitney Strand"],"phones":["615-555-0198"],"emails":[],"dates_of_birth":[],"addresses":[],"identifiers":[]}} +{"id":"v13-en-sms-045","channel":"sms","lang":"en","text":"My birthday is [[DATE|September 30, 1988]] and my husband's is [[DATE|1/9/86]]","known":{"names":["Danielle Ruiz"],"phones":["305-555-0133"],"emails":[],"dates_of_birth":["1988-09-30"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-chat-046","channel":"chat","lang":"en","text":"Visitor: do you take Tricare?\nAssistant: Cumberland Fertility Partners works with many plans. Coverage for fertility treatment varies, so please call 615-555-0700 and ask for billing.\nVisitor: ok thanks","known":{}} +{"id":"v13-en-chat-047","channel":"chat","lang":"en","text":"Subject: Question about my results\nHi Nurse [[PROVIDER|Girgis]], my beta was 214 on [[DATE|11/18]]. Is that a good number? I'm trying not to spiral. Thanks,\n[[NAME|Anjali Deshmukh]]","known":{"names":["Anjali Deshmukh"],"phones":["615-555-0143"],"emails":[],"dates_of_birth":["1989-05-27"],"addresses":[],"identifiers":["MRN 0063391"]}} +{"id":"v13-en-chat-048","channel":"chat","lang":"en","text":"Assistant: Hi, I'm the Cumberland Fertility virtual assistant. How can I help?\nPatient: I need to reschedule my transfer\nAssistant: I can help with that. Can I get your full name and date of birth?\nPatient: [[NAME|Fatima Yusuf]] [[DATE|04/22/1990]]\nAssistant: Thanks [[NAME|Fatima]]. I see an appointment on [[DATE|December 2]]. I'm connecting you with a nurse now. [[PROVIDER|Hodan]] will pick up shortly.","known":{"names":["Fatima Yusuf"],"phones":["615-555-0109"],"emails":[],"dates_of_birth":["1990-04-22"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-chat-049","channel":"chat","lang":"en","text":"Visitor: How much is egg freezing?\nAssistant: A freezing cycle typically includes monitoring, retrieval (CPT 58970) and first-year storage. The current fee schedule is on the Pricing page, or I can book a consult at Suite 410.\nVisitor: Can I do it in Ottawa?\nAssistant: Yes, our Ottawa office at 340 Laurier Avenue West offers monitoring. Retrievals are done in Nashville.","known":{}} +{"id":"v13-en-chat-050","channel":"chat","lang":"en","text":"Hello, my employer needs FMLA certification for my retrieval. Employee number [[ID|E-208841]], employer is [[EMPLOYER|Riverbend Logistics]]. HR fax is [[PHONE|615-555-0119]]. Please have Dr. [[PROVIDER|Whitfield]] sign it. Thank you, [[NAME|Corinne Maddox]]","known":{"names":["Corinne Maddox"],"phones":["615-555-0181"],"emails":[],"dates_of_birth":["1987-11-09"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-chat-051","channel":"chat","lang":"en","text":"Assistant: What is your last name?\nVisitor: [[NAME|Hernández]]\nAssistant: And your date of birth?\nVisitor: [[DATE|3/14/88]]\nAssistant: And a phone number we can reach you on?\nVisitor: [[PHONE|615 555 0142]]\nAssistant: Thank you, Ms. [[NAME|Hernández]]. A coordinator will message you back within one business day.","known":{"names":["Rosa Hernández"],"phones":["615-555-0142"],"emails":[],"dates_of_birth":["1988-03-14"],"addresses":[],"identifiers":[]}} +{"id":"v13-en-chat-052","channel":"chat","lang":"en","text":"Assistant: Hello [PATIENT NAME], your next visit is <> at <