Skip to content

Blind v11: phones in pieces, addresses, payment details, patient initials - #27

Merged
Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v11
Nov 20, 2026
Merged

Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v11

Conversation

@Sergei-gorlovetsky

Copy link
Copy Markdown
Contributor

Summary

  • New blind set v11 (400 hand-written messages, 1,408 labelled identifiers + 277 staff names, 68 audit corrections), scored once against frozen 7fb0f01: 88.4% recall with all detectors (72.0% precision default, 84.0% with clinic details and staff list).
  • Fixes for the gaps it showed: phone numbers given in pieces across lines or turns, grouped national numbers after a phone cue, rural and hard addresses (rural routes, GPS, counties, reserves, directions to a home), payment handles and bank numbers, a patient's initials after a patient cue, dates in photo file names, fewer false dates.
  • Three features were tried and removed during review because each kept leaking patient data in new ways: allowing the clinic street without its full allow entry, letting staff initials pass, and leaving clinic closure and policy "effective" dates visible.
  • Review: 4 rounds + 8 final rounds of adversarial review confirmed 135 problems, every claim re-run by a separate verifier; all fixed or listed as known limits. 1,685 tests.

Results

  • v11 now, all detectors with clinic details and staff list: recall 88.4% -> 99.3%, precision 84.0% -> 86.2%.
  • All blind sets now 96.4% to 99.3% with all detectors (results files regenerated).
  • Regression check vs frozen 7fb0f01 on v0 to v11, four configurations (regex+known and all detectors, with and without the staff list): no identifier the frozen code caught is missed.
  • Speed: 2.9 ms per message on the 4,380 blind messages (2.0 ms before), worst case 15 ms.
  • New CI gate: v11 regex+known >= 0.90 (now 91.1%).

Notes

  • Not a release: the PyPI publish of 0.1.0a13 needs a separate go.
  • A client-derived example domain in a v10 docstring and test was replaced with a neutral one.

🤖 Generated with Claude Code

Sergei-gorlovetsky and others added 2 commits November 20, 2026 08:25
Written by four blind authors from one brief (Atlanta clinic with a
Halifax office: long threads with numbers given in pieces and read back,
staff shorthand with patient initials, rural routes, counties, reserves
and GPS coordinates, payment handles and bank numbers, photo and
attachment names, foster care and adoption). Two blind annotators per 100
messages and an adjudicator kept 68 label corrections. Scored once
against 7fb0f01 before any change: 88.4% recall with the local model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…nitials

Phone numbers given in pieces across lines or turns after a phone cue
(area code, exchange, line number, a close correction), national numbers
in groups after a phone cue, "plus"/"mas" + country code said as a word,
Vietnamese spoken digits, spelled emails and names with commas, dashes or
dotted letters.

Addresses (cue-gated, new _geo module): rural routes, GPS pairs,
directions to a home, counties, First Nation reserves, military and campus
addresses, colonias, bare street or town replies to a "where do you
live?" question, towns after residence verbs in five languages; not
directions to the clinic, public institutions, countries or continents.

Accounts and IDs (new _accounts module): payment handles and cashtags
(not the clinic's own in a "pay us" sentence), transit, routing and
grouped account numbers, cheque numbers with a first-person cue,
usernames, personal device serials, court and military file numbers,
grouped national IDs and health cards, licence plates, the last four
digits given in a reply.

Names: a patient's initials after a patient cue, with about 300 clinical
and chart abbreviations excluded; Polish case forms of a found surname;
a found handle repeated bare. Dates in photo file names; fewer false
dates (scores, BMI and lab decimals, equipment upkeep). Bounded spoken
lookback and no int() limit crash on long digit runs.

Removed during review because each kept leaking patient data in new
ways: allowing the clinic street without its full allow entry, letting
staff initials pass, and leaving clinic closure and "effective" dates
visible. Four review rounds and eight final rounds confirmed 135
problems; all fixed or listed as known limits (1,685 tests).

v11 with clinic details and staff list: recall 88.4% -> 99.3%,
precision 84.0% -> 86.2%. No identifier the frozen code caught is missed
on v0 to v11 in four configurations. Results files regenerated, docs and
a v11 CI gate added. A client-derived example domain in a docstring and
a test was replaced with a neutral one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
@Sergei-gorlovetsky
Sergei-gorlovetsky merged commit 11334db into main Nov 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant