Skip to content

Blind v13: new set, fixes, narrower clinic-site and staff passes - #31

Merged
Sergei-gorlovetsky merged 3 commits into
mainfrom
blind-v13
Nov 21, 2026
Merged

Sergei-gorlovetsky merged 3 commits into
mainfrom
blind-v13

Conversation

@Sergei-gorlovetsky

Copy link
Copy Markdown
Contributor

Adds blind benchmark set v13 and the fixes made after its single frozen score.

The set (commit 3d01ff6): 400 hand-written messages from four blind authors (Nashville clinic with an Ottawa office): document page headers, forum posts and profile links, work and benefits paperwork, safety and legal paperwork, automation output, one-word replies. Two blind annotators and an adjudicator per 100 messages; 58 label corrections.

First score (frozen 0.1.0a14, all detectors, default policy): 95.8% recall (v12's was 93.8%). With clinic details and staff list: 95.6% recall, 87.8% precision.

Fixes

  • Names in page headers ("LOPEZ, MARIA T" above "Page 1 of 3"), English letter-name spelling, sign-off name lists, Nepali/Hindi and Lao names, bylines, couples.
  • Non-Latin digit dates, Somali dates, spaced DOBs after a DOB cue, months tied to a person, ages over 89 after "lived to" and in brackets.
  • Forum usernames, plates, passports, CPF, ITIN/SSN lists and separators, last-four answers, employee/claim/member numbers, profile and link-in-bio URLs.
  • Volunteer/driver employers, Hebrew workplace and residence cues.
  • Narrowed pass-throughs (privacy): the clinic-website allowance now passes only the bare clinic address (plus generic labels such as "portal" and generic pages); a patient subdomain, profile path or record number is masked, and a path-style allow entry no longer opens the whole host. The staff list no longer lets a patient's first name through because a non-staff full name with the same first name sat after a "Cc:" cue.
  • Speed: merge_entities uses bisection (identical output on 20,000 random cases) and the echo run is bounded; a 100 KB repeated-name message drops from ~20-90 s to under 2 s.

Checks

  • Three review rounds (25 + 14 + 8 confirmed findings), every fix re-checked by an independent verifier against the frozen code.
  • Regression vs frozen on v0-v13 in four configurations (regex+known and all detectors, with and without the staff list): no identifier the frozen code caught is missed. New false positives on older sets: four clinic links with a page path, one partial health-card readback, and (with the staff list) seven bare staff first names.
  • v13 now: 100% recall with all detectors and the staff list (after fixes made with it; the first score is the fair measure); regex+known 92.7% (new CI gate at 0.90).
  • 2,193 tests pass. Mean 4.0 ms per message vs 3.3 ms before (loaded machine).

No release in this PR; publishing 0.1.0a15 needs a separate owner go.

🤖 Generated with Claude Code

Sergei-gorlovetsky and others added 3 commits November 21, 2026 08:53
Fictional Nashville clinic with an Ottawa office. Four blind authors wrote
100 messages each from a brief aimed at clinical documents with patient
headers and footers, public posts and reviews, work and benefits
paperwork, safety and legal records, automation output (templates with
placeholders, phone-menu logs) and one-word replies. Two blind annotators
per 100 messages and an adjudicator kept 58 corrections. Scored once
against 59f1f80 (0.1.0a14) before any change: 95.8% recall with all
detectors (87.7% with regex + known entities).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…Bs, narrower clinic-site and staff passes

Blind v13 (400 messages, Nashville + Ottawa brief) scored 95.8% recall at first
against frozen 0.1.0a14. Fixes after that score:

- Names: LAST, FIRST M in document page headers and footers; English
  letter-name spelling; sign-off first-name lists; Devanagari and Lao names;
  Hebrew surname after "in Hebrew"; "Posted by: Jane D."; couple second names.
- Dates and ages: Arabic, Persian and Devanagari digit dates, Somali dates,
  spaced DOB after a DOB cue, month alone tied to a person, ages over 89 after
  "lived to" and in brackets.
- IDs: forum usernames, plates, passports, CPF, ITIN/SSN lists and separators,
  last-four answers, employee/claim/member numbers, profile and link-in-bio URLs.
- Employers and addresses: volunteer/driver employers, Hebrew workplace and
  residence cues.
- Pass-throughs narrowed: Masker._own_site only passes the bare clinic host
  (plus generic labels and pages) and ignores path-style allow entries;
  StaffList.cued_words needs all-staff spans and the echo pass uses
  StaffList.passes, so a patient sharing a staff first name stays masked.
- Speed: merge_entities uses bisection and the echo run is bounded; a 100 KB
  repeated-name message drops from ~20-90 s to under 2 s.

Three review rounds (25 + 14 + 8 confirmed findings), each re-checked against
the frozen code. No identifier the frozen code caught is missed on v0-v13 in
any of four configurations. v13 now 100% with all detectors and the staff list
(after fixes made with it; the 95.8% first score is the fair measure).
Docs, results and a v13 CI gate are updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
The whole detector takes about 2.5 s on the slowest input locally (0.1.0a14:
2.4 s); a CI runner took 5.2 s against the 5 s limit. Raise it to 15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
@Sergei-gorlovetsky
Sergei-gorlovetsky merged commit d122b87 into main Nov 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant