Skip to content

Blind v10: 400 new messages, fixes for spoken and obfuscated identifiers - #26

Merged
Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v10
Nov 19, 2026
Merged

Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v10

Conversation

@Sergei-gorlovetsky

Copy link
Copy Markdown
Contributor

Blind benchmark v10 and the fixes it pointed to.

Data (160a35f). 400 hand-written messages by four blind authors from a new brief: a Minneapolis clinic with a Winnipeg office, calls through phone interpreters, numbers and emails spoken as words or half-hidden, quoted and forwarded email, hard date forms, names in odd places. Two blind annotators per 100 messages and an adjudicator kept 63 label corrections. Scored once against 8db92f4 (0.1.0a12) before any change: 89.7% recall with all detectors (v9 was 93.4%), 81.3% with regex + known entities.

Fixes (b5fd6d9).

  • Obfuscated and spoken emails, social handles and usernames (ID), personal sites without https after a personal cue.
  • Numbers and codes read one digit or letter at a time, group numbers spoken as words, Israeli IDs in groups, phones split across a line, spelled names.
  • Date ranges, xx-masked birth dates, dates in file names, abbreviated months, Hebrew calendar dates; fractions, lab values, product expiry and billing periods are not dates, while card and document expiry stay masked.
  • US court file numbers, partial SSN/SIN, card numbers after a card cue, health cards with typos, spoken ages over 89.
  • Masker: the allow list matches spelling variants of the clinic's own address; a name found once is masked on later mentions (ordinary-word names only in a person context); invisible characters and Hebrew vowel points are ignored for matching.

Review. Five rounds of adversarial review, with every finding re-run by a separate verifier, confirmed 81 problems in the drafts. 17 of them were new filters hiding something the old code masked. All are fixed and covered by tests (861 tests).

Results.

  • v10 with clinic details and the staff list: recall 89.4% → 99.2%, precision 83.8% → 87.2%.
  • I compared the frozen code with the current code on v0 to v10 in four configurations: regex + known and all detectors, each with and without the staff list. No identifier the frozen code caught is missed now in any of them.
  • False positives went down on every set except v7 with regex + known.
  • Current recall with all detectors is 96% to 99% on the ten blind sets.
  • CI gains a v10 gate: regex + known must stay at or above 90% recall. It is at 93.4% now.

🤖 Generated with Claude Code

Sergei-gorlovetsky and others added 2 commits November 19, 2026 11:20
Written by four blind authors from one brief (Minneapolis clinic with a
Winnipeg office: interpreted and spoken calls, half-hidden identifiers,
quoted and forwarded email, hard date forms, names in odd places, patients
who are doctors). Two blind annotators per 100 messages and an adjudicator
kept 63 label corrections. Scored once against 8db92f4 (0.1.0a12) before
any change: 89.7% recall with the local model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…of names

Emails written to dodge filters ([at], (at), {at}) or spoken in five
languages; social handles and usernames (ID); personal sites without https
after a personal cue. Numbers read one digit or letter at a time, "dash",
group numbers spoken as words, Israeli IDs in groups, phones split across
a line or with hidden last digits, spelled names ("F as in Foxtrot").
Date ranges, xx-masked birth dates, dates in file names, abbreviated
Spanish/Polish/French months, Hebrew calendar dates; fractions, lab
values, product expiry and billing periods are not dates. US court file
numbers, partial SSN/SIN after "ends in", card numbers after a card cue,
health cards with typos, spoken ages over 89. Names in file names, plural
family names, initials, patients who are doctors.

Masker: the clinic allow list matches spelling variants of the clinic's
address; a name found once is masked on its later mentions (ordinary-word
names only in a person context); a patient's city and ZIP next to their
street stay masked; invisible characters and Hebrew vowel points are
ignored for matching with exact offsets.

Five rounds of adversarial review confirmed 81 problems in drafts of these
rules, 17 of them filters hiding what the old code masked; all fixed with
tests. v10 now: 99.2% recall with clinic and staff lists (first score
89.4%); no identifier lost on v0 to v9 in any configuration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
@Sergei-gorlovetsky
Sergei-gorlovetsky merged commit 7fb0f01 into main Nov 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant