Repository navigation
Blind v10: 400 new messages, fixes for spoken and obfuscated identifiers - #26
Merged
Merged
Conversation
Written by four blind authors from one brief (Minneapolis clinic with a Winnipeg office: interpreted and spoken calls, half-hidden identifiers, quoted and forwarded email, hard date forms, names in odd places, patients who are doctors). Two blind annotators per 100 messages and an adjudicator kept 63 label corrections. Scored once against 8db92f4 (0.1.0a12) before any change: 89.7% recall with the local model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…of names
Emails written to dodge filters ([at], (at), {at}) or spoken in five
languages; social handles and usernames (ID); personal sites without https
after a personal cue. Numbers read one digit or letter at a time, "dash",
group numbers spoken as words, Israeli IDs in groups, phones split across
a line or with hidden last digits, spelled names ("F as in Foxtrot").
Date ranges, xx-masked birth dates, dates in file names, abbreviated
Spanish/Polish/French months, Hebrew calendar dates; fractions, lab
values, product expiry and billing periods are not dates. US court file
numbers, partial SSN/SIN after "ends in", card numbers after a card cue,
health cards with typos, spoken ages over 89. Names in file names, plural
family names, initials, patients who are doctors.
Masker: the clinic allow list matches spelling variants of the clinic's
address; a name found once is masked on its later mentions (ordinary-word
names only in a person context); a patient's city and ZIP next to their
street stay masked; invisible characters and Hebrew vowel points are
ignored for matching with exact offsets.
Five rounds of adversarial review confirmed 81 problems in drafts of these
rules, 17 of them filters hiding what the old code masked; all fixed with
tests. v10 now: 99.2% recall with clinic and staff lists (first score
89.4%); no identifier lost on v0 to v9 in any configuration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blind benchmark v10 and the fixes it pointed to.
Data (160a35f). 400 hand-written messages by four blind authors from a new brief: a Minneapolis clinic with a Winnipeg office, calls through phone interpreters, numbers and emails spoken as words or half-hidden, quoted and forwarded email, hard date forms, names in odd places. Two blind annotators per 100 messages and an adjudicator kept 63 label corrections. Scored once against 8db92f4 (0.1.0a12) before any change: 89.7% recall with all detectors (v9 was 93.4%), 81.3% with regex + known entities.
Fixes (b5fd6d9).
ID), personal sites without https after a personal cue.Review. Five rounds of adversarial review, with every finding re-run by a separate verifier, confirmed 81 problems in the drafts. 17 of them were new filters hiding something the old code masked. All are fixed and covered by tests (861 tests).
Results.
🤖 Generated with Claude Code