Repository navigation
Blind v13: new set, fixes, narrower clinic-site and staff passes - #31
Merged
Merged
Conversation
Fictional Nashville clinic with an Ottawa office. Four blind authors wrote 100 messages each from a brief aimed at clinical documents with patient headers and footers, public posts and reviews, work and benefits paperwork, safety and legal records, automation output (templates with placeholders, phone-menu logs) and one-word replies. Two blind annotators per 100 messages and an adjudicator kept 58 corrections. Scored once against 59f1f80 (0.1.0a14) before any change: 95.8% recall with all detectors (87.7% with regex + known entities). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…Bs, narrower clinic-site and staff passes Blind v13 (400 messages, Nashville + Ottawa brief) scored 95.8% recall at first against frozen 0.1.0a14. Fixes after that score: - Names: LAST, FIRST M in document page headers and footers; English letter-name spelling; sign-off first-name lists; Devanagari and Lao names; Hebrew surname after "in Hebrew"; "Posted by: Jane D."; couple second names. - Dates and ages: Arabic, Persian and Devanagari digit dates, Somali dates, spaced DOB after a DOB cue, month alone tied to a person, ages over 89 after "lived to" and in brackets. - IDs: forum usernames, plates, passports, CPF, ITIN/SSN lists and separators, last-four answers, employee/claim/member numbers, profile and link-in-bio URLs. - Employers and addresses: volunteer/driver employers, Hebrew workplace and residence cues. - Pass-throughs narrowed: Masker._own_site only passes the bare clinic host (plus generic labels and pages) and ignores path-style allow entries; StaffList.cued_words needs all-staff spans and the echo pass uses StaffList.passes, so a patient sharing a staff first name stays masked. - Speed: merge_entities uses bisection and the echo run is bounded; a 100 KB repeated-name message drops from ~20-90 s to under 2 s. Three review rounds (25 + 14 + 8 confirmed findings), each re-checked against the frozen code. No identifier the frozen code caught is missed on v0-v13 in any of four configurations. v13 now 100% with all detectors and the staff list (after fixes made with it; the 95.8% first score is the fair measure). Docs, results and a v13 CI gate are updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
The whole detector takes about 2.5 s on the slowest input locally (0.1.0a14: 2.4 s); a CI runner took 5.2 s against the 5 s limit. Raise it to 15 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds blind benchmark set v13 and the fixes made after its single frozen score.
The set (commit 3d01ff6): 400 hand-written messages from four blind authors (Nashville clinic with an Ottawa office): document page headers, forum posts and profile links, work and benefits paperwork, safety and legal paperwork, automation output, one-word replies. Two blind annotators and an adjudicator per 100 messages; 58 label corrections.
First score (frozen 0.1.0a14, all detectors, default policy): 95.8% recall (v12's was 93.8%). With clinic details and staff list: 95.6% recall, 87.8% precision.
Fixes
merge_entitiesuses bisection (identical output on 20,000 random cases) and the echo run is bounded; a 100 KB repeated-name message drops from ~20-90 s to under 2 s.Checks
No release in this PR; publishing 0.1.0a15 needs a separate owner go.
🤖 Generated with Claude Code