Repository navigation
Blind v9 and fixes: structured text, case numbers, health numbers, nicknames - #24
Merged
Merged
Conversation
Written by four blind authors from one brief (Denver clinic with a Calgary office: system dumps, long legal and clinical documents, names that look like words, unusual identifier formats). Two blind annotators per 100 messages and an adjudicator kept 47 label corrections. Scored once against 3aaa537 (0.1.0a11) before any change: 93.4% recall with the local model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…icknames A structured-text reader finds values by field name in JSON, key: value lines, simple XML, CSV columns under a header row and HL7 v2 PID, NK1, GT1 and IN1 segments; each value must look like what its field says. New rules: court and case numbers (Colorado, Polish, Israeli, "Case no."), Rx numbers, Colorado licences, cedulas, plates, usernames, Alberta health numbers and Medicare, digits read one at a time after a record word, card last four in five languages and masked first4/last4 cards, donor numbers with words between, phones after Hebrew prefixes, trunk "(0)" and "00" prefixes, Polish landlines, ages over 89 of "the patient", ISO timestamps, nicknames, greeting-line addressees, "<Name> and I", more closings, Tagalog and Vietnamese relation words. CRLF text is matched as LF. Four rounds of adversarial review confirmed 67 problems in drafts of these rules (including three privacy regressions from earlier review fixes); all fixed with tests. v9 now: 98.5% recall with clinic and staff lists (first score 93.2%); no identifier lost on v0 to v8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blind benchmark v9 and the fixes it led to.
Data (d91e75a, committed before any fix)
audit_v9.jsonl.Fixes (78955e7)
New
detectors/_structured.py. It finds values by their field name in:key: valuelines and simple XMLEach value must look like what its field says. Ambiguous keys such as
name,firstornomcount only beside a field that is surely personal.New rules:
(0)or a00prefix, and Polish landlinesReview. Four rounds of adversarial review ran on the drafts, and every claim was re-run before it counted. They confirmed 67 problems, including three privacy regressions that earlier review fixes had introduced. All were fixed and turned into tests: 586 pass.
CI. New gate: v9 regex+known must stay at or above 90% (it scores 91.7%).
Before/after on every set
The frozen 3aaa537 code was compared with this branch on v0 to v9:
Known label calls left as annotated: one bare year marked as a date, and 3 bank routing numbers marked as accounts (v6 left them unlabelled, and the code follows v6). Both are documented in
benchmark/README.md.🤖 Generated with Claude Code