Skip to content

Blind v9 and fixes: structured text, case numbers, health numbers, nicknames - #24

Merged
Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v9
Nov 19, 2026
Merged

Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v9

Conversation

@Sergei-gorlovetsky

Copy link
Copy Markdown
Contributor

Blind benchmark v9 and the fixes it led to.

Data (d91e75a, committed before any fix)

  • 400 messages written by hand by four blind authors from one brief: a Denver clinic with a Calgary office. The brief covered record-system dumps, CSV, JSON and HL7 text, web forms with IP addresses, long legal and clinical documents, names that look like words or places, multi-part names, unusual ID formats, and known-donor and two-parent families.
  • Two blind annotators per 100 messages plus an adjudicator: 47 label corrections in audit_v9.jsonl.
  • Scored once against 3aaa537 (0.1.0a11): 93.4% recall with all detectors and 84.4% without the local model. Precision was 73.0% with the default policy, 82.2% with the clinic details allowed and 85.2% with the staff list too.

Fixes (78955e7)

  • New detectors/_structured.py. It finds values by their field name in:

    • JSON, including camelCase keys and Python dicts
    • key: value lines and simple XML
    • CSV columns under a header row
    • HL7 v2 PID, NK1, GT1 and IN1 segments

    Each value must look like what its field says. Ambiguous keys such as name, first or nom count only beside a field that is surely personal.

  • New rules:

    • case numbers: Colorado, Polish sygnatura, Israeli, "Case no."
    • Rx numbers, Colorado licences, cédulas, plates and usernames
    • Alberta health numbers and Medicare
    • digits read one at a time after a record word
    • card last-four in five languages, and first-four/last-four masked cards
    • donor numbers with words in between
    • phones after a Hebrew prefix, with a trunk (0) or a 00 prefix, and Polish landlines
    • ages over 89 of "the patient", and ISO timestamps
    • names: nicknames, an addressee alone on a greeting line, " and I", more closings, Tagalog and Vietnamese relation words
    • CRLF text is read as if it used plain line endings
  • Review. Four rounds of adversarial review ran on the drafts, and every claim was re-run before it counted. They confirmed 67 problems, including three privacy regressions that earlier review fixes had introduced. All were fixed and turned into tests: 586 pass.

  • CI. New gate: v9 regex+known must stay at or above 90% (it scores 91.7%).

Before/after on every set
The frozen 3aaa537 code was compared with this branch on v0 to v9:

  • v9 recall with the clinic and staff lists: 93.2% → 98.5%. Precision went from 85.2% to 85.8%.
  • No identifier was lost on v0 to v8, and v2, v4, v5, v7 and v8 each gained 1 to 5.
  • New false positives:
    • 2 card last-fours on v1 (v1 does not label them; v8 and v9 do)
    • 3 staff first names on a greeting or closing line with no staff cue (v5, v6)
    • 1 lab case number (v7)
    • 2 v9 timestamps that its authors labelled inconsistently

Known label calls left as annotated: one bare year marked as a date, and 3 bank routing numbers marked as accounts (v6 left them unlabelled, and the code follows v6). Both are documented in benchmark/README.md.

🤖 Generated with Claude Code

Sergei-gorlovetsky and others added 2 commits November 19, 2026 07:59
Written by four blind authors from one brief (Denver clinic with a Calgary
office: system dumps, long legal and clinical documents, names that look
like words, unusual identifier formats). Two blind annotators per 100
messages and an adjudicator kept 47 label corrections. Scored once against
3aaa537 (0.1.0a11) before any change: 93.4% recall with the local model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…icknames

A structured-text reader finds values by field name in JSON, key: value
lines, simple XML, CSV columns under a header row and HL7 v2 PID, NK1,
GT1 and IN1 segments; each value must look like what its field says.
New rules: court and case numbers (Colorado, Polish, Israeli, "Case
no."), Rx numbers, Colorado licences, cedulas, plates, usernames, Alberta
health numbers and Medicare, digits read one at a time after a record
word, card last four in five languages and masked first4/last4 cards,
donor numbers with words between, phones after Hebrew prefixes, trunk
"(0)" and "00" prefixes, Polish landlines, ages over 89 of "the patient",
ISO timestamps, nicknames, greeting-line addressees, "<Name> and I",
more closings, Tagalog and Vietnamese relation words. CRLF text is
matched as LF.

Four rounds of adversarial review confirmed 67 problems in drafts of
these rules (including three privacy regressions from earlier review
fixes); all fixed with tests. v9 now: 98.5% recall with clinic and staff
lists (first score 93.2%); no identifier lost on v0 to v8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
@Sergei-gorlovetsky
Sergei-gorlovetsky merged commit 063331d into main Nov 19, 2026
4 checks passed
@Sergei-gorlovetsky
Sergei-gorlovetsky deleted the blind-v9 branch November 19, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant