Skip to content

Blind v12: travel IDs, pedigree ages, spelled names, foreign addresses - #29

Merged
Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v12
Nov 21, 2026
Merged

Sergei-gorlovetsky merged 2 commits into
mainfrom
blind-v12

Conversation

@Sergei-gorlovetsky

Copy link
Copy Markdown
Contributor

Summary

  • New blind set v12 (400 hand-written messages, 1,386 labelled identifiers + 135 staff names, 70 audit corrections), scored once against frozen 650fe26 (0.1.0a13): 93.8% recall with all detectors (up from v11's 88.4%: the v10 and v11 fixes carried over).
  • Fixes for the gaps it showed: travel booking codes, e-tickets, passports and foreign national IDs; bar, notary and court numbers; ages over 89 in family-history pedigrees; names spelled with Spanish or French letter names, Cyrillic and Japanese names; Philippine, UK, Indian and Mexican addresses and postcodes; schools as employers in four languages.
  • Removed during review because each could leave patient data visible: a pass for clinic-domain URLs, a phone exclusion near "NHS", and two date exclusions ("the 4th floor", lab ranges like "3.5-12"). A pedigree age rule that was quadratic on long blank runs is now linear.

Results

  • v12 now, all detectors with clinic details and staff list: recall 93.7% -> 98.9%, precision 88.1% -> 89.0%.
  • All twelve blind sets: 96.4% to 99.3% (results files regenerated).
  • Regression check vs frozen 0.1.0a13 on v0 to v12, four configurations: no identifier it caught is missed; false positives the same or lower on every set.
  • Speed: 2.9 ms per message on the 4,780 blind messages (2.7 ms before).
  • New CI gate: v12 regex+known >= 0.90 (now 92.2%). 1,792 tests.

Notes

  • Not a release: a PyPI publish needs a separate go.

🤖 Generated with Claude Code

Sergei-gorlovetsky and others added 2 commits November 20, 2026 20:55
Fictional Seattle clinic with a Regina office. Four blind authors wrote
100 messages each from a brief aimed at family history and genetics
pedigrees, third-party reproduction paperwork, travel and cross-border
care, speech-recognition errors, form and app exports, and minors with
guardians. Two blind annotators per 100 messages and an adjudicator kept
70 corrections. Scored once against 650fe26 (0.1.0a13) before any change:
93.8% recall with all detectors (85.2% with regex + known entities).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
…gn addresses

IDs: travel booking and record-locator codes after a booking cue (five
languages and CJK), e-ticket numbers, passports after a passport cue,
foreign national IDs (Aadhaar, PhilSys, NIN, Belgian national number),
lawyers' bar and notary numbers, court cause numbers, insurance member
IDs in letter-digit groups, NHS and "My Number" values.

Ages and dates: ages over 89 after death and life words and after a
relative with an optional name, in English, Polish, Russian and
Ukrainian; Russian and Ukrainian day + month dates.

Names: Spanish and French letter-name spelling after a cue, Cyrillic
names and vocatives after cues, Japanese names in kanji after a name cue,
first names in time-stamped transcripts, pedigree markers.

Addresses: Philippine lot-and-block and barangay addresses with an
address context, UK street types and postcodes, Indian addresses with a
PIN, Mexican floor and door, reserve names after a cue, local postcode
cues, the clinic's own city when a patient lives there; schools and
universities as EMPLOYER in Spanish, French, Polish and Hebrew.

Removed during review because each could leave patient data visible: a
pass for URLs on the clinic's own domain, a phone exclusion near "NHS",
"the 4th floor" and lab-range ("3.5-12") date exclusions. A pedigree age
rule that ran in quadratic time on long blank runs is now linear.

v12 with clinic details and staff list: recall 93.7% -> 98.9%,
precision 88.1% -> 89.0%. No identifier the frozen 0.1.0a13 code caught
is missed on v0 to v12 in four configurations. Results files
regenerated, docs and a v12 CI gate added (1,792 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sergei Gorlovetsky <136336554+Sergei-gorlovetsky@users.noreply.github.com>
@Sergei-gorlovetsky
Sergei-gorlovetsky merged commit a0f8236 into main Nov 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant