Repository navigation
Redteam scorecard counts and PII hard negatives; date of 0.14.0 - #85
Merged
Merged
Conversation
v0.14.0 was tagged on 2026-10-08.
…scorecard Each cell of admina redteam --format md gives the recall followed by the detected and expected counts: samples for injection and loop, PII types for the PII detector. A note under the table says that a class often has one to three samples, so the figures are indicative, and that the labels of the corpora are assigned in this repository.
Eight negative samples tagged hard_negative hold numbers and codes that are not personal data: an order number that fails the Luhn check, a version, an invoice number with a date, an ISBN, a build number, room and chapter numbers, and two Italian reference codes. SHA256SUMS and the baseline are regenerated; only the PII false-positive counts change (Python 8/24, Rust 3/24, Presidio 12/24). MODEL_CARD lists the measured false positives: CREDIT_CARD and PHONE from the Rust scanner, PERSON from spaCy and Presidio on the Italian sentences. The [Unreleased] section of CHANGELOG describes the scorecard counts and the hard negatives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #37.
admina redteam --format mdgives the recall followed by the detected and expected counts (89% (8/9)). A note under the table says the figures are indicative (a class often has one to three samples).hard_negativesamples (an order number that fails the Luhn check, a version, an invoice number with a date, an ISBN, a build number, room and chapter numbers, two Italian reference codes).SHA256SUMSand the baseline are regenerated; only the PII false-positive counts change: Python 6/16 → 8/24, Rust 0/16 → 3/24, Presidio 9/16 → 12/24. Measured false positives: RustCREDIT_CARDon the non-Luhn number andPHONEon the ISBN and the build number; spaCy and PresidioPERSONon the two Italian sentences. MODEL_CARD lists them.loop-en-009(redteam suite: non-blocking follow-ups from #29 review #37): thecounter_resetsample is labelledloopon purpose; MODEL_CARD §9 documents that the Python loop breaker (last-5 window) misses it and the Rust one (full window) catches it. No change.[0.14.0]dated 2026-10-08, the day of the tag.Baseline regenerated in the CI measurement modes (
nlp:en_core_web_sm@3.8.0,presidio:2.2.363/en+it) with admina-core built from this tree (0.14.0); the gate passed against the committed baseline before the corpus change and passes against the new one. Full suite: 4555 passed.