Repository navigation
Release 0.14.1 - #86
Merged
Merged
Conversation
…mats of the Python redactor The Rust PII scanner reports a credit_card match only when its digits pass the Luhn checksum. Its phone pattern is the one of the Python redactor: North American numbers and Italian mobile and landline numbers, with or without +39 or 0039. The regex crate has no lookarounds, so each pattern carries an accept function that checks what the regex cannot: the checksum, and the characters before and after a phone number. redact() masks and counts only the accepted matches, and scan() counts them. An ISBN, a build number and a 16-digit number that fails the Luhn check are no longer reported.
admina.domains.data_sovereignty.names.is_name_like() holds for a span with
no digit and a word with a capital initial. The PII redactor, the
spacy-regex engine and the Presidio engine skip the PERSON entities of the
NER model for which it does not hold: the English spaCy model labels
lowercase phrases of other languages as PERSON ("il codice articolo",
"la pratica n. 2026/000457"). A name written all in lowercase is not
masked either.
The event-loop tests run a 1 s job and accept a stall shorter than the job. A loop that runs the job, or waits for it without yielding, stalls for at least the whole job, so the tests fail for it whatever the load; with the loop free, a stall reaches the bound only if the scheduler holds the loop for a second. Checked with an executor that waits for the worker thread on the loop: the three tests fail with stalls of 1.0 to 8.0 s.
PII false positives: Python 4/24, Rust 0/24, Presidio 8/24; Presidio type-level recall 0.586. MODEL_CARD gives the numbers, the Luhn check and the phone formats of both engines, the IBAN checks of each engine, and the false positives that remain: ORG/GPE on non-English sentences, PERSON on German sentences, and PERSON on "Friday" and PHONE on an Italian reference number with Presidio.
…ords
is_name_like() rejects a span with a digit, a character of an e-mail
address or a path (@ / \ _ # : =), or a lowercase word that is a stop word
of English, Italian, German, French, Spanish or Portuguese (the lists of
spaCy, without the words that are also first names: will, may, sara, mia,
ali). Capitalised words are not checked. Names written in lowercase, which
the English spaCy model labels as PERSON ("john smith", "giovanni
esposito"), are masked again; phrases such as "il codice articolo", "ci
vediamo domani" or "das Wetter" are not.
…RSON PII false positives: Python 3/24, Rust 0/24, Presidio 9/24; Presidio type-level recall 0.586. MODEL_CARD and the [0.14.1] section of CHANGELOG describe the rule and the remaining false positives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Patch release; the
[0.14.1]section ofCHANGELOG.mdlists the changes.PERSONentity is masked only when it has no digit, no e-mail/path character and no lowercase stop word (English, Italian, German, French, Spanish, Portuguese); names written in lowercase are masked (PII redactor,spacy-regex, Presidio).Local checks: pytest 4584 passed; cargo fmt, clippy and 30 tests; redteam gate;
twine check --strict.