feat(textguard): per-lid text-fidelity gate (golden-text)#727
feat(textguard): per-lid text-fidelity gate (golden-text)#727humanintheloop25 wants to merge 1 commit into
Conversation
Adds a two-layer text-fidelity guard for the corpus: - script/textguard.py — `bless` captures each article's normalised per-lid `text:` chunks into committed fixtures (sha256 + text + provenance); `check` recomputes the hashes and compares them, exiting non-zero on any drift. A chunk is one blank-line-delimited paragraph (one lid/onderdeel/aanhef); normalisation is the single whitespace rule from the drift-check reference. - textguard/ — seeded fixtures for the whole corpus (21 laws, 1751 chunks), provenance verified_against: pending until the live drift-check upgrades it. - CI job `text-fidelity` runs `textguard check` as a hermetic fail-closed gate; Justfile recipes `textguard-check` / `textguard-bless`; yamllint ignores the generated fixtures. The gate guards "YAML text == last verified text" (catches silent edits); fidelity to the live geldende wettekst stays the methodological drift-check. See textguard/README.md.
There was a problem hiding this comment.
Review
Summary
This PR introduces a two-layer text-fidelity guard: a Python script (script/textguard.py) that can bless (capture) and check (verify) per-chunk SHA-256 hashes of article text in the corpus, plus 21 seeded fixture files and a CI job. The design rationale is sound — making silent YAML edits immediately visible in CI while deferring live-web verification to the periodic drift-check skill.
🟠 Significant
bless silently discards already-verified provenance on re-bless (script/textguard.py, cmd_bless, the "verified_against": "pending" line)
Every call to just textguard-bless unconditionally writes verified_against: "pending" for every chunk in every law — including chunks whose text is unchanged and that a previous drift-check had already upgraded to a real wetten.overheid.nl date. The PR description says the drift-check skill upgrades verified_against from pending to a date, but a subsequent re-bless (after any text change to that file) resets the entire fixture, discarding that provenance for unchanged chunks.
Concrete scenario:
- Drift-check runs →
verified_against: "wetten.overheid.nl/BWBR.../2025-01-01"for all chunks of a law. - Article 3 text is corrected.
- Developer runs
just textguard-bless. - All chunks — including articles 1, 2, 4, 5 — are reset to
verified_against: "pending".
The fix: when an existing fixture is present, read it and build a (article_number, chunk_index, sha256) → verified_against lookup, then carry that value forward for any chunk whose hash is unchanged. Only new or changed chunks get "pending".
🟡 Minor
pyyaml installed without version pin in CI (.github/workflows/ci.yml, pip install pyyaml step)
pip install pyyaml resolves to whatever is latest at run time. Use pip install pyyaml==6.0.2 or pin via a requirements.txt.
Bare open() calls — file handles not closed (script/textguard.py, the yaml.safe_load(open(path)) calls in cmd_bless and cmd_check)
Three calls rely on CPython's reference-counting GC to close handles. Replace with with open(path, encoding="utf-8") as fh: — the explicit encoding also guards against non-UTF-8 system locales misreading legal text.
n_chunks is undercounted when chunk-count mismatches occur (script/textguard.py, cmd_check)
When len(chs) != len(expected) the code appends a finding and continues before the zip loop that increments n_chunks. The "N chunks vergeleken" summary line then undercounts. This doesn't affect gate correctness but makes the diagnostic misleading.
textguard-check is not wired into just check (Justfile)
just check (the standard local quality gate per CLAUDE.md) does not include textguard-check. Developers get no local signal before push. Easy fix: add textguard-check to the check recipe's dependency list.
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main()) |
There was a problem hiding this comment.
verified_against is always reset to "pending" on re-bless. When an existing fixture already has chunks with a real verification date (set by the drift-check skill), calling just textguard-bless after any text change silently discards that provenance for all unchanged chunks too.
Fix: before writing, read the existing fixture and build a (article_num, chunk_index, sha256) → verified_against lookup; preserve the existing verified_against for any chunk whose hash matches.
Wat
Een twee-lagen tekstgetrouwheid-guard voor de corpus, per lid/onderdeel.
bless(methodologisch, periodiek) capture't per artikel de genormaliseerdetext:-chunks als gecommitte fixture (sha256 + tekst + herkomst).check(deterministisch, CI) herberekent de hashes uit de corpus en vergelijkt; mismatch → faal. Hermetisch (zuivere functie van repo-inhoud).Een chunk = één lege-regel-gescheiden alinea van een artikel-
text:— in deze corpus precies één lid/onderdeel/aanhef. Normalisatie = de enige whitespace-regel uitlaw-version-drift-check/reference.md§2.Waarom zo
De vergelijking is triviaal deterministisch; het niet-deterministische zit in het orakel. De geldende wettekst leeft op wetten.overheid.nl — remote, mutabel, te interpreteren → geen CI-orakel. Daarom splitsen:
law-version-drift-check-skill; die zetverified_againstvanpendingnaar een wetten.overheid.nl-datum.Eerlijke grens: de gate bewaakt "YAML ≡ laatst geverifieerde tekst" (vangt stille edits direct), niet "YAML ≡ huidige live wet" (dat blijft de periodieke drift-check). Approval-testing + per-chunk provenance, verwant aan RFC-013's
regulation_hashmaar per lid.Inhoud
script/textguard.py(bless/check, gedeelde normalisatie)textguard/— seeded fixtures: 21 wetten, 1751 chunks,verified_against: pendingtext-fidelity(fail-closed gate) + Justfiletextguard-check/textguard-blesstextguard/README.md— workflow + de eerlijke grensVerifieerd
checkop de seeded corpus: 1751 chunks, 0 afwijkingen, exit 0art 2.75 [lid 1]), exit 1; revert → groenLet op
Seeding is een baseline-bless (
pending): de gate is meteen actief, maar de tekst is nog niet tegen wetten.overheid.nl gekruist. Dat upgraden gebeurt wanneer de drift-check per wet draait.🤖 Generated with Claude Code