Skip to content

feat(textguard): per-lid text-fidelity gate (golden-text)#727

Open
humanintheloop25 wants to merge 1 commit into
mainfrom
feat/textguard
Open

feat(textguard): per-lid text-fidelity gate (golden-text)#727
humanintheloop25 wants to merge 1 commit into
mainfrom
feat/textguard

Conversation

@humanintheloop25

Copy link
Copy Markdown
Collaborator

Wat

Een twee-lagen tekstgetrouwheid-guard voor de corpus, per lid/onderdeel.

  • bless (methodologisch, periodiek) capture't per artikel de genormaliseerde text:-chunks als gecommitte fixture (sha256 + tekst + herkomst).
  • check (deterministisch, CI) herberekent de hashes uit de corpus en vergelijkt; mismatch → faal. Hermetisch (zuivere functie van repo-inhoud).

Een chunk = één lege-regel-gescheiden alinea van een artikel-text: — in deze corpus precies één lid/onderdeel/aanhef. Normalisatie = de enige whitespace-regel uit law-version-drift-check/reference.md §2.

Waarom zo

De vergelijking is triviaal deterministisch; het niet-deterministische zit in het orakel. De geldende wettekst leeft op wetten.overheid.nl — remote, mutabel, te interpreteren → geen CI-orakel. Daarom splitsen:

  • Laag 1 (bless): de live tekst ophalen + verifiëren doet de law-version-drift-check-skill; die zet verified_against van pending naar een wetten.overheid.nl-datum.
  • Laag 2 (check): CI vergelijkt tegen de gecommitte fixture.

Eerlijke grens: de gate bewaakt "YAML ≡ laatst geverifieerde tekst" (vangt stille edits direct), niet "YAML ≡ huidige live wet" (dat blijft de periodieke drift-check). Approval-testing + per-chunk provenance, verwant aan RFC-013's regulation_hash maar per lid.

Inhoud

  • script/textguard.py (bless/check, gedeelde normalisatie)
  • textguard/ — seeded fixtures: 21 wetten, 1751 chunks, verified_against: pending
  • CI-job text-fidelity (fail-closed gate) + Justfile textguard-check/textguard-bless
  • yamllint negeert de gegenereerde fixtures (verbatim wettekst > 125 tekens)
  • textguard/README.md — workflow + de eerlijke grens

Verifieerd

  • check op de seeded corpus: 1751 chunks, 0 afwijkingen, exit 0
  • Negatieve test: één woord wijzigen → exact gedetecteerd (art 2.75 [lid 1]), exit 1; revert → groen

Let op

Seeding is een baseline-bless (pending): de gate is meteen actief, maar de tekst is nog niet tegen wetten.overheid.nl gekruist. Dat upgraden gebeurt wanneer de drift-check per wet draait.

🤖 Generated with Claude Code

Adds a two-layer text-fidelity guard for the corpus:

- script/textguard.py — `bless` captures each article's normalised per-lid
  `text:` chunks into committed fixtures (sha256 + text + provenance); `check`
  recomputes the hashes and compares them, exiting non-zero on any drift.
  A chunk is one blank-line-delimited paragraph (one lid/onderdeel/aanhef);
  normalisation is the single whitespace rule from the drift-check reference.
- textguard/ — seeded fixtures for the whole corpus (21 laws, 1751 chunks),
  provenance verified_against: pending until the live drift-check upgrades it.
- CI job `text-fidelity` runs `textguard check` as a hermetic fail-closed gate;
  Justfile recipes `textguard-check` / `textguard-bless`; yamllint ignores the
  generated fixtures.

The gate guards "YAML text == last verified text" (catches silent edits);
fidelity to the live geldende wettekst stays the methodological drift-check.
See textguard/README.md.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Summary

This PR introduces a two-layer text-fidelity guard: a Python script (script/textguard.py) that can bless (capture) and check (verify) per-chunk SHA-256 hashes of article text in the corpus, plus 21 seeded fixture files and a CI job. The design rationale is sound — making silent YAML edits immediately visible in CI while deferring live-web verification to the periodic drift-check skill.


🟠 Significant

bless silently discards already-verified provenance on re-bless (script/textguard.py, cmd_bless, the "verified_against": "pending" line)

Every call to just textguard-bless unconditionally writes verified_against: "pending" for every chunk in every law — including chunks whose text is unchanged and that a previous drift-check had already upgraded to a real wetten.overheid.nl date. The PR description says the drift-check skill upgrades verified_against from pending to a date, but a subsequent re-bless (after any text change to that file) resets the entire fixture, discarding that provenance for unchanged chunks.

Concrete scenario:

  1. Drift-check runs → verified_against: "wetten.overheid.nl/BWBR.../2025-01-01" for all chunks of a law.
  2. Article 3 text is corrected.
  3. Developer runs just textguard-bless.
  4. All chunks — including articles 1, 2, 4, 5 — are reset to verified_against: "pending".

The fix: when an existing fixture is present, read it and build a (article_number, chunk_index, sha256) → verified_against lookup, then carry that value forward for any chunk whose hash is unchanged. Only new or changed chunks get "pending".


🟡 Minor

pyyaml installed without version pin in CI (.github/workflows/ci.yml, pip install pyyaml step)

pip install pyyaml resolves to whatever is latest at run time. Use pip install pyyaml==6.0.2 or pin via a requirements.txt.


Bare open() calls — file handles not closed (script/textguard.py, the yaml.safe_load(open(path)) calls in cmd_bless and cmd_check)

Three calls rely on CPython's reference-counting GC to close handles. Replace with with open(path, encoding="utf-8") as fh: — the explicit encoding also guards against non-UTF-8 system locales misreading legal text.


n_chunks is undercounted when chunk-count mismatches occur (script/textguard.py, cmd_check)

When len(chs) != len(expected) the code appends a finding and continues before the zip loop that increments n_chunks. The "N chunks vergeleken" summary line then undercounts. This doesn't affect gate correctness but makes the diagnostic misleading.


textguard-check is not wired into just check (Justfile)

just check (the standard local quality gate per CLAUDE.md) does not include textguard-check. Developers get no local signal before push. Easy fix: add textguard-check to the check recipe's dependency list.

Comment thread script/textguard.py


if __name__ == "__main__":
sys.exit(main())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verified_against is always reset to "pending" on re-bless. When an existing fixture already has chunks with a real verification date (set by the drift-check skill), calling just textguard-bless after any text change silently discards that provenance for all unchanged chunks too.

Fix: before writing, read the existing fixture and build a (article_num, chunk_index, sha256) → verified_against lookup; preserve the existing verified_against for any chunk whose hash matches.

@humanintheloop25 humanintheloop25 requested a review from tdjager June 1, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant