Skip to content

chore: reconcile skills + scenario-traces + casus leak-guard#785

Open
humanintheloop25 wants to merge 11 commits into
mainfrom
chore/reconcile-skills
Open

chore: reconcile skills + scenario-traces + casus leak-guard#785
humanintheloop25 wants to merge 11 commits into
mainfrom
chore/reconcile-skills

Conversation

@humanintheloop25

Copy link
Copy Markdown
Collaborator

Wat

Voegt de uiteengelopen user-level en engine-repo kopieën van de regelrecht-skills samen tot één canonieke set, en borgt dat er geen trajectinformatie naar deze publieke repo lekt.

Waarom

De skills bestonden in twee kopieën (~/.claude/skills/ en .claude/skills/ hier) die uit elkaar waren gelopen. Per-bestand gereconcilieerd:

  • stelselanalyse / dossier / audit-products: de engine-versies waren inhoudelijk vooruit (Step 0 drift-check, source-refs integriteitsscan, striktere private-repo-guard, "Bewijs-poort"). De scenario-traces-kruisverwijzingen die alléén user-level bestonden zijn erin geport.
  • Onderweg bleek elk user-level bestand een stray </content>-artefact te bevatten; de engine-versies zijn schoon.

Wijzigingen

  • regelrecht-scenario-traces als nieuwe, dossier-agnostische skill (9 bestanden).
  • ✏️ scenario-traces-verwijzingen geport in stelselanalyse (SKILL.md, engine-tests.md), dossier (SKILL.md, routing.md), audit-products (testcase-scenarios.md).
  • 🔒 script/check-skills-no-casus.sh + pre-commit hook skills-no-casus: blokkeert casus-specifieke inhoud (HHNK / Noorderkwartier / WS[0-9]{4} / corpus-repo-namen) in .claude/skills/. Fictieve test-BSN's (9999-reeks) bewust niet gevangen.

Verificatie

  • script/check-skills-no-casus.sh → casus-vrij ✅

🤖 Generated with Claude Code

humanintheloop25 and others added 11 commits May 27, 2026 12:00
Voegt drie samenhangende Claude-skills toe onder .claude/skills/:

- regelrecht-dossier: front-door router die een binnenkomende casus triageert
  en routeert (feitelijke defecten -> desk, oordeels-/praktijkvragen -> workshop),
  met routing.md (canonieke flow) en zakkaart.md (onboarding-A4 voor analisten).
- regelrecht-stelselanalyse: desk-review & corpus-completion-cyclus met 4-weg-
  classificatie, multi-agent review, verificatie, /loop- en /schedule-ondersteuning
  en een fail-closed private-only push-guard.
- regelrecht-audit-products: producten voor live expert-validatie (scope-analyse,
  per-artikel audit-doc, facilitatiekit, testcases, verslagen).

Dossier-agnostisch: geen casus-specifieke referenties.
Adds the law-version-drift-check skill alongside the existing desk-layer skills.
It runs as Step 0 of every regelrecht-stelselanalyse micro-cycle and detects drift
between a YAML's text blocks (plus competent_authority names and url anchors) and
the binding geldende wettekst at the YAML's valid_from.

Core principle: the geldende wettekst is leading and is mirrored verbatim,
including errata; one allowed normalization (whitespace within a single
paragraph). Two diff axes per article: structural first (lid- and onderdeel-
counts), then textual at word level. Verdicts: CLEAN / DRIFT-tekst /
DRIFT-structureel / NIET-VERIFIEERBAAR. Strict by design — no bypass; partial
failure produces scope-restrictie (frozen articles), not a green report.

Calibration ijkpunten live in the consuming corpus, not in the skill, keeping
the skill dossier-agnostisch. Findings flow into resolutie-tracker as 4-weg
klasse 1 (modellering-fout).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- assert-private-repo.sh: expliciete host-check vóór slug-extractie;
  niet-github.com remotes (GitLab, GHE, Bitbucket) blokkeren fail-closed.
  Voorheen strippeerde de sed-keten elke host blind en queryde github.com,
  waardoor een toevallig matchende private repo daar GUARD OK gaf voor
  een push naar een ander forge.
- assert-private-repo.sh: sleep 1 tussen retry-pogingen, zodat de retry
  zin heeft bij transiënte gh-haperingen.
- scheduled-routine.md: heading 'geïnstalleerd' → 'lokaal instellen' +
  expliciete noot dat de hook machine-lokaal is en niet met de repo
  meereist.
- routing.md: ASCII-diagram opgeschoond (hangende rand opgeruimd,
  terugkoppel-pijl recht).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ram placeholders, no dangling bindings)

- law-generate: hard rule that every cross-law input needs a real source:
  block, never just a description; add target output first if missing;
  multiple source: bindings per article are supported (schema v0.5.2);
  name-mismatch rule for source.output vs local name.
- regelrecht-stelselanalyse: mandatory non-skippable source-refs integrity
  scan algorithm; dangling + plain-param are always modelling errors, never
  engine limitations; add reusable references/cross-law-integriteit.py.
- law-reverse-validate: treat a source binding to a non-existent target
  output as a hallucination (dangling-binding check).
- engine-limitaties template: a claimed engine limitation may only be
  recorded after a reproducible failing engine run; untested assumptions are
  open questions, not limitations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r parameters)

The most common hidden cross-law defect: a source: block placed under
parameters: instead of input:. The engine's Parameter struct has no source
field, so the binding is silently dropped at parse time and degrades to a
plain caller-supplied parameter; scenarios that inject the value directly
mask it.

- cross-law-integriteit.py: rewritten YAML-based (robust vs regex). Adds
  MISPLACED detection (source under parameters:), keeps DANGLING (target
  output missing) and PLAIN-PARAM (conceptueel/tijdelijk marker without
  source). Drops the over-eager 'forward naar' marker (false-positives on
  legitimate leaf parameters that feed a binding mapping). Exit!=0 as CI gate.
- law-generate: hard 'Section placement' rule — a source: MUST live under
  input:, never parameters:; leaf inputs that feed source.parameters stay
  under parameters:; verify with a BDD scenario that loads the target law
  and sets leaf inputs so the dependent output flips.
- regelrecht-stelselanalyse: source-refs integrity scan now reports
  clean/misplaced/dangling/plain-param; MISPLACED listed as the most common
  hidden form; all three are always modelling errors, never engine limits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s Step 0

- Remove trailing </content> (40 files) and </invoke> (audit-products SKILL)
  leaked from the authoring tool-call format. All files keep their final newline.
- Integrate law-version-drift-check into regelrecht-stelselanalyse: new Step 0
  section + Werkstroom item 0, a Zustersskills cross-reference block, and a
  Step 0 gate in references/cycle-workflow.md (section + volgorde-diagram).
  Closes the orphan where the drift-check called itself a hard prerequisite
  while the owning skill never referenced it.
…truth, ASCII

- Add scripts/test-assert-private-repo.sh: stubs git/gh and asserts the guard
  invariants (gitlab/enterprise/bitbucket/public/internal/no-remote/unreachable
  → block, github private → pass). 11 cases, all green. Referenced from SKILL.md.
- Point the two 4-weg-classificatie copies that lacked it (dossier SKILL,
  zakkaart) at classification.md as the canonical definition source, containing
  definition-drift while keeping each skill standalone-readable.
- Redraw the routing.md flow diagram: the return-loop now forms one continuous
  vertical (┐│┘ aligned), annotations moved to an omlaag/omhoog legend below.
- Extend cross-law-integriteit.py with IoC checks: IMPL-DANGLING (implements
  pointing at an undeclared open_term) and IMPL-NO-DATE (implementing regulation
  without valid_from, which RFC-003's temporal filter then matches for every date).
- Add a `cross-law-integrity` CI job running the script over corpus/regulation as
  a fail-closed gate (gap 1); add the script path to the `ci` paths filter.
- Clarify coverage_score is a COMPLETENESS metric, not correctness (doc-comment
  on LawEntry.coverage_score) (gap 4).
- Add references/check-keten.md: full inventory of zuiverheid/cross-law checks,
  each with its oracle (waartegen) and whether it is a CI-gate or methodological.
  Documents why text-fidelity (gap 3) needs a separate golden-text mechanism.
… van de wet

Nieuwe casus-agnostische skill: auditeert of een machine_readable-model trouw is aan de LETTER
van de wettekst en scheidt wettekst (leidend) strikt van toelichting (uitleg). Detecteert vier
afwijkingsklassen (toelichting-bleed, ontbrekend verbatim-bestanddeel, verkeerde legal_basis-
verankering, verbatim-drift), checkt positief-lid vs uitsluitings-lid + de "in afwijking van"-
precedence in de chapeau, en classificeert elke bevinding als modelfout / wettekst-gevolg /
letter-vs-toelichting-vraag (jurist beslist wat leidend is + herzieningssignaal). Fideliteit
vóór outcome-toetsing, om outcome-bias te vermijden. Verwijst naar law-version-drift-check,
regelrecht-stelselanalyse en law-generate.
…-ontologie

Beslist en audit het referentietype van een waarde (cross-law / intra-law source-binding /
open-norm leaf / extern-feit leaf) en vangt begrip-conflatie tussen wetten. Kern: het type volgt
uit WIE het afleidt; een binding is sound alleen als producer en consumer hetzelfde BEGRIP
aanduiden — het woord is niet het begrip. Detecteert false friends (één term, verschillende
begrippen: register-gebaseerd vs feitelijke "naar de omstandigheden"-weging), orphan plain-params,
negatie/complement-re-entry (¬X als losse leaf naast X) en duplicatie (één begrip als meerdere
onafhankelijke leafs). Levert een law-level glossary + stelsel-level concept-map met sound
same-concept-edges en gemarkeerde lexical-match-but-concept-mismatch-edges. Cardinale regel: een
register mag een feitelijke open norm voeden (indicator), nooit stilzwijgend vervangen; symmetrie
over een stelsel. Orthogonaal aan law-letter-fidelity-audit; bovenop regelrecht-stelselanalyse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Voegt de uiteengelopen user-level en engine-repo skill-kopieën samen tot één
canonieke set. Engine-versies waren inhoudelijk vooruit (Step 0 drift-check,
source-refs integriteitsscan, striktere private-repo-guard); de scenario-traces-
kruisverwijzingen die alleen user-level bestonden zijn erin geport.

- regelrecht-scenario-traces toegevoegd als nieuwe (dossier-agnostische) skill
- script/check-skills-no-casus.sh + pre-commit hook 'skills-no-casus' die
  casus-specifieke inhoud (HHNK/waterschap-codes/corpus-repo-namen) blokkeert
  zodat trajectinformatie niet in de publieke skills lekt

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-law integrity gate + leak-guard — review

Security & correctness

🟠 Significant — check-skills-no-casus.sh: fixed /tmp path is a race condition

if grep -rEnI "$PATTERN" "$SKILLS" > /tmp/skills_casus_leak 2>/dev/null; then
    sed "s#${ROOT}/##" /tmp/skills_casus_leak >&2
    ...
    rm -f /tmp/skills_casus_leak

/tmp/skills_casus_leak is a shared path. Two concurrent jobs on the same GitHub-hosted runner pool can race: job B's grep overwrites job A's file mid-flight, causing either a garbled error message or a false-pass (rm between write and read). Use $(mktemp):

LEAKFILE=$(mktemp)
if grep -rEnI "$PATTERN" "$SKILLS" > "$LEAKFILE" 2>/dev/null; then
    ...
    rm -f "$LEAKFILE"
    exit 1
fi
rm -f "$LEAKFILE"

🟠 Significant — PLAIN_MARKERS in cross-law-integriteit.py contradicts law-generate/SKILL.md

law-generate/SKILL.md (new section, this PR) lists three forbidden plain-param markers:

"conceptueel", "forward naar", or "tijdelijk als directe parameter"

cross-law-integriteit.py explicitly excludes the middle one:

# Note: do NOT include generic words like "forward naar" — those legitimately
# describe leaf parameters that FEED a binding's parameters-mapping
PLAIN_MARKERS = ('conceptueel', 'tijdelijk als directe parameter')

A description containing "forward naar [andere regeling] output Y" without a source: block silently passes the CI gate but violates the skill's stated rule. The Python comment's rationale is reasonable — "forward naar" can appear in legitimate parameter descriptions — but the skill text doesn't reflect this distinction. One of the two must be updated: either remove "forward naar" from the SKILL.md's forbidden list, or add a note that it's deliberately excluded from the automated gate with the reason.


Minor

🟡 cross-law-integriteit.py — unclosed file handle and implicit encoding

doc = yaml.safe_load(open(path))

open() is never closed; on a large corpus this can exhaust file descriptors under concurrent processes. The broad except Exception: continue silently swallows encoding errors too — a law file with a platform-encoding mismatch is silently skipped rather than flagged. Use:

with open(path, encoding='utf-8') as f:
    doc = yaml.safe_load(f)

🟡 CI: pip install pyyaml unpinned

An uncontrolled pyyaml upgrade (including a future major release) could silently change YAML parsing behaviour or break the script. Pin to a tested range, e.g. pip install "pyyaml>=6.0,<7", or use a requirements-ci.txt file.


🟡 check-skills-no-casus.sh: comment says "Woordgrenzen" but pattern has no \b

# Casus-markers (uitbreidbaar). Woordgrenzen waar een losse term anders zou matchen.
PATTERN='HHNK|Hollands Noorderkwartier|Noorderkwartier|regelrecht-corpus-(HHNK|BES)|WS[0-9]{4}'

The comment implies word-boundary anchors, but the grep pattern has none. WS[0-9]{4} would match inside longer tokens (e.g. AWS1234). Either add \b anchors around the relevant terms or drop the misleading comment. Low practical risk in this directory, but the comment is incorrect.


Not flagged

The changes.outputs.ci guard on the new cross-law-integrity job is fine — corpus/regulation/** is in the ci filter, so corpus-only PRs will trigger the gate.

The doc comment added to coverage_score in models.rs is clear and accurate.

@github-actions

Copy link
Copy Markdown

Preview Deployment — harvester-admin

Your changes have been deployed to a preview environment:

URL: https://harvester-admin.pr785.rig.prd1.gn2.quattro.rijksapps.nl

This deployment will be automatically cleaned up when the PR is closed.

@github-actions

Copy link
Copy Markdown

Preview Deployment — harvester-worker

Your changes have been deployed to a preview environment:

URL: https://harvester-worker-pr785-regel-k4c.rig.prd1.gn2.quattro.rijksapps.nl

This deployment will be automatically cleaned up when the PR is closed.

@github-actions

Copy link
Copy Markdown

Preview Deployment — enrichworker

Your changes have been deployed to a preview environment:

URL: https://enrichworker-pr785-regel-k4c.rig.prd1.gn2.quattro.rijksapps.nl

This deployment will be automatically cleaned up when the PR is closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant