Add conformance page reconciliation test and off-page vector sets by ojongerius · Pull Request #957 · agent-receipts/obsigna

ojongerius · 2026-07-02T22:12:47Z

What

Adds a CI-enforced reconciliation test that binds the published conformance page's matrix and prose figures to the authoritative count.py script, preventing stale numbers from reaching main. Introduces an on_page flag to VectorSet to distinguish reference fixtures (currently did:key resolution) that are counted for reproducibility but omitted from the published matrix.

Why

The conformance page claims its vector counts are "generated, not asserted," but nothing enforced that the hand-typed table cells and prose figures matched count.py's output. A change to a vector file could silently desync the two — worse than no number on a page cited to a standards body.

The did:key resolution vectors are reference fixtures that no SDK suite currently consumes. They should be counted (corpus reproducibility) but shown as an all-dashes row invites the wrong question. The on_page flag lets them stay in the count while being excluded from the published matrix.

Changes

cross-sdk-tests/conformance_page_test.go (new): Integration test that runs count.py --format json, parses the conformance page, and asserts:
- Every on-page vector set from count.py appears as a matrix row with the exact count
- No matrix row exists that count.py does not know about
- The two prose figures in the honesty callout (canonicalization total and MUST-reject case count) match count.py
- Skips gracefully if python3 is not available (local dev); CI runners have it
scripts/conformance_matrix/count.py:
- Added on_page: bool = True field to VectorSet
- Marked did:key resolution with on_page=False
- Updated _render_md() to skip off-page sets (paste-ready for the published matrix)
- Updated _render_json() to include onPage in output
scripts/conformance_matrix/test_count.py:
- Updated test name and assertion to reflect that markdown output includes only on-page sets
- Added test that off-page sets do not leak into the markdown output
- Added test that did:key resolution is marked off-page
scripts/conformance_matrix/AGENTS.md: Documented the reconciliation test and on_page flag
site/src/content/docs/conformance.mdx: Minor prose clarifications ("independent" → "separate codebases that share no code")
cross-sdk-tests/canonicalization_vectors.json: Added Unicode normalization test vector
ADR and daemon references: Updated cross-references from ADR-0008 to ADR-0015 Phase B checkpoint anchoring where applicable

Checklist

Tests pass for all changed components (new integration test + updated unit tests in test_count.py)
Linter passes
No real keys or secrets in the diff
Cross-language tests pass (vector format unchanged; new test is Go-only)
AGENTS.md updated (documented reconciliation test and on_page flag)

Security

This PR does not touch crypto, auth, or secrets handling

…DIT review Precision pass over the conformance page and the claims around it ahead of external standards-reviewer scrutiny. Conformance page + counts: - Add a Unicode-normalisation canon vector (string_no_unicode_normalization): NFD "e + U+0301" must canonicalise verbatim (65 CC 81), not fold to NFC U+00E9. Confirmed all three SDKs preserve it (no bug). \u escapes keep the fixture bytes ASCII so the source cannot itself be NFC-folded. Canon set 43 -> 44. - Wire the page counts so they cannot silently drift: new CI-run reconciliation test (cross-sdk-tests/conformance_page_test.go) execs count.py and asserts every matrix cell and the exact prose figures equal its output. count.py gains an on_page flag; the page copy now cites the reconciliation test. - Tighten "independent implementations" language: state the real property (three languages, no shared code) and its limit (catches serialisation drift, not spec-interpretation ambiguity; single author). No claim of independent authorship/interpretation remains. - did:key row was all dashes (no SDK implements resolution). Removed it from the matrix, kept the fixtures counted (on_page=False), noted out-of-scope in the honesty callout, and filed #956 for the real wiring. ADR hygiene: - ADR-0015 header was stale (Rotate()/anchor sink "not implemented", Phase B "deferred"/"unstarted"). Updated Status + implementation-status to match the shipped state (offline rotation, anchor-first writes, Phase B checkpoint anchoring end-to-end) and ADR-0019; index status updated to agree. - Pin checkpoint anchoring to ADR-0015 Phase B (the mechanism) instead of ADR-0008, in the daemon flag help, threat-model.md, and daemon emission comments. ADR-0008 retained for the truncation gap, the anchoring freeze, and the --against-anchor verifier check.

Follow-up to the IETF-audit hardening pass, fixing 8 review findings: - Page copy no longer claims counts "cannot silently drift" absolutely: the reconciliation test runs in cross-sdk-tests.yml, which triggers on the vector corpus / spec (not site/** or scripts/**), so the copy now states what is actually enforced (a corpus edit that outdates a number turns CI red). Test header comment records the same trigger-scope caveat. - Retract the "independent implementations" overclaim in the two docs the page links to (README.md, cross-sdk-tests/README.md); state the real property (separate, no-shared-code, single author; serialisation-drift detector, not spec-ambiguity detector). - ADR-0015: the dated 2026-05-12 snapshot's "Phase B ... deferred" line now forward-points to the update where Phase B checkpoint anchoring landed, so it no longer contradicts the header. - Reconciliation test: scope the matrix regex to the "## Results matrix" section and the prose regexes to the honesty callout, so an unrelated table or stray figure elsewhere on the page can't be misread; surface count.py's stderr on failure; load count.py output + page once and share across subtests (was exec'd/read twice). - Remove the unpinned "2-3 pinned receipt/chain vectors per spec version" prose figure, pointing to the (pinned) matrix instead. - Pin the verifier-side checkpoint reference to "ADR-0008 §3 out-of-band check over ADR-0015 Phase B checkpoints" (was bare "ADR-0008 follow-through").

Copilot

Pull request overview

Adds CI reconciliation between the published Conformance page and the authoritative vector-count generator (scripts/conformance_matrix/count.py), and introduces an “off-page” classification for reference-only vector sets so they remain reproducible/countable without appearing in the public matrix.

Changes:

Added an integration Go test that runs count.py --format json, parses site/src/content/docs/conformance.mdx, and asserts matrix + prose figures match the computed counts.
Extended count.py’s VectorSet with on_page and updated Markdown/JSON rendering + unit tests to omit off-page sets from the paste-ready Markdown output while still counting them in totals.
Updated the Conformance page and related docs/ADRs to clarify the “independence” claim, reconcile updated vector counts, and refresh ADR cross-references; added a new canonicalization vector.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`site/src/content/docs/conformance.mdx`	Updates conformance narrative, matrix count(s), and explains the reconciliation gate + omits off-page did:key row.
`scripts/conformance_matrix/count.py`	Adds `on_page` flag, omits off-page sets from `--format md`, and includes `onPage` in JSON.
`scripts/conformance_matrix/test_count.py`	Updates/extends tests for on-page-only markdown output and did:key off-page behavior.
`scripts/conformance_matrix/AGENTS.md`	Documents the reconciliation test and the `on_page` flag semantics.
`cross-sdk-tests/conformance_page_test.go`	New integration test reconciling `count.py` JSON vs conformance page matrix + prose figures.
`cross-sdk-tests/canonicalization_vectors.json`	Adds a Unicode normalization-related canonicalization vector (no NFC folding).
`README.md`	Clarifies “separate codebases” / what interop proves vs doesn’t.
`cross-sdk-tests/README.md`	Same clarification for the cross-sdk test corpus README.
`docs/threat-model.md`	Updates anchoring references from ADR-0008 framing toward ADR-0015 Phase B checkpoint anchoring.
`docs/adr/README.md`	Updates ADR-0015 status summary to reflect shipped phases.
`docs/adr/0015-key-rotation-byok-anchoring.md`	Updates ADR status and adds explicit Phase B checkpoint anchoring “landed” update details.
`daemon/internal/verifycli/anchor.go`	Comment updates to reference ADR-0015 Phase B + ADR-0008 §3 framing.
`daemon/internal/pipeline/build.go`	Comment updates for checkpoint anchoring terminology.
`daemon/internal/checkpoint/checkpoint.go`	Comment updates for ADR references around checkpoint types.
`daemon/internal/anchor/anchor.go`	Comment updates for checkpoint event-type ADR reference.
`daemon/daemon.go`	Comment updates for checkpoint anchoring ADR reference and wiring description.
`daemon/cmd/obsigna-daemon/main.go`	Flag help text updated to reference ADR-0015 Phase B checkpoint anchoring.

- Reword the md renderer's "paste-ready" claim in count.py and test_count.py: the published matrix splits its combined Consumers column into separate Go/Py/TS columns, so it's a reference table, not a drop-in paste. - Widen cross-sdk-tests.yml's path filter to include the conformance page and scripts/conformance_matrix/**, so a direct edit to either can no longer bypass the reconciliation gate. Update the test's header comment and the page copy to match.

Merges origin/main, which landed the on_page mechanism and CI reconciliation test (cross-sdk-tests/conformance_page_test.go) that issue #956's did:key row was meant to flip. Resolves conflicts by: - count.py: flip did:key resolution's on_page back to True (default) now that all three SDKs consume it. - test_count.py: replace test_did_key_is_off_page with test_did_key_is_on_page, asserting the flip. - conformance.mdx: re-add the did:key resolution row to the matrix with real Go/Py/TS checkmarks; adopt main's count-agnostic honesty callout (44 canonicalization / 7 MUST-reject, matching the reconciliation test's regexes) and drop the now-resolved "no SDK implements did:key resolution yet" caveat.

ojongerius added 2 commits July 2, 2026 21:10

ojongerius requested a review from Copilot July 2, 2026 22:25

Copilot started reviewing on behalf of ojongerius July 2, 2026 22:26 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread scripts/conformance_matrix/count.py

Comment thread scripts/conformance_matrix/test_count.py Outdated

Comment thread cross-sdk-tests/conformance_page_test.go Outdated

ojongerius force-pushed the claude/conformance-page-ietf-audit-uluf3h branch from 938c49d to c2e2f51 Compare July 2, 2026 22:34

ojongerius merged commit f3a702b into main Jul 2, 2026
30 checks passed

ojongerius deleted the claude/conformance-page-ietf-audit-uluf3h branch July 2, 2026 22:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add conformance page reconciliation test and off-page vector sets#957

Add conformance page reconciliation test and off-page vector sets#957
ojongerius merged 3 commits into
mainfrom
claude/conformance-page-ietf-audit-uluf3h

ojongerius commented Jul 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ojongerius commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Changes

Checklist

Security

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ojongerius commented Jul 2, 2026 •

edited

Loading