docs(eventing): Phase 2 — record the claims a docs review disproved, and how to check them

[mrsabath]

Phase / scope

Phase	Phase 2 — identity on the event path (DESIGN_PHASE2.md)
What this changes	That phase's design record only. No code, no manifests, no tests.
Phases it does not touch	Phase 0/1 (wire contract, KEDA) and Phase 3 (tenancy) are untouched. The one line this does not edit is the DESIGN_PHASE3.md index row, which #883 rewrites.
Why Phase 2	The thirteen claims are ones DESIGN_PHASE2.md itself made about the identity and signing controls. The procedure derived from them is phase-agnostic and says so.

Updated per @aslom's review (15b81da): CLAIMS.md is merged into
DESIGN_PHASE2.md §8 and the standalone file is gone — see "Addressing the review"
below.

---

Lessons from the eventing docs review (rossoctl/rossoctl#2609, now approved), written
back into the design record.

What happened

Writing the user-facing pages meant restating every claim in DESIGN_PHASE2.md for a
reader who cannot see the code. Six review rounds then checked each restated claim by
running it, against eventing/ at d9677ddd. Thirteen did not hold.

Not because the design record is vague about mechanism — §4.4 is unusually precise. Each
claim was written from the change that introduced the control, and stayed true for the
configuration that change was exercised in. The half-configured cases, and the cases a
later change moved, are where all thirteen were found. Every one coexisted with a
passing test suite.

Addressing the review

@aslom asked for CLAIMS.md to be merged into a design document, "as the claims are
part of design and they should reflect the phase of the design they are related to".
Done in 15b81da:

• CLAIMS.md deleted; its content is now §8.5–§8.8 of DESIGN_PHASE2.md, directly
  after the thirteen findings it was derived from. The content moved rather than being
  rewritten (133 lines out, 135 in); the five questions became §8.5.1–§8.5.5 and the
  three references that assumed a separate file are re-pointed.

• §8 now opens by stating which half is phase-specific, which is the substance of
  the request:

  §8.1–§8.4 are about Phase 2 specifically: thirteen claims this document made
  about the identity and signing controls, which do not hold in the code. §8.5–§8.8
  are the general procedure that came out of them, and apply to a claim made in any
  phase — Phase 3's design makes claims of the same shape about tenancy and the signed
  attribute set, and nothing here is Phase 2 only.

• agentdocs/README.md: standalone row removed, the Phase 2 row covers both halves, and
  the reading-order entry is "Writing a claim about a control, in any phase?" with
  "The findings are Phase 2's; the questions are not" stated outright.

That also settles question 1 from the original description, which asked whether
CLAIMS.md was the right home — the answer is no, and the phase-aligned placement is
better than repo root for the reason §8.5 now gives: a procedure filed away from the
findings that motivated it is a procedure nobody reads.

The changes

File	What
DESIGN_PHASE2.md	New §8: the thirteen claims, what the code does instead, the issue for each — and §8.5–§8.8, the procedure for checking a claim in any phase
agentdocs/README.md	Index and reading-order entries pointing at §8, with the phase distinction explicit

§8 is additive — §§2.6, 3, 4.3 and 4.4 are left as written, following this repo's
existing convention ("where Phase 1 supersedes Phase 0 it says so explicitly rather than
editing in place"). A correction that deletes the intent loses the more useful half.

The four §4.4 promises that fail

§4.4 says	The code does	Issue
"A keyset alone verifies and logs"	Verifies, returns a verdict, logs nothing — so the observable reject rate the two-flag rollout depends on does not exist	#886
Accepts "only" EventBridge's kid for group events, so a runner "cannot forge a group.completed"	Flags it, then applies it anyway: the rewrite keeps final and groupid	#885
"This proves who finished a run"	Not once stored — INSERT OR REPLACE lets a later unsigned frame overwrite the verified terminal row	#885
"Refuses to start with a keyset but no EB_SIGNING_KID"	True, but nothing checks the kid is in the keyset; it starts, then flags its own group events	#888

And a fifth, promised nowhere and worse than any of them:
EB_REQUIRE_RESPONSE_SIGNATURE=true with no keyset refuses nothing. require is
never consulted without a keyset, so a forged terminal is stored as an answer and
startup says response verification OFF. The request side with no key at all refuses
every request. The two sides fail in opposite directions and only one is loud (#888).

Relationship to the other drafts in flight

Stated by phase, since all three touch eventing/:

PR	Phase	Overlap with this
#884 signing deployable	Phase 2 — makes that phase's signing switchable on a cluster	None. #884 is manifests, scripts and tests; this is docs only.
#883 tenancy foundation	Phase 3 — steps 1–4 plus T5	Touches every file these issues cover but fixes none of them; kafka_in.py only re-points the store for tenancy. §8 stays accurate on top of #883, though line numbers move. We both edit agentdocs/README.md; whichever merges second resolves one table row and one reading-order entry.

Branched from main rather than #883 so this is reviewable on its own.

Remaining question

Should §8 name the fix, or only the finding? Right now it says what the code does and
links the issue, with proposed fixes left in the issues on the grounds that a design
record should not pre-empt the fix discussion. If you would rather §8 carry the intended
resolution so the record is self-contained, that is a small change.

(The original description also asked whether CLAIMS.md belonged at repo root — answered
above and implemented.)

Not included

No code changes. Issues #885–#889 carry those, and the fixes should land as their own PRs
with tests rather than riding along with a docs change.

🤖 Generated with Claude Code

[aslom]

@mrsabath please merge CLAIMS into DESIGN_PHASE2.md or DESIGN_PHASE3.md as I think the claims are part of design and they should reflect the phase of the design they are related to

[aslom]

Reviewed at 15b81da, treating §8 the way §8 asks to be treated: I checked every claim
it makes by running the code, not by reading it. All eleven enumerated findings hold.
One thing does not — the count.

Folding CLAIMS.md into §8 is the right resolution

15b81da does what I asked and the phase split is the part that makes it work. §8's
opening paragraph is the substance of the request: §8.1–§8.4 are Phase 2's findings,
§8.5–§8.8 are phase-agnostic, and it says which is which before a reader has to guess.
The README reading-order entry — "Writing a claim about a control, in any phase? …
The findings are Phase 2's; the questions are not" — closes it.

§§2.6, 3, 4.3, 4.4 is also the right attribution: §8.1 is §4.4, §8.2 is §3, §8.3 is
§2.6 and §4.3. I checked each.

Every finding verified against the code

eventing/ is byte-identical between d9677ddd and this PR's base 10980948
(git diff --stat is empty), so §8's "against eventing/ at d9677ddd" is still exactly
what a reader gets. Each item:

§	Claim	Verified
8.1	"keyset alone verifies and logs" → logs nothing	✔ response_decision returns (not require), so audit mode returns ok=True and kafka_in's if not ok: — the only print and the only self._rejected += 1 — never runs. Its own docstring says "hand back the reason so the caller can log it"; the caller does not.
8.1	group-event pinning flagged then applied	✔ ran it: an approved runner-01 signing a group.completed gives ok=False, "expected a signature from kid 'eb-01', got 'runner-01'", and it still routes to on_group_event
8.1	INSERT OR REPLACE lets an unsigned frame overwrite the verified row	✔ store.py:124, on (correlationid,sequence)
8.1	refuses without EB_SIGNING_KID, never checks kid ∈ keyset	✔ __main__.py:68-74 raises; no membership check anywhere
8.1	REQUIRE=true + no keyset refuses nothing	✔ ran it: ok=True why='not checked', forged terminal stored as the answer, banner says response verification OFF. And verify_event returns False with no key, so the request side refuses everything — the two sides do fail in opposite directions
8.2	GET /v0/groups unauthenticated, 100 most recent	✔ no auth.resolve in list_groups; all_groups(limit=100)
8.2	/status lists every member id; PUT /transcript unauthenticated	✔ auth.resolve appears at exactly two call sites, start_agent and create_group
8.2	ntfy suppression reads the event's own ce_groupid	✔ ntfy.py:188
8.3	§2.6 says "Both are unsigned"; they are in SIGNED_ATTRS	✔ §2.6:189 vs signing.py's tuple
8.3	SPIRE: EdDSA only, and keyset.load rejects a JWKS	✔ signing.py:344; fed it a real JWKS document — ValueError: key for kid 'keys' must be a string
8.3	flat keyset = attribution, not restriction	✔ EB's own key must be in the keyset for the pinning to work, and select(kid) then verifies it on any run's answer

Also checked: all five issues exist, are open, and their titles match what §8 attributes
to each — including #889 being cited in §8.7 as not one of the thirteen. §8.6's
reproduction recipe works verbatim: python3 -m venv .venv && .venv/bin/pip install -q -e .
then from shared import signing, keyset, ce imports cleanly. And the suite is green with
all of it present — 549 passed, 7 skipped — which is §8.6's own point.

The one claim that does not hold is the count

"Thirteen" appears six times in DESIGN_PHASE2.md and twice in README.md. §8.1–§8.3
enumerate eleven items: four table rows plus the fifth, three bullets in §8.2, three in
§8.3. Thirteen is reachable by splitting two compound bullets, but a reader who counts
gets 11 — and in a section whose thesis is that unchecked claims survive because nobody
counts, the one number that can be checked in ten seconds is the wrong one to leave
unreconciled. Inline comment with both ways to fix it.

Answering your question

Finding only — do not put the fix in §8. Three reasons, all from the document's own
reasoning:

• §8.7 already draws the line: "These documents are the record of what was intended."
  A finding is a fact about a named commit and stays true; a proposed fix is a hypothesis
  that review will change, and §8.8 is specifically about suggested wording being a
  hypothesis. A fix written into the design record acquires authority it has not earned.
• "What the code does" already carries the fix's shape. "require is never consulted
  without a keyset" tells a maintainer what to change without pre-empting how — whether
  that is consulting require outside the branch or refusing at startup is exactly the
  argument #888 should host.
• The §8.1 table's third column is the issue, and that is the right indirection. A fix in
  §8 would be the fourth place the same decision is written down, after the issue, the PR
  and the code comment — and §8.4's diagnosis is that claims rot when they are restated
  away from the thing they describe.

If you want the record self-contained, the cheaper version is one sentence per row saying
which direction the fix goes, not what it is.

On the other drafts

Two corrections to the overlap table, both in your favour and one against:

• "#883 fixes none of them" is right. I diffed kafka_in.py across #883's head:
  nothing in the verification block moves — no print, no counter, no change to the
  if self._keyset is not None guard. It threads topics/stores and nothing else, so
  §8 stays accurate on top of it, as you say.
• The README collision is one hunk, not two. git merge-tree against #883's head
  gives exactly one conflict, covering the Phase 2 and Phase 3 table rows together;
  the reading-order entries merge cleanly. Resolving it means keeping your Phase 2 row and
  #883's Phase 3 row (which drops "Design only — not implemented"). Worth knowing because
  #883 is now approved, so it is likely to merge first and the rebase will be yours.

Verification summary

Check	Result
Claims verified against code	11 of 11 hold
§8.6's documented recipe	works verbatim
Suite at this base	549 passed, 7 skipped
Issues #885–#889	all exist, open, titles match their attribution
eventing/ at d9677ddd vs base	identical
CI	11/11 green
Commits	2, both Signed-off-by, no Co-Authored-By
Scope	docs only — 2 files, no code, no manifests, no tests

Approving. The findings are accurate, the phase attribution is right, and the
procedure in §8.5–§8.8 is the most reusable thing in this repository's documentation.
The three comments are a count, a missing issue link and one imprecise mechanism — none
of them blocking, and the first is the only one I would not merge without.

[aslom]

[suggestion] "Thirteen" is the one claim in §8 a reader can check in ten seconds, and
it does not reconcile. §8.1–§8.3 enumerate eleven items:

§	Items
8.1 table	4 rows — and logs, group-event pinning, INSERT OR REPLACE, kid-not-in-keyset
8.1	+ 1, "A fifth, promised nowhere"
8.2	3 bullets — §3.1 capability URLs, §3.2 PUT /transcript, ntfy suppression
8.3	3 bullets — §2.6 signed-after-all, §4.3 SPIRE, flat keyset
	11

Thirteen is reachable by splitting two compound bullets — §8.2's first covers both
GET /v0/groups and GET /v0/groups/{id}/status, and §8.3's SPIRE bullet covers both the
algorithm and the file format — but nothing in the text marks them as two, so a reader
counting gets 11 every time.

The number appears six times in this file (lines 450, 458, 524, 532, 606, 639) and twice
in agentdocs/README.md (the Phase 2 table row and the reading-order entry), so it is
load-bearing for the section's own argument: §8.4 says "all thirteen were found" in the
half-configured cases, and §8.6 says "every one of the thirteen coexisted with a passing
suite". Both are claims about a set whose size the list does not support.

This matters more here than it would anywhere else in the repository, because §8 exists to
argue that unchecked claims survive review. A count that does not survive counting is the
same failure in miniature — and it is the first thing a sceptical reader will test.

Two ways out. Split the two compound bullets so the list reads 13, which is honest if
that is what the review found:

Suggested change

	**Scope.** §8.1–§8.4 are about **Phase 2** specifically: thirteen claims *this
	document* made about the identity and signing controls, which do not hold in the code.
	§8.5–§8.8 are the general procedure that came out of them, and apply to a claim made in
	any phase — Phase 3's design makes claims of the same shape about tenancy and the signed
	attribute set, and nothing here is Phase 2 only.
	Writing the user-facing pages for this path meant restating every claim in this
	document for a reader who cannot see the code. Six review rounds then checked each
	restated claim **by running it**, against `eventing/` at `d9677ddd`. Thirteen did
	not hold.
	**Scope.** §8.1–§8.4 are about **Phase 2** specifically: thirteen claims *this
	document* made about the identity and signing controls, which do not hold in the code —
	eleven items below, two of which carry a second finding in the same bullet (§8.2's
	capability-URL entry covers both the group list and the member list; §8.3's SPIRE entry
	covers both the algorithm and the keyset format).
	§8.5–§8.8 are the general procedure that came out of them, and apply to a claim made in
	any phase — Phase 3's design makes claims of the same shape about tenancy and the signed
	attribute set, and nothing here is Phase 2 only.
	Writing the user-facing pages for this path meant restating every claim in this
	document for a reader who cannot see the code. Six review rounds then checked each
	restated claim **by running it**, against `eventing/` at `d9677ddd`. Thirteen did
	not hold.

Or say eleven and change the eight occurrences, which is simpler and loses nothing —
the argument in §8.4 and §8.6 does not depend on the number being 13.

Either way worth doing in this PR rather than later: the count is quoted into
README.md, and #883 is about to touch the row next to it.

[aslom]

[suggestion] This is the only one of the eleven findings with no issue link, and §8.7
two screens down says the rule outright:

And file the issue. Most of the thirteen were code bugs rather than wording mistakes,
and they are tracked as #885, #886, #887 and #888.

The other two bullets in §8.2 both carry (#887). This one carries nothing, and it is
not obviously covered by any of the four: #887 is "unauthenticated PUT /transcript
rewrites what an agent resumes from, and correlation ids are listable", which is the
first two bullets and not this.

I verified the finding itself — ntfy.py:188 is groupid = event.get("groupid"), the
event's own attribute, with no membership lookup — so this is about the record, not the
claim. It is also the finding with the most concrete user-visible consequence of the three
in §8.2: a priority-5 push for a forged terminal with NTFY_GROUP_NOTIFY_ERRORS off, and
the member not counted as failed.

Either file it and link it, or say explicitly that it is untracked and why — "no issue:
the fix is subsumed by #885's enforcement work" would be a complete answer if that is the
case. An unlinked finding in the section that tells people to file the issue is the one
inconsistency a reader will notice fastest after the count.

[aslom]

[nit] The finding and its consequence are both right; the mechanism in the middle
names the wrong field and the wrong causal step.

I ran it. An approved runner-01 signing a group.completed against bridge_kid="eb-01":

verdict      : ok=False  why="expected a signature from kid 'eb-01', got 'runner-01'"
after rewrite: type=dev.rossoctl.agent.group.completed.v1
               groupid='eager-weasel-0787'  final=None
is_group_event(evt) -> True        <- the routing key
OUTCOME      : reaches on_group_event -> complete_group

Two corrections:

• A group event carries no final at all (final=None above) — ce.new_event is
  called without it in publish_group_event, and on_group_event never reads it. final
  is what matters for the member-answer row in this table, two rows down, not for this
  one.
• The rewrite is not why it still reaches on_group_event. kafka_in.py routes on
  ce.is_group_event(evt) — computed on the original event, before and regardless of
  the rewrite. Even a rewrite that stripped every attribute would still route there.
  Keeping groupid is why complete_group then finds the group to end; keeping type is
  why on_group_event dispatches to the completion branch.

So the precise version is one clause different:

Suggested change

| The verifier accepts "**only**" EventBridge's kid for group events, so an approved runner "cannot forge a `group.completed` and end a batch early" | Flags it, then applies it anyway. `kafka_in.py` rewrites `phase`/`data` but keeps `final` and `groupid`, so the rewritten event still reaches `on_group_event`. The group mirror replays group events unverified on restart. | [#885](https://github.com/rossoctl/examples/issues/885) |

| The verifier accepts "**only**" EventBridge's kid for group events, so an approved runner "cannot forge a `group.completed` and end a batch early" | Flags it, then applies it anyway. `kafka_in.py` routes on `ce.is_group_event(evt)` — the *original* event, independently of the rewrite — and the rewrite replaces only `phase`/`data`, so `type` and `groupid` survive and `on_group_event` ends the batch. The group mirror replays group events unverified on restart. | [#885](https://github.com/rossoctl/examples/issues/885) |

Flagging a one-clause imprecision because §8.5.1 is "Which code path, by name?" and
this row is the worked example a reader will copy the style from.

[mrsabath]

Done in 15b81da — CLAIMS.md is gone and its content is now §8.5–§8.8 of
DESIGN_PHASE2.md, directly after the thirteen findings it was derived from.

You put your finger on something the PR description was uneasy about and asked out loud:
"is CLAIMS.md the right home? It is process guidance sitting among design and build
records." The answer is no, and Phase 2 is the right home for a reason better than
tidiness — §8.4 already ended by handing off to the procedure, so the two halves were a
section boundary apart for no reason at all.

The content moved rather than being rewritten (133 lines out, 135 in): the five questions
became §8.5.1–§8.5.5, and the three references that assumed a separate file are
re-pointed — "see its §8 for the list" is now "see above", and "DESIGN_PHASE2.md §8 is
that shape" is now "§8.1–§8.3 above are that shape".

On "they should reflect the phase of the design they are related to" — that is the
part I wanted to get right rather than just comply with, because the two halves of §8
have different scopes. The findings are Phase 2's; the five questions are not. §8 now
opens by saying which is which:

§8.1–§8.4 are about Phase 2 specifically: thirteen claims this document made
about the identity and signing controls, which do not hold in the code. §8.5–§8.8 are
the general procedure that came out of them, and apply to a claim made in any phase —
Phase 3's design makes claims of exactly the same shape about tenancy and the signed
attribute set, and nothing here is Phase 2 only.

And §8.5's own lead-in gives the reason it is filed here rather than at repo root: a
procedure kept away from the findings that motivated it is a procedure nobody reads.

agentdocs/README.md: the standalone index row is gone, the Phase 2 row now covers both
halves, and the reading-order entry reads "Writing a claim about a control, in any
phase?" → §8.5 for the questions, §8.1–§8.3 for the findings, with "The findings are
Phase 2's; the questions are not" stated outright.

Verified every §N.N reference in the merged document resolves to a heading that exists,
and that no CLAIMS reference survives anywhere.

---

On phase clarity across the three open PRs, since this came up here and the same
confusion was possible on all three — each now carries its phase in the title and a
Phase / scope table at the top of the body saying which phase's goals it serves,
which sections it implements, and which phases it does not touch:

PR	Phase	Scope
#883	Phase 3	§9 steps 1–4 plus T5 — tenancy keys, per-user stores, AgentSpec. multi does not work end to end until T6.
#884	Phase 2	Makes that phase's signing deployable on a cluster. No tenancy.
#890	Phase 2	That phase's design record only. No code.

The one remaining open question from the original description is unchanged: should §8
name the fix, or only the finding? Right now it says what the code does and links the
issue, with proposed fixes left in #885–#889 on the grounds that a design record should
not pre-empt the fix discussion. Happy to carry the intended resolution in §8 instead if
you would rather the record were self-contained.