Repository navigation
docs(eventing): Phase 2 — make §8's count checkable, and correct one mechanism - #892
Conversation
…mechanism Follow-up to the #890 review, which landed before these were addressed. Three findings, all about §8 rather than the code it describes. **The count did not reconcile, in §8 of all places.** "Thirteen" appears six times in this file and twice in `agentdocs/README.md`, and §8.1–§8.3 enumerated **eleven**. The number was reachable only by knowing that two bullets each carried two findings in one paragraph — §8.2's capability-URL entry covered both `GET /v0/groups` and `GET /v0/groups/{id}/status`, and §8.3's SPIRE entry covered both the algorithm and the keyset file format — and nothing in the text marked them as two. Both are now split, so the list reads 13, and §8 carries the tally explicitly: five in §8.1 (four rows plus the fifth, promised nowhere), four in §8.2, four in §8.3. This mattered more here than it would anywhere else in the repository, which is the reviewer's point and the right one: §8 exists to argue that unchecked claims survive review, so a count that does not survive counting is the same failure in miniature — and it is the first thing a sceptical reader tests. **§8.1's group-event row named the wrong field and the wrong causal step.** It said `kafka_in.py` "keeps `final` and `groupid`, so the rewritten event still reaches `on_group_event`". Verified against the code: a group event carries **no `final` at all** (`publish_group_event` never sets it, `on_group_event` never reads it), and routing is on `ce.is_group_event(evt)` — computed on the *original* event, before and regardless of the rewrite. So the rewrite is not why it reaches the handler; `type` and `groupid` surviving is why the batch then ends. Corrected to the reviewer's wording. Worth getting exactly right because §8.5.1 is "**Which code path, by name?**" and this row is the worked example a reader copies the style from. **One finding had no issue, in the section that says to file the issue.** The ntfy group-suppression finding was the only one of the thirteen with no link, and #887 does not cover it — that one is `PUT /transcript` and listable ids. Filed as #891 and linked, with a pointer to §8.5.4 as the general rule it instantiates. While checking that, §8.3's items turned out to have no links either — correctly, because they are documentation drift rather than code bugs, so the correction *is* the fix. §8.3 now says so, since §8.7 tells readers to file issues and an unlinked finding otherwise reads as an oversight. Verified every `§N.N` reference in the file still resolves to a heading that exists; the only unresolved one is the pre-existing, explicitly qualified "Phase 1 §16". 575 passing, ruff clean. No code changes. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
The remaining nit from the #884 approval, which merged before it was addressed. My comment on both patch blocks warned about the failure that is **loud** — an `env/-` append failing with "path does not exist" when no list exists — and not about the one that is **silent**: `op: add` on `…/containers/0/env` *replaces* the list when one already exists. Safe in `kind-signed`, which builds on `../kind` whose containers have no `env`. Not safe in the overlay the PR plans to copy this patch into. Reproduced by building a throwaway `demo-signed` that changes nothing but the base: demo-signed eventrunner env = ['ER_SIGNING_KEY_PATH'] demo's own = ['ER_MOCK_CLAUDE', 'HOME', 'CLAUDE_CONFIG_DIR'] -> all three LOST, silently `kustomize build` succeeds and nothing warns. Losing `ER_MOCK_CLAUDE=false` means a `demo-signed` overlay would run the demo in **mock mode** — which is exactly what `test_the_test_overlay_mounts_no_credential_so_mock_mode_is_automatic` exists to keep honest at the other end, so the failure would be invisible from both directions. The comment now says that, and names the fix for whoever writes that overlay: `op: add` on `env/-` per entry, against a base that already has the key. Reviewer's words, and the right call to spend half a sentence on — this file is the starting point for three more (preflight, e2e, demo flow) by the PR's own plan. 575 passing, `kind-signed` still builds, no behaviour change. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
aslom
left a comment
There was a problem hiding this comment.
Summary
The tally this PR adds reconciles, and I checked it by counting rather than reading: §8.1 has four table rows plus the fifth at "A fifth, promised nowhere" (5), §8.2 has four bullets, §8.3 has four — 13. The §8.1 group-event correction is accurate against the code: kafka_in.py:136 routes on ce.is_group_event(evt), the original event, and the rewrite at kafka_in.py:122 is dict(d, phase="error", data={...}), so it replaces only phase/data and type/groupid survive. §8.3's new "no issue links, deliberately" note is also correct — those four bullets carry no links, and every item in §8.1 and §8.2 does carry one. #891 exists and matches the ntfy finding.
Two problems, both in the second commit's new kustomize comment, and both the shape §8 is about — a claim written from the change that introduced it. The warning says dropping the env list would "run the demo in MOCK MODE". It would not: envFrom is a sibling field that an env patch cannot touch, so anthropic-credentials still supplies ANTHROPIC_AUTH_TOKEN, and config.py:127 then auto-detects real mode. The actual loss is HOME/CLAUDE_CONFIG_DIR — the §16 Gap B transcript-durability failure the demo patch's own comment warns about. Separately, the copy of that warning in the eventbridge patch cites three env entries that ../demo only ever adds to eventrunner; ../demo patches eventbridge's volumes/0 and nothing else.
Neither blocks merge — no code, no behavior, CI green — but the mock-mode claim is the one I'd want corrected before this lands, since §8.5.1 is "Which code path, by name?" and this comment is now the counter-example.
Two notes too small for inline comments:
- The PR description says the only unresolved
§N.Nreference is the qualified "Phase 1 §16". There is a second, equally qualified one: "Phase 1 §1.1" at line 48. Every in-document reference does resolve, so the file is fine — only the description is incomplete. - Line 450's scope line still reads "§8.1–§8.4 ... thirteen claims", while the new tally localises all thirteen to §8.1–§8.3 (§8.4 is commentary and enumerates nothing). Defensible as a range, slightly odd now that the count is explicit.
Author: mrsabath (MEMBER — maintainer)
Areas reviewed: Docs (DESIGN_PHASE2.md), K8s/kustomize (kind-signed/kustomization.yaml, comments only), plus code verification of the claims in kafka_in.py and config.py
Agent/IDE config (.claude/.vscode): none
Commits: 2 commits, all signed-off: yes
CI status: passing (11/11 — CodeQL, DCO, Trivy, codeql, dependency-review, eventing-test, hadolint, lint, test, test-startup, trivy-scan)
| # `../kind`, whose containers have no `env` — but `../demo` has three entries, so | ||
| # a `demo-signed` overlay built by changing only the base would silently drop | ||
| # `ER_MOCK_CLAUDE=false`, `HOME` and `CLAUDE_CONFIG_DIR` and run the demo in MOCK | ||
| # MODE. `kustomize build` succeeds and nothing warns. Whoever writes that overlay | ||
| # wants `op: add` on `env/-` per entry, against a base that already has the key. |
There was a problem hiding this comment.
[suggestion] Mock mode is not what this would cause, and the mechanism is worth getting right because §8.5.1 is "Which code path, by name?".
op: add on …/env replaces the env list, but envFrom is a sibling field — a JSON patch on /spec/template/spec/containers/0/env cannot touch it. So in the hypothetical demo-signed, ../demo's envFrom.secretRef: anthropic-credentials (optional: false) survives and still supplies ANTHROPIC_AUTH_TOKEN. config.py then takes the auto-detect branch:
mock_env = (e("ER_MOCK_CLAUDE") or "").strip()
if mock_env: ...
elif has_api_credentials(): # ANTHROPIC_AUTH_TOKEN / ANTHROPIC_API_KEY
cfg.mock_claude = False
cfg.mock_reason = "auto: API credential present"With ER_MOCK_CLAUDE dropped, has_api_credentials() is true, so the runner comes up in real mode, reason auto: API credential present — the opposite of the warning. The demo patch's own comment already says this ("Phase 0's auto-detection would pick real mode on its own once the Secret mounts").
What actually breaks is HOME and CLAUDE_CONFIG_DIR: claude writes session transcripts under $HOME/.claude, so losing them moves transcripts off the mounted volume onto the container's ephemeral layer and a restart loses every resumable session on the pod — the §16 Gap B failure that patch calls out. Quieter than mock mode, which is the better argument for the warning.
| # `../kind`, whose containers have no `env` — but `../demo` has three entries, so | |
| # a `demo-signed` overlay built by changing only the base would silently drop | |
| # `ER_MOCK_CLAUDE=false`, `HOME` and `CLAUDE_CONFIG_DIR` and run the demo in MOCK | |
| # MODE. `kustomize build` succeeds and nothing warns. Whoever writes that overlay | |
| # wants `op: add` on `env/-` per entry, against a base that already has the key. | |
| # `../kind`, whose containers have no `env` — but `../demo` has three entries, so | |
| # a `demo-signed` overlay built by changing only the base would silently drop | |
| # `ER_MOCK_CLAUDE=false`, `HOME` and `CLAUDE_CONFIG_DIR`. Mock mode is *not* the | |
| # failure: `envFrom` is a sibling field and survives, so the credential is still | |
| # present and config.py auto-detects **real** mode ("auto: API credential | |
| # present"). What is lost is `HOME`/`CLAUDE_CONFIG_DIR`, so claude writes session | |
| # transcripts to the container's ephemeral layer instead of the mounted volume and | |
| # a restart loses every resumable session on the pod — the §16 Gap B failure. | |
| # `kustomize build` succeeds and nothing warns. Whoever writes that overlay wants | |
| # `op: add` on `env/-` per entry, against a base that already has the key. |
There was a problem hiding this comment.
Both corrections taken in 3e1dfa1, and the mock-mode one was right — I checked it by running config.load() rather than reading it, which is the rule this PR is about:
A. ../demo as it stands (env list present)
mock_claude = False
mock_reason = 'explicit ER_MOCK_CLAUDE=false'
B. demo-signed: `op: add` on .../env replaced the list
mock_claude = False
mock_reason = 'auto: API credential present'
ER_MOCK_CLAUDE is set in exactly one place in the tree — eventrunner-demo-patch.yaml — so with the env list replaced it is unset, envFrom still carries ANTHROPIC_AUTH_TOKEN, and config.py:127 takes the auto-detect branch. Real mode, reason auto: API credential present. The warning claimed the opposite.
Your eventbridge point is confirmed too: ../demo's eventbridge target is op: replace on volumes/0 and nothing else, so there is no env list to drop there at all. That patch now carries the general rule and points here for the worked example, per your suggestion.
一针见血 (yī zhēn jiàn xiě — "one needle, and blood shows") — the §8.5.1 counter-example was sitting in the same PR that added the checklist. That is the second time on this document that a sentence written from the change that introduced it did not survive being run, which is the finding generalizing rather than the finding being wrong.
Both description notes are also correct and fixed in the file: "Phase 1 §1.1" at line 48 is a second qualified reference I missed, and line 450's scope line now reads §8.1–§8.3, with §8.4 named as commentary that enumerates nothing.
| # `../kind`, whose containers have no `env` — but `../demo` has three entries, so | ||
| # a `demo-signed` overlay built by changing only the base would silently drop | ||
| # `ER_MOCK_CLAUDE=false`, `HOME` and `CLAUDE_CONFIG_DIR` and run the demo in MOCK | ||
| # MODE. `kustomize build` succeeds and nothing warns. Whoever writes that overlay | ||
| # wants `op: add` on `env/-` per entry, against a base that already has the key. |
There was a problem hiding this comment.
[suggestion] Same mock-mode correction as on the eventrunner patch below, plus one specific to this target: the three env entries cited here are never added to eventbridge.
../demo patches eventbridge only to replace volumes/0:
- target:
kind: Deployment
name: eventbridge
patch: |-
- op: replace
path: /spec/template/spec/volumes/0ER_MOCK_CLAUDE/HOME/CLAUDE_CONFIG_DIR come from eventrunner-demo-patch.yaml, which targets eventrunner. So for this container a demo-signed overlay would hit exactly the same empty-env case as ../kind — there is nothing to drop. The hazard is real but belongs to the eventrunner patch; stating it here with eventrunner's variable names makes the eventbridge patch look riskier than it is.
Suggest keeping the general rule here and leaving the worked example on the eventrunner patch:
| # `../kind`, whose containers have no `env` — but `../demo` has three entries, so | |
| # a `demo-signed` overlay built by changing only the base would silently drop | |
| # `ER_MOCK_CLAUDE=false`, `HOME` and `CLAUDE_CONFIG_DIR` and run the demo in MOCK | |
| # MODE. `kustomize build` succeeds and nothing warns. Whoever writes that overlay | |
| # wants `op: add` on `env/-` per entry, against a base that already has the key. | |
| # `../kind`, whose containers have no `env`. That is a property of this base, not | |
| # a general guarantee — see the eventrunner patch below, where `../demo` does add | |
| # an `env` list that a `demo-signed` overlay would silently replace. `kustomize | |
| # build` succeeds and nothing warns either way. Whoever writes that overlay wants | |
| # `op: add` on `env/-` per entry, against a base that already has the key. |
There was a problem hiding this comment.
Taken as suggested in 3e1dfa1. Verified your specific point first: ../demo's eventbridge target is
- target:
kind: Deployment
name: eventbridge
patch: |-
- op: replace
path: /spec/template/spec/volumes/0and nothing else, so there is no env list on this container to drop in a demo-signed overlay — the three variables I cited come from eventrunner-demo-patch.yaml, which targets eventrunner. You are right that naming them here made this patch look riskier than it is.
This one now carries the general rule and points at the eventrunner patch for the worked example; the mock-mode correction is applied there.
|
|
||
| ### 8.3 Three things about the keys | ||
|
|
There was a problem hiding this comment.
[nit] Removing the old ### 8.3 Three things about the keys heading left two consecutive blank lines between the new "No issue links in this subsection" paragraph and the §2.6 bullet. Renders the same; just an artifact of the edit. One blank line here.
There was a problem hiding this comment.
Fixed in 3e1dfa1 — one blank line. Artifact of removing the old ### 8.3 Three things about the keys heading, as you diagnosed.
The warning said that a `demo-signed` overlay dropping `../demo`'s `env` list
would "run the demo in MOCK MODE". It would not.
`op: add` on `/spec/template/spec/containers/0/env` replaces the `env` list,
but `envFrom` is a sibling field a JSON patch on `env` cannot reach. So
`anthropic-credentials` (`optional: false`) still supplies
`ANTHROPIC_AUTH_TOKEN`, `ER_MOCK_CLAUDE` is unset, and
`eventrunner/config.py:127` takes the auto-detect branch. Driving the real
`config.load()` with the surviving environment:
mock_claude = False
mock_reason = 'auto: API credential present'
Real mode, the opposite of the warning. What is actually lost is
`HOME`/`CLAUDE_CONFIG_DIR`: claude writes session transcripts under
`$HOME/.claude`, so they move to the container's ephemeral layer and a restart
loses every resumable session on the pod — the §16 Gap B failure the demo
patch's own comment warns about.
The copy of the warning on the **eventbridge** patch also cited three `env`
entries that `../demo` only ever adds to **eventrunner**; `../demo` patches
eventbridge's `volumes/0` and nothing else. That patch now carries the general
rule and points at the eventrunner patch for the worked example.
Also from the same review: the §8 scope line localises the thirteen findings to
§8.1–§8.3 now that the tally is explicit, since §8.4 is commentary and
enumerates none; and a stray double blank line left by removing the old §8.3
heading.
`kustomize build` on the overlay still succeeds. No manifest field changed —
comments, and three lines of DESIGN_PHASE2 prose.
Assisted-By: Claude Code
Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
…ermission (#897) * docs(eventing): Phase 4 design — enrollment, and asking a human for permission Two gaps Phase 2 and Phase 3 leave, which are the two halves of one idea: the notification channel is bidirectional. Phase 3 §4.2 derives an unguessable per-user ntfy topic and §4.3 provisions the reader account, but both are operator-side — a user who signs in with GitHub has no way to learn their own topic, and the token is printed to the *operator's* terminal. And every notification so far has been an announcement: Phase 3 §5.1 fixes each agent's permission mode and tool allowlist before the run, so an agent that needs a permission it was not granted cannot ask and a human cannot answer. There is no approval vocabulary anywhere in the tree. That second gap is what makes Phase 3's triggers risky to enable. §7.4 concedes a trigger's rendered prompt is attacker-influenced text and §5.1 answers with default-deny tools; default-deny with no escalation path means the operator's rational move is to widen the allowlist permanently for a capability needed once. An approval path is what keeps a narrow allowlist narrow. §2 is the measured part, and the rest depends on it Phase 2 §8.6's rule earned its keep twice. Against the CLI: a PreToolUse hook blocks 75 s with the tool not running (93 s wall clock), honours a per-hook "timeout": 120, refuses on `deny`, and delivers permissionDecisionReason to the agent — which then explained the refusal in its answer. `allow` runs the tool. Two negative findings, both attractive wrong turns. `--permission-prompt-tool` does not exist; the flag is `--permission-prompts <host|none>` and its own help defines `host` as the SDK host, so it needs the Agent SDK in-process — as does `canUseTool`. EventRunner spawns a subprocess, so the hook is the only mechanism available to it. And `plan` mode never attempts the guarded call at all: the agent wrote a plan and reached for ExitPlanMode, so the hook adjudicated nothing. Phase 3 §5.1 proposes `plan` as the safe default; for an agent subject to approval it is the wrong mode. The hook needs no new plumbing — AgentSpec already carries `settings_file`, resolved under the agent dir by `_resolve_under` and passed as `--settings`, and Phase 3 §5.2 already requires that directory be unwritable by the agent. The control that matters most Phase 3 §4.4 MACs `userkey|corr|exp`, which is right for continuing a conversation. Authorising an *action* has to bind the action: the approval key covers a digest of the canonical tool input, and the hook re-checks that digest before returning `allow`. Without it a key that approved `echo hi` could be replayed against a rewritten command on the same approval id. The verdict is a row read from the store, never an attribute off the event — Phase 2 §8.5.4, whose cautionary tale is group-member push suppression being defeated by omitting the attribute it reads. Once-only is a guarded UPDATE on `decided_utc IS NULL`, the pattern mark_member_finished already uses. A timeout is recorded as `timeout`, not `deny`: "nobody was awake" and "a human said no" need different fixes, and collapsing them hides which is happening. The budget inequality is pinned, because a killed hook is a non-blocking error — execution proceeds, i.e. the tool runs unapproved. Three ntfy action buttons is a hard ceiling and `_publish` already fills it, so an approval cannot be the result notification with a button added. §9 answers Phase 2 §8.5's five questions before this ships, including the hole that remains: the hook runs in the pod the agent runs in, so the mitigation is a filesystem permission rather than a cryptographic control. Design only — no code, no manifests, no tests, matching #892/#893 and the convention that a phase design lands as a reviewable record first. Nothing in it is implemented; §11 orders the tasks behind Phase 3's ntfy isolation and capability key. Not verified: no cluster access, and §2 was measured on CLI 2.1.270 while Dockerfile-eventrunner-claude pins 2.1.278 — §10 says so, and §9.1's pinned-CLI test is the first thing to build. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> * docs(eventing): Phase 4 — address review of #897 Review: #897 (approved), five inline comments — two substantive, three nits. - §5.7's budget chain now carries §6's bound: the hook's wait, not just the turn, fits inside ER_DRAIN_TIMEOUT_S < terminationGracePeriodSeconds. The paragraph says why that link matters — ER_APPROVAL_TIMEOUT_S is the knob an operator raises when two minutes is not enough, and at 600 it outlives the drain budget, so a rolling update lands a SIGKILL mid-approval and §6's late-approval path replays a turn whose partition was taken away mid-flight. The startup check prints all five values; test_manifests.py's drain test is named as gaining the one-more-assert (ER_APPROVAL_TIMEOUT_S < ER_DRAIN_TIMEOUT_S, in the rendered ConfigMap). - §7.2's ER_APPROVAL_TIMEOUT_S row: "below the per-hook timeout AND below ER_DRAIN_TIMEOUT_S", pointing at §5.7's full chain. - §5.6 gains the missing 503 row (EB_CAPABILITY_SECRET_PATH empty, §7.1), distinguishing it from 404: the endpoint exists, the remedy is the operator's. - §3.4 and §6: dropped the duplicated "Phase 1" / "Phase 3" across the line wraps. - §9.5's VERIFY marker: dropped the ```markdown fence so both instances render the same (invisible HTML comments); grep-ability unchanged. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Mariusz Sabath <mrsabath@gmail.com> --------- Signed-off-by: Mariusz Sabath <mrsabath@gmail.com>
Phase / scope
DESIGN_PHASE2.md§8)Addresses the three findings in the #890 approval. All about §8 rather than the code it
describes, and all verified against the tree.
The count did not reconcile — in §8 of all places
"Thirteen" appears six times in
DESIGN_PHASE2.mdand twice inagentdocs/README.md,and §8.1–§8.3 enumerated eleven. Thirteen was reachable only by knowing that two
bullets each carried two findings in one paragraph:
GET /v0/groupsandGET /v0/groups/{id}/statusNothing in the text marked either as two, so a reader counting got 11 every time.
Both are split now, and §8 carries the tally explicitly: five in §8.1 (four rows plus
the fifth, promised nowhere), four in §8.2, four in §8.3.
@aslom's framing is why this was worth more than a wording tweak: §8 exists to argue that
unchecked claims survive review, so a count that does not survive counting is the same
failure in miniature — and it is the first thing a sceptical reader tests.
§8.1's group-event row named the wrong field and the wrong causal step
It said
kafka_in.py"keepsfinalandgroupid, so the rewritten event still reacheson_group_event". Checked against the code:finalat all —publish_group_eventnever sets it andon_group_eventnever reads it.finalmatters for the member-answer row two rowsdown, not this one.
ce.is_group_event(evt), computed on the original event, before andregardless of the rewrite. So the rewrite is not why it reaches the handler;
typeandgroupidsurviving is why the batch then ends.Corrected to the reviewer's wording. Worth being exact because §8.5.1 is "Which code
path, by name?" and this row is the worked example a reader copies the style from.
One finding had no issue, in the section that says to file the issue
The ntfy group-suppression finding was the only one of the thirteen with no link, and
#887 does not cover it — that one is
PUT /transcriptand listable correlation ids.Filed as #891 and linked, with a pointer to §8.5.4 as the general rule it
instantiates ("if the decision reads an attribute off the event, the forger controls that
attribute — including by leaving it out").
While checking that, §8.3's items turned out to have no links either — correctly,
because they are documentation drift rather than code bugs, so the correction is the
fix. §8.3 now says so, since §8.7 tells readers to file issues and an unlinked finding
otherwise reads as an oversight rather than a decision.
Verification
§N.Nreference in the file resolves to a heading that exists; the onlyunresolved one is the pre-existing, explicitly qualified "Phase 1 §16".
ruff checkandruff format --checkclean. No codechanged.
Not included
The fifth suggestion from that review — making
render()fall back tokustomize buildso
test_manifests.py's assertions run in CI without a cluster — is deliberately out ofscope. It is a change to test infrastructure that affects five assertions across two
phases, and @aslom's own note said it is bigger than a docs PR should carry. Worth its own
issue alongside #885–#891.
🤖 Generated with Claude Code