From ae89f327661b9e27a5fe04dc59cff52633ed6ded Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Mon, 5 Oct 2026 18:57:04 -0400 Subject: [PATCH 1/4] =?UTF-8?q?docs(eventing):=20KubeCon=20NA=20knowledge?= =?UTF-8?q?=20base=20=E2=80=94=20abstract,=20status,=20plan,=20slides?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reference for the accepted KubeCon NA talk, kept beside the designs it cites so its claims can be checked against the code the same way theirs can. Four parts: the abstract as submitted with its five demo beats broken out and numbered; the status of each against the tree as of 2026-10-05; the W1-W6 plan to close the gaps; and the 30-minute slide and demo shape. Two of the four seams in the abstract name SPIRE, and there is no SPIRE in the implementation -- seven mentions, all prose. The egress block the demo list promises has no NetworkPolicy, Istio or sidecar behind it at all. Those are the two gaps the plan is ordered around, and SPIRE is its own section because the choice between wiring it and reframing it decides both the plan and the slides. The status section was written by running the decision functions rather than reading them, which is DESIGN_PHASE2.md section 8.6's rule. It changed two conclusions: a forged group.completed is refused by the verifier and then applied anyway (#885, traced through kafka_in.py to on_group_event), and enforcement with no keyset refuses nothing while printing "response verification OFF" (#888). An unsigned non-terminal frame is also accepted, but that one is the signing policy on the other side, not a hole -- recorded as such. Also recorded: the signed path has never run on a cluster, and k8s_demo_flow.py cannot drive the kind-signed overlay, so there is no evidence it survives a real broker. Section 4.4 step 2 is the precedent -- the signer/verifier policy mismatch was found against a live broker and by no unit test. Section 7 keeps what the review found that no single document states, including a false negative of its own: the first group-forgery harness used wrong event-type constants and reported nothing. A negative result from a hand-written harness is worth little until the harness has produced a positive one. No code, no manifests, no tests. Adds a row and a reading-order entry to the agentdocs index. Assisted-By: Claude Code Signed-off-by: Mariusz Sabath --- eventing/agentdocs/KUBECON_NA_2026.md | 533 ++++++++++++++++++++++++++ eventing/agentdocs/README.md | 5 + 2 files changed, 538 insertions(+) create mode 100644 eventing/agentdocs/KUBECON_NA_2026.md diff --git a/eventing/agentdocs/KUBECON_NA_2026.md b/eventing/agentdocs/KUBECON_NA_2026.md new file mode 100644 index 00000000..c1bacf93 --- /dev/null +++ b/eventing/agentdocs/KUBECON_NA_2026.md @@ -0,0 +1,533 @@ +# KubeCon NA — "The Seams Are the Trap" + +Status: **knowledge base, living document.** Accepted talk; 30-minute session slot. +Scope: everything needed to deliver the talk — the abstract as submitted, what we +committed to show, the gap between that and the code, the plan to close it, and the +slide/demo shape that fits 30 minutes. + +This file is the reference other analyses should cite. It is deliberately specific +about what is **not** done, because the talk's own thesis is that unchecked control +claims are where compositions leak — and `DESIGN_PHASE2.md` §8 is the record of this +repository making thirteen of them. + +Read alongside: + +| For | Read | +|---|---| +| The wire contract that must not change | [`DESIGN_PHASE0.md`](DESIGN_PHASE0.md) | +| KEDA scaling, scale-to-zero, the §16 gaps | [`DESIGN_PHASE1.md`](DESIGN_PHASE1.md) | +| Identity on the event path, **and §8's thirteen findings** | [`DESIGN_PHASE2.md`](DESIGN_PHASE2.md) | +| Per-user isolation, `AgentSpec`, triggers (design only) | [`DESIGN_PHASE3.md`](DESIGN_PHASE3.md) | +| Measured results, what is blocked | [`IMPLEMENTATION_REPORT1.md`](IMPLEMENTATION_REPORT1.md) | + +--- + +## 1. The abstract as submitted + +**Title:** The Seams Are the Trap: Agent Fleets from CNCF Parts You Already Have + +> Every vendor is shipping an agent platform. You probably don't need one — what you +> need to know is how to compose the CNCF projects you already are running to get +> headless agents running safely in your clusters. +> +> We identify challenges: cold-start identity gap (the agent egresses before SPIRE +> issues an SVID), trust-scope drift (egress allowlist decided at deploy, stale by +> wake), unverified delivery (the event broker hands off events before signature +> check), and unbound audit (completion not chained back to the signed wake). +> +> We'll walk through one reference pattern: KEDA scaling deployments from queue depth, +> signed CloudEvents (SPIFFE source + JWS) as the wake contract, sidecar policy reading +> SVIDs at runtime, and the event broker log as tamper-evident audit. In live demo we +> show: zero agent replicas → signed task event to agent wake → blocked egress on a +> wrong-scope call → completion bound to the wake event → scale to zero when agents are +> idle. + +**Benefits to ecosystem, as submitted** (paraphrased to the four commitments it makes): + +1. A four-point **seam taxonomy** — cold-start identity gap, trust-scope drift, + unverified delivery, unbound audit — each a named integration point between two + graduated/sandbox projects. +2. A **composition counter-argument** to monolithic agent runtimes, built from KEDA, + Knative Eventing (or Strimzi-backed Kafka), SPIFFE/SPIRE, CloudEvents + JWS. +3. **Vendor neutrality** — stock CNCF projects; Kagenti/rossoctl referenced as OSS and + itself a composition, not a platform. +4. A **demo-backed artifact** the community can reference after the talk. + +### 1.1 The five demo beats we committed to + +The abstract names these in order. Numbering them is useful because the plan below +tracks each one separately. + +| # | Beat | Backed by | +|---|---|---| +| D1 | zero agent replicas | ✅ KEDA `minReplicaCount: 0`, measured | +| D2 | signed task event → agent wake | 🟡 JWS real, never run on a cluster | +| D3 | blocked egress on a wrong-scope call | ❌ **no implementation at all** | +| D4 | completion bound to the wake event | ✅ `ce_causationid` | +| D5 | scale to zero when agents are idle | ✅ measured, 22 s full cycle | + +--- + +## 2. Status against the abstract + +Assessed 2026-10-05 against `main`. Method: read all nine `agentdocs/` files and the +full source, then **ran the identity decision functions** rather than trusting +docstrings — which is this repository's own §8.6 rule, and it changed two conclusions. + +Test suite at review time: **575 passed, 5 skipped**. (`DESIGN_PHASE2.md` §6 says +490/5 — the suite has grown; the doc is stale, not wrong.) + +### 2.1 The four seams + +| Seam | Named CNCF integration | Code status | +|---|---|---| +| Cold-start identity gap | SPIRE → KEDA | ❌ **No SPIRE/SPIFFE/SVID in the implementation.** 7 mentions total, all prose, all aspirational | +| Trust-scope drift | SPIRE → egress sidecar | ❌ **No NetworkPolicy, no Istio, no AuthorizationPolicy, no sidecar** | +| Unverified delivery | CloudEvents verifier → Knative trigger | 🟡 Verification exists and is real; **no Knative** (Strimzi instead, which the abstract permits) | +| Unbound audit | event bus → work record | 🟡 `ce_causationid` binds response to request; **the replay path verifies nothing** | + +### 2.2 What is genuinely done and demo-ready + +- **KEDA scale-to-zero, measured on two clusters.** OpenShift and Kind. POST → KEDA + decision 1.7–3.0 s; POST → pod Ready ~14 s; full cycle back to zero 22 s. +- **Detached JWS over the CloudEvent envelope.** Ed25519 implemented from RFC 8032 in + pure Python, checked against the RFC's own test vectors. +- **The canonicalization is injective.** Length-prefixed (netstring-style) fields, after + a plain `key=value` form was found to let a newline in a value synthesize an extra + attribute — two structurally different events encoding to identical bytes, one + signature validating both. +- **`ce_causationid`** on every response, binding it to its triggering request. +- **User identity end to end.** GitHub device flow → `GET /user` → approved-list check, + `401`/`403` deliberately distinct, `ce_submitter` + `ce_submitteriss` on the wire. + Verified against a real account (§6 of Phase 2). +- **`k8s_demo_flow.py`** asserts the four-stage loop with timings rather than narration. + +### 2.3 Two results that are better than the abstract claims + +Worth slide space; they are the strongest engineering in the repo. + +**Lag means "work not finished," not "not read."** Deferring the Kafka offset commit to +the terminal event makes consumer lag a *correct* queue gauge and makes it structurally +impossible for KEDA to scale away a streaming pod. Proven by accident: a real run sat in +429 retry backoff for 300 s and replicas stayed ≥ 1 for all 300 s. + +**`ce_causationid` already closes the "unbound audit" seam.** The abstract presents it as +a challenge; the code has it. + +### 2.4 Confirmed by running the code + +Not from docstrings. These five lines are the honest state of response verification: + +``` +A) require=True, keyset=None -> ACCEPTED ("verification not enabled") ← #888 +B) forged unsigned terminal -> refused ✅ +C) signed by unapproved kid -> refused ✅ +D) forged unsigned NON-terminal -> ACCEPTED (by design, see below) +E) forged group.completed, enforced -> refused by verifier… then applied anyway ← #885 +``` + +**(D) is a deliberate policy, not a bug.** `emit()` signs terminal events only, because +it runs per `stdout` frame and Ed25519 costs ~150–200 ms here; signing every frame would +add minutes to a chatty run. The verifier matches that policy. The honest limit: this +proves *who finished a run*, not *what it said along the way*. Say that on stage. + +**(E) is the one that will bite on stage.** `signing.response_decision` correctly refuses +the forged `group.completed`. But `kafka_in.py:122` rewrites only `phase` and `data` — it +preserves `type` and `groupid`, then routes on `is_group_event(evt)` using the +**original** event. `group_service.on_group_event` reads only those two fields, so it +calls `complete_group()` anyway. A forged event the verifier *correctly rejected* still +ends the batch and fires a "100 finished" notification for work that never ran. The group +mirror also replays group events with **no verification at all** on restart. + +So "the event broker log is tamper-evident audit" is currently half-true: the log is +replayable, but the replay trusts it unconditionally. + +### 2.5 Open bugs, all still open + +From the `DESIGN_PHASE2.md` §8 docs review. None fixed as of 2026-10-05. + +| Issue | What | +|---|---| +| [#885](https://github.com/rossoctl/examples/issues/885) | Response-signature enforcement is advisory — rejected events still end batches and overwrite verified answers | +| [#886](https://github.com/rossoctl/examples/issues/886) | Audit mode records nothing, so the documented two-flag rollout cannot be performed | +| [#887](https://github.com/rossoctl/examples/issues/887) | Unauthenticated `PUT /transcript` rewrites what an agent resumes from; correlation ids are listable | +| [#888](https://github.com/rossoctl/examples/issues/888) | `EB_REQUIRE_RESPONSE_SIGNATURE` without a keyset refuses nothing and reports nothing | +| [#889](https://github.com/rossoctl/examples/issues/889) | Stale pidfile naming PID 1 crash-loops the pod after an abrupt exit | + +#888 is the sharpest asymmetry and the easiest for a reviewer to find: the response side +with enforcement on and no keyset refuses **nothing** while printing +`response verification OFF`; the request side with `ER_REQUIRE_SIGNATURE=true` and no key +refuses **everything**. Both were documented as "refusal is on." Only one was true. + +#887 also defeats Phase 2 §3.1's capability-URL argument: `GET /v0/groups` returns the +100 most recent groups without sign-in, and `/v0/groups/{id}/status` lists every member's +correlation id. For any conversation in a group, the capability is published. + +### 2.6 The signed path has never run on a cluster + +`k8s/overlays/kind-signed/` exists and is carefully written. But: + +- No document records a run of it. +- `k8s_demo_flow.py --overlay` accepts only `test`, `kind`, `demo` — it **cannot drive + `kind-signed`**. +- Every signing result is unit-tested only. + +This matters because of this repository's own precedent: `DESIGN_PHASE2.md` §4.4 step 2 +records that the signer/verifier policy mismatch (signing terminals only, verifying +all-or-nothing) was found **by running against a live broker, never by a unit test** — +because every test until then used terminal events. + +### 2.7 The SPIRE upgrade path does not currently hold + +`keyset.py` claims the keyset "is the same shape as a JWKS, so the upgrade path — SPIRE +issuing and rotating the keys — replaces *where the keys come from* without changing a +line of verification logic." + +`DESIGN_PHASE2.md` §8.3 already records that this is false, and it is worth repeating +here because it is load-bearing for the talk: + +- The verifier accepts **EdDSA only**; SPIRE issues EC or RSA. +- The keyset is a flat JSON map of kid → key; `keyset.load` **rejects a JWKS document + outright**. +- The kid lookup survives. The algorithm and the file format do not. + +Also from §8.3, and under-appreciated: **a flat keyset makes the asymmetric keys +attribution, not restriction.** EventBridge's own key is in `EB_VERIFY_KEYSET_PATH` by +requirement, so its signature verifies on any run's answer — as does any approved +runner's. Phase 2 §4.4 step 5 draws this conclusion for group events and pins them to +`EB_SIGNING_KID`; it does not draw the same conclusion for member answers. + +To its credit, `keyset.py` is honest where it counts: *"What it does not do is attest +anything. The set records which keys an operator approved, not which workloads a platform +vouched for. Say that plainly when demoing it."* + +--- + +## 3. The decision that shapes the rest: SPIRE + +Two of the four seams name SPIRE. There is no SPIRE. This is the one call to make before +anything else, because it determines both the plan and the slides. + +### Option A — reframe the talk (recommended) + +Keep the Ed25519 keyset as the *working* control and present SPIRE as the named upgrade +path with the honest gap. Three seams delivered and demo-backed; the fourth scoped +explicitly as future work, with the §8.3 obstacles named. + +**Why this is recommended.** You have a strong, measured, honest talk about three seams. +A fourth faked on stage is a bigger risk than one openly scoped as future work — and the +talk's own thesis is that unchecked control claims are the trap. Overselling the guarantee +in a talk *about* overselling guarantees is the one failure mode with no recovery. + +There is also a real seam you can speak to without SPIRE: **signing is opt-in, and the +two sides fail in opposite directions** (#888). That *is* a cold-start trust gap — a +different one than the abstract names, but a genuine, measured, reproducible one. + +### Option B — wire a minimal SPIRE + +Deliverable: SPIRE server + agent on Kind, EventRunner fetches an X.509 or JWT SVID, the +SPIFFE ID becomes the CloudEvent `source`, and the verifier resolves keys from the SPIFFE +trust bundle instead of a ConfigMap. + +**Cost, honestly:** this is not a flag flip. It requires an issuer-agnostic verifier +(add EC/RSA to a hand-rolled pure-Python Ed25519 implementation, or relax the §1.1 +no-C-extensions rule and adopt `cryptography`), plus JWKS/trust-bundle parsing, plus the +SVID fetch and rotation handling. Call it the dominant engineering cost of the whole +remaining plan. + +**The §1.1 tension is the real decision inside Option B.** Phase 2 §4.3 already notes +that if the pure-Python constraint is relaxed, `cryptography`'s Ed25519 is the better +trade than hardening the hand-rolled one. SPIRE forces that question: supporting EC/RSA +by hand is substantially more code than Ed25519 was, and `_scalar_mult` already carries a +documented "not side-channel resistant — do not promote this to a trust boundary that +assumes it is" warning that now has production callers. + +### Recommendation + +**Option A, with one carve-out.** Reframe SPIRE as the upgrade path, *and* build the +egress block (D3) with NetworkPolicy — which needs no SPIRE at all and restores the most +visual demo beat. See W3 below. + +--- + +## 4. Implementation plan + +Ordered by ratio of talk-value to effort. W1–W3 are the ones that change what you can +honestly say on stage; W4–W6 are polish. + +### W1 — Fix #885 and #888 (highest value, smallest change) + +These are the difference between a security demo and security theater, and a sharp +audience member can find #888 from the slides alone. + +- **#888:** consult `require` even when no keyset is loaded. `kafka_in.py` has + `ok, why = True, "not checked"` behind `if self._keyset is not None`, so `require` is + never reached. Fail closed when enforcement is on and verification is unconfigured, and + make the startup line say so instead of `response verification OFF`. +- **#885:** route on the **rewritten** event, not the original — or clear `groupid` and + `type` on rejection so `on_group_event` cannot be reached. Then decide what the group + mirror does on replay: verifying on replay is the correct answer, but note that it + changes restart behaviour for *existing* unsigned topic data, so it needs a flag. +- Add a test per bug that asserts the **outcome** (the batch does not complete), not the + verdict. Phase 2 §8.6's rule: `response_decision` returning `False` is not the same as + the batch not completing. + +Exit criteria: forged `group.completed` with enforcement on does **not** complete the +batch and does **not** notify; enforcement with no keyset refuses and says why. + +### W2 — Run `kind-signed` end to end, and make it demo-drivable + +Without this there is no evidence the signed path survives a real broker, and the +repository's own precedent says unit tests do not catch this class of bug. + +- Add `kind-signed` to `k8s_demo_flow.py --overlay` choices (currently + `("test", "kind", "demo")`). +- Add two demo stages: **a signed wake that verifies** (D2) and **a forged event that is + refused** — the latter is the money shot and currently has no script. +- Record the result in `IMPLEMENTATION_REPORT1.md` §8, which still says + `ER_REQUIRE_SIGNATURE=true` has no end-to-end run. +- Measure the signing cost on-cluster. Ed25519 here is ~150–200 ms/op in pure Python; + the wake-path latency number on the slides must include it. + +Exit criteria: one command drives zero → signed wake → verified completion → forged +event refused → zero, with assertions and timings. + +### W3 — Build the egress block (D3) + +The only promised demo beat with zero implementation, and the most visceral one. + +Minimum viable, no SPIRE needed: a `NetworkPolicy` on the EventRunner namespace with a +DNS + broker + model-endpoint egress allowlist, then have the demo agent attempt a +call outside it and show the block. + +- **Make the failure legible.** A `NetworkPolicy` denial is a connection timeout, which + on stage is indistinguishable from a slow network. Either surface the agent's own error + frame (`phase=error` already renders as a red card in the SSE transcript) or add a + deliberately short client timeout so it fails in ~2 s rather than 30. +- **Name the limit out loud:** this is a deploy-time allowlist, which is precisely the + "trust-scope drift" seam the abstract says is the *problem*. Demoing the static version + and naming the drift is honest and still makes the point. Claiming it is scope-aware at + wake time would not be. +- If Option B lands later, the same beat upgrades to sidecar-reading-SVID without + changing the demo script. + +Exit criteria: a wrong-scope call is blocked, visibly, within a few seconds, in under +60 s of demo time. + +### W4 — Fix #887 and #889 + +- **#887** is the one with the widest blast radius for a live demo: an unauthenticated + `PUT /transcript` changes what the agent *resumes from*, and request signing does not + cover it. If anyone in the room has the Route hostname, this is reachable. At minimum, + gate it behind the Phase 2 §3.1 planned HMAC capability key + (`key = HMAC(server_secret, correlationid + exp)`), which is stdlib-only and stores + nothing. +- **#889** is a pidfile crash-loop after an abrupt exit. Low glamour, high demo risk — + this is the bug that bites when you restart a pod between rehearsal and the live run. + +### W5 — Honesty pass on the docs the talk points at + +The talk will send people to this repository. Three fixes: + +- `keyset.py`'s "upgrade path changes no verification logic" claim — correct it per + §8.3, in place, since the module docstring is what a reader lands on. +- `DESIGN_PHASE2.md` §6's test count (490/5 → current). Small, but the talk's thesis is + about claims that quietly stopped being true. +- Add the `` comments §8.5.5 prescribes to any statement in this file that + depends on #885–#889 staying open. That is this document eating its own cooking. + +### W6 — Stretch, only if W1–W4 land early + +- Minimal SPIRE (Option B). Treat as a separate, post-talk workstream unless everything + else is done with weeks to spare. +- Phase 3's `userkey` / per-user topic isolation is drafted (`DESIGN_PHASE3.md` §2–§3) + and **not implemented**. Do not promise it. It is good "what's next" slide material and + nothing more. + +### Sequencing + +W1 → W2 → W3 are strictly ordered: W2 needs W1's fixes to be worth running, and W3's +beat slots into the W2 demo script. W4 is parallel. W5 is last and cheap. Rehearse on +Kind, not OpenShift — Kind sets `group.initial.rebalance.delay.ms: 0` while the shared +cluster leaves it at 3000 ms, so **Kind timings are not comparable to cluster timings**; +pick one for the recorded numbers and say which. + +--- + +## 5. The 30-minute shape + +30 minutes is roughly **22–24 minutes of content, 4 minutes of demo, 3–4 minutes of +questions.** The demo is the constraint: D1–D5 as written cannot be driven live in four +minutes, because POST → pod Ready alone is ~14 s and a full cycle to zero is 22 s. + +### 5.1 Demo strategy: pre-record, narrate live + +**Record the demo. Do not run it live.** Reasons, in order of how much they would hurt: + +1. The full four-stage loop is ~6 minutes in mock mode (`k8s_demo_flow.py`), and the + real-agent path has been blocked on an exhausted LiteLLM team budget + (`IMPLEMENTATION_REPORT1.md` §7) — a 429 on stage is unrecoverable. +2. Image pull is ~12 s of the ~14 s wake. Pre-pull on the node and the number improves, + but it is still dead air. +3. Conference wifi. Phase 2 §2.5 already lists offline operation as a design + requirement for exactly this reason. +4. #889 crash-loops a pod after an abrupt exit. + +Record at 1× with a visible clock, cut the dead air with explicit on-screen jumps +("+12 s"), and narrate live over it. Have `watch_topics.py` output on screen for the lag +numbers — it shows the claimed/unclaimed split, which is the detail that makes the KEDA +story land. Keep a terminal ready to run one *short* thing live if the room is engaged. + +**Mock mode is the right choice for the recording** and should be stated on the slide. +It costs no tokens, is deterministic, and is auto-selected when no credential is present. +The one thing it cannot show is *retained context* across a scale-to-zero (stage 2c), so +either accept that or record that one beat separately against a funded key. + +### 5.2 Running order + +| Min | Section | Notes | +|---|---|---| +| 0–2 | The vendor-platform framing | "You probably don't need one." Land the thesis fast | +| 2–5 | The composition: KEDA + Strimzi/Kafka + CloudEvents + JWS | One architecture diagram. Say **Strimzi, not Knative** | +| 5–9 | The seam taxonomy — all four, named | The artifact people take home. One slide, four rows | +| 9–13 | Seam 1 + 4: the wake contract and bound audit | Signed CloudEvents, `ce_causationid`. Lead with the lag insight (§2.3) | +| 13–17 | Seam 2 + 3: scope drift and delivery | Where the egress block and the keyset live. Name what they do **not** prove | +| 17–21 | **Demo** (recorded, narrated) | D1 → D2 → D3 → D4 → D5. Hard-stop at 4 min | +| 21–25 | **§8: thirteen claims that did not hold** | The differentiator. See below | +| 25–27 | What's next, honestly | SPIRE upgrade path + obstacles; Phase 3 tenancy as design | +| 27–30 | Questions | | + +### 5.3 The §8 section is the differentiator — give it 3–4 minutes + +`DESIGN_PHASE2.md` §8 — thirteen documented claims that do not hold in code, each with a +filed issue, plus the five-question checklist that came out of them — is the most valuable +artifact in this repository, and it is more on-theme than it first appears. + +The abstract promises "a named, demo-backed taxonomy of where those compositions leak." +§8.4 names *why* they leak: + +> **"each claim was written from the change that introduced it, and stayed true only for +> the configuration that change was exercised in."** + +That is a seam taxonomy for security *documentation*, and nobody else at the conference +will be presenting one. The five questions (§8.5) compress to one slide: + +1. Which code path, by name? +2. What is the default? +3. What happens when it is half-configured? +4. Can an attacker choose the input the check reads? +5. What does the claim become one release from now? + +Plus the two rules from §8.6 that generalize best: **call the real function** (a +reimplementation of the guard proves nothing about the guard), and **assert the outcome, +not the verdict**. + +**This section also inoculates you.** Saying the thirteen out loud turns the open bugs +from something a sharp attendee catches into evidence of rigor. Go first. + +### 5.4 Slide-by-slide sketch + +| # | Slide | Content | +|---|---|---| +| 1 | Title | | +| 2 | "Every vendor is shipping an agent platform" | The thesis | +| 3 | The parts you already run | KEDA, Strimzi/Kafka, CloudEvents, SPIFFE/SPIRE *(as upgrade path)* | +| 4 | Architecture | HTTP → EventBridge → Kafka:requests → EventRunner → agent → Kafka:responses. KEDA on lag | +| 5 | **The four seams** | The take-home artifact | +| 6 | Seam 4 first: unbound audit | `ce_causationid`. Short — it's solved | +| 7 | **Lag means "not finished," not "not read"** | The best result. The 300 s/429 anecdote | +| 8 | The wake contract | Detached JWS, Ed25519, signed attribute set | +| 9 | **Canonicalization must be injective** | The newline collision. Concrete, memorable, genuinely instructive | +| 10 | What a signature proves — and does not | Terminals only; proves who *finished*, not what it *said*. Keyset ≠ attestation | +| 11 | Trust-scope drift | The egress block, and why a deploy-time allowlist *is* the seam | +| 12 | Demo (recorded) | Mock mode, stated on-slide | +| 13 | **Thirteen claims that did not hold** | | +| 14 | The five questions | | +| 15 | "Call the real function. Assert the outcome." | | +| 16 | What's next | SPIRE upgrade + the EdDSA/JWKS obstacles; Phase 3 tenancy as design | +| 17 | Links | Repo, `agentdocs/`, the five issues | + +### 5.5 Things to say out loud, verbatim + +Lifted from the code's own docstrings, which are better than a paraphrase: + +- *"What it does not do is attest anything. The set records which keys an operator + approved, not which workloads a platform vouched for."* (`keyset.py`) +- *"This proves who finished a run, not what it said along the way."* (§4.4 step 2) +- *"A security demo that oversells its guarantee is worse than one that does not exist."* + (Phase 2 §7) + +And one framing worth preparing: **"compromising EventBridge means being able to claim +any user."** That is standard PEP design and the alternative is worse (spreading a live +GitHub credential across every runner), but somebody will ask. Phase 2 §2.7 has the +answer ready. + +### 5.6 Likely questions, with answers that exist + +| Question | Where the answer is | +|---|---| +| Why not SPIRE already? | §3 above; §8.3's EdDSA/JWKS obstacles | +| Why not a JWT library / Keycloak? | Phase 2 §2.1 (opaque token), §4.3 (Keycloak proves holding a secret, which a compromised pod also holds) | +| Why hand-rolled Ed25519? | Phase 1 §1.1; Phase 2 §4.3. Note the `_scalar_mult` timing caveat before someone else does | +| Why not HMAC — it's 170,000× faster? | Phase 2 §4.3: symmetric, so EventBridge could forge any agent's response | +| Kafka ACLs? | Phase 2 §3.3: the authorizer is global, not per-listener, so the demo is either broken or vacuous | +| Multi-tenancy? | `DESIGN_PHASE3.md` §2–§3, **design only** | +| What's the signing overhead? | ~150–200 ms/op. Get the on-cluster number in W2 | + +--- + +## 6. Risks + +| Risk | Mitigation | +|---|---| +| D3 (egress block) never gets built | W3 is the smallest honest version; if it slips, **cut the beat from the demo list and say so**, rather than narrating something that didn't happen | +| A reviewer finds #888 from the slides | Fix in W1; present it in the §8 section regardless | +| Live demo fails | Pre-record (§5.1) | +| Real-agent path still budget-blocked | Mock mode for the recording; state it on-slide | +| SPIRE question dominates Q&A | Prepare §3's two options as a 30-second answer | +| Talk runs long | The §8 section is the one thing not to cut. Compress slides 6–8 | +| Kind vs cluster timings get mixed | Pick one for the recorded numbers and say which (§4 sequencing) | + +--- + +## 7. Observations worth carrying forward + +Five things from the 2026-10-05 review that are not obvious from any single document: + +1. **The docs are more rigorous than the code, and that is unusual and valuable.** §8 + exists because somebody restated every claim for a reader who could not see the code, + then checked each by running it. That process is the talk's best content. + +2. **Reading the code is not enough, and neither is the test suite.** All thirteen + findings coexisted with a passing suite, because the tests exercised the configuration + each claim was true in. 575 passing tests did not catch #885 or #888. + +3. **My own first attempt at the group-forgery test reported no issue** — because I used + wrong event-type constants (`io.rossoctl.…` instead of `dev.rossoctl.agent.group. + completed.v1`). The finding only appeared once I used `ce.TYPE_GROUP_COMPLETED`. A + negative result from a hand-written harness is worth very little until you have proved + the harness can produce a positive one. Re-verify E before relying on it. + +4. **Half-configured is where everything breaks.** §8.5.3 is the highest-yield of the + five questions. A control with two variables has four cases, and the three that are + not "both set" are the ones a reader hits. The worst finding (#888) is exactly that. + +5. **Attribution vs restriction is the subtler keyset point** (§8.3) and is under-drawn + in the design: a flat keyset tells you *which approved key* signed, not *that this + particular agent was permitted to answer this particular request*. Phase 2 draws this + conclusion for group events and pins them; it does not for member answers. If anyone + in the room works on authorization, this is the question they will ask. + +### Caveats on the review this document is based on + +The code was read and the decision functions were run, but there was **no cluster +access** — everything stated about cluster behaviour comes from `IMPLEMENTATION_REPORT1.md` +rather than from observation. Cluster-dependent claims in §2 should be re-confirmed in W2. + +--- + +## 8. Change log + +| Date | Change | +|---|---| +| 2026-10-05 | Created. Captures the accepted abstract, status against it as of `main`, the SPIRE decision, the W1–W6 plan, and the 30-minute slide/demo shape | diff --git a/eventing/agentdocs/README.md b/eventing/agentdocs/README.md index 34d02306..2a72c072 100644 --- a/eventing/agentdocs/README.md +++ b/eventing/agentdocs/README.md @@ -19,6 +19,7 @@ and what is *not* verified. | [`IMPLEMENTATION_REPORT1.md`](IMPLEMENTATION_REPORT1.md) | What Phase 1 built, the measured results on both clusters, 28 findings including two real bugs only a scale-to-zero deployment could expose, and an honest account of what is still blocked and why. | | [`README_PHASE1.md`](README_PHASE1.md) | The Phase 1 runbook in full detail, covering all four ways to run it: locally without containers, under Docker, on a Kind cluster, and on OpenShift. | | [`DESIGN_PHASE2.md`](DESIGN_PHASE2.md) | The Phase 2 design — identity on the event path. Why GitHub's opaque user token rules out local verification and forces a `GET /user` lookup plus a load-bearing cache; why `401` and `403` are kept distinct; what `ce_submitter` is and is not worth, and what a signature over it does and does not prove; how the `kid` and the approved-key set prove *which agent* answered, why group events are pinned to EventBridge's own key, and why a rejected response is stored as an error rather than dropped. **§8 records the thirteen claims across §§2.6, 3, 4.3 and 4.4 that a docs review found do not hold in the code, with the issue filed for each — and §8.5-§8.8 the procedure that came out of them: the five questions to ask of any control claim, in any phase, before it ships.** | +| [`KUBECON_NA_2026.md`](KUBECON_NA_2026.md) | **The KubeCon NA knowledge base.** The accepted abstract and the five demo beats it commits to; the status of each against the code as of 2026-10-05, including what was confirmed by *running* the decision functions rather than reading them; the SPIRE decision that shapes everything else (two of the four seams name it and there is no SPIRE); the W1-W6 plan and the **M1-M4 MVP** cut of it; and the 30-minute slide and demo shape, with the recommendation to pre-record. Cites the other documents rather than restating them. | | [`DESIGN_PHASE3.md`](DESIGN_PHASE3.md) | The Phase 3 design — per-user isolation, declarative agents, and event triggers. Why a raw `submitter` cannot be a tenancy key and why the derived `userkey` ends in a hash; per-user Kafka topics (`{prefix}-{userkey}-requests`), one EventRunner Deployment and ScaledObject per user, and the honest difference between separately-named topics and ACL-enforced isolation; why a per-user ntfy topic must *not* contain the user's name, and the option of running ntfy as a service in the cluster; the `AgentSpec` — an agent baked into the image or assembled from skills fetched from GitHub, S3 or a URL, and the five gates a fetched bundle passes; authenticating and scoping the session transcript; and a Knative-inspired trigger API with the four loop controls that keep an event-driven agent system from becoming a cost incident. **Partly implemented: §9's steps 1-4 (T1, T2, T3, T4, T13, T22) plus T5 (per-user stores and the global ownership index) are in the tree; `EB_TENANCY_MODE` still defaults to `single`, which reproduces Phase 2 exactly. Owner-scoped reads (T7), transcript auth (T8), `k8s_tenant.py` (T12), ntfy isolation, triggers, fetched skills and Kafka ACLs are design only — so `multi` mode routes each tenant to its own store and topics, but does not yet check that a reader is the owner.** | | [`IMPLEMENTATION_REPORT3.md`](IMPLEMENTATION_REPORT3.md) | What Phase 3 has built so far (§9 steps 1–4 plus T5), measured rather than estimated: the per-tenant file-descriptor cost against §6.1's estimate, the ownership index's per-operation cost, and the 270 ms that settles §2.6's objection to seeding. Seven deltas from the design recorded as decisions, and the findings with their root causes — including a `single`-mode regression that shipped behind a headline claiming it could not happen, an eviction contract whose first fix addressed only the visible case, and `except Exception` converting three separate programming errors into log lines. **Partial: it will be extended as later steps land rather than rewritten.** | @@ -44,6 +45,10 @@ and what is *not* verified. - **Presenting it?** [`DESIGN_PHASE2.md`](DESIGN_PHASE2.md) §2.6 and §3, for what the identity controls do *not* prove — and §8, because four of them promise more than the code delivers. +- **Working on the KubeCon talk?** [`KUBECON_NA_2026.md`](KUBECON_NA_2026.md) — §2 for + what is and is not backed by code, §3 for the SPIRE decision that has to be made + first, §5 for the 30-minute shape. §7 records what the review that produced it found + that no single document states. - **Working on multi-tenancy, agent definitions or triggers?** [`DESIGN_PHASE3.md`](DESIGN_PHASE3.md) — §9 for the rollout order (the first four steps are refactors; everything risky is behind a flag that defaults off), §3.6 for what isolation does From 3468bea68d7bee31173c6339a234fd661ed8abe7 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Mon, 5 Oct 2026 19:31:50 -0400 Subject: [PATCH 2/4] =?UTF-8?q?docs(eventing):=20KubeCon=20NA=20=E2=80=94?= =?UTF-8?q?=20add=20the=20MVP=20cut,=20and=20correct=20the=20egress=20MVP?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes to KUBECON_NA_2026.md. **A new section 5: the M1-M4 MVP.** Section 4 is the full plan and stays as the record of why each item matters; section 5 is the subset that makes the talk honest and demo-backed, which is a smaller target than finishing Phase 2. Four items, roughly two days: fix #888 then #885, run kind-signed end to end, substitute the egress beat, and clear #889 plus the two stale doc claims. It cuts SPIRE and Phase 3 explicitly, defers #887 to an operational mitigation, and states the talk it buys -- three seams delivered and demo-backed, one named as the upgrade path with its obstacles. M2 is marked as the item that must not slip, because it is what converts "we wrote a verifier" into "we ran one". Section 2.6 is the argument: 575 passing tests did not catch #885 or #888, and section 4.4 step 2 records that the one comparable bug was found against a live broker and by no unit test. **A correction to W3.** The first version recommended a NetworkPolicy as the no-SPIRE MVP for the egress beat. That was written without checking the CNI. k8s/kind/kind-cluster.yaml creates a default Kind cluster, and kindnetd accepts NetworkPolicy objects while enforcing none of them -- so the demo would have shown a policy that appeared to work and blocked nothing. Three options are tabulated instead (swap the CNI, record on ykt1, or cut the beat), and M3 takes the third: substitute the #888 asymmetry, which is a real measured trust-scope failure needing no new infrastructure. Recorded in section 8 as observation 6, because it is the same shape as the thirteen -- a control claim true of the configuration imagined rather than the one that exists -- and because this document cited section 8.5.1 while breaking it. "Which code path, by name?" has to include which CNI, which broker, which cluster. The caveat note also marks the CNI finding as read rather than run, by the same rule. Renumbering: the 30-minute shape moves to section 6, risks to 7, observations to 8, change log to 9. Internal cross-references and the index entry follow. Rebased onto #883. No code, no manifests, no tests. Assisted-By: Claude Code Signed-off-by: Mariusz Sabath --- eventing/agentdocs/KUBECON_NA_2026.md | 229 +++++++++++++++++++++++--- 1 file changed, 205 insertions(+), 24 deletions(-) diff --git a/eventing/agentdocs/KUBECON_NA_2026.md b/eventing/agentdocs/KUBECON_NA_2026.md index c1bacf93..dcb3cf01 100644 --- a/eventing/agentdocs/KUBECON_NA_2026.md +++ b/eventing/agentdocs/KUBECON_NA_2026.md @@ -5,6 +5,9 @@ Scope: everything needed to deliver the talk — the abstract as submitted, what committed to show, the gap between that and the code, the plan to close it, and the slide/demo shape that fits 30 minutes. +**If you are picking this up to do the work:** §5 is the MVP — four items, ~two days, +and the honest talk they buy. §4 is the full plan it was cut from. + This file is the reference other analyses should cite. It is deliberately specific about what is **not** done, because the talk's own thesis is that unchecked control claims are where compositions leak — and `DESIGN_PHASE2.md` §8 is the record of this @@ -255,6 +258,11 @@ visual demo beat. See W3 below. Ordered by ratio of talk-value to effort. W1–W3 are the ones that change what you can honestly say on stage; W4–W6 are polish. +**This is the full plan, kept for the reasoning.** The subset actually proposed for the +talk is **§5's M1-M4**, which cuts W6 entirely, defers half of W4, and replaces W3 with a +substitution rather than a build. Read this section for *why* each item matters; read §5 +for what to do. + ### W1 — Fix #885 and #888 (highest value, smallest change) These are the difference between a security demo and security theater, and a sharp @@ -292,13 +300,29 @@ repository's own precedent says unit tests do not catch this class of bug. Exit criteria: one command drives zero → signed wake → verified completion → forged event refused → zero, with assertions and timings. -### W3 — Build the egress block (D3) +### W3 — The egress block (D3) — and why the obvious MVP does not work The only promised demo beat with zero implementation, and the most visceral one. -Minimum viable, no SPIRE needed: a `NetworkPolicy` on the EventRunner namespace with a -DNS + broker + model-endpoint egress allowlist, then have the demo agent attempt a -call outside it and show the block. +**The obvious MVP does not work, and the way it fails is on-theme.** The first version of +this section said: add a `NetworkPolicy` to the EventRunner namespace with a DNS + broker ++ model-endpoint egress allowlist, no SPIRE needed. That recommendation was made without +checking the CNI. `k8s/kind/kind-cluster.yaml` creates a default Kind cluster, and Kind's +default CNI is **kindnetd, which does not implement NetworkPolicy**: the object is +accepted by the API server, `kubectl get networkpolicy` shows it, and **nothing is +enforced**. The demo would have shown a policy that appeared to work and blocked nothing +— which is precisely the class of failure this talk is about, reached by skipping §8.5.1 +("which code path, by name?") on a piece of infrastructure rather than on code. + +So the real options are: + +| Option | What it costs | Verdict | +|---|---|---| +| Swap Kind's CNI for Calico or Cilium | `disableDefaultCNI: true`, install a CNI, then re-verify the whole Phase 1 stack on it — Strimzi, KEDA, ingress-nginx, the scale-to-zero timings | ❌ Too much risk this close to the talk, for one demo beat | +| Demo the egress block on **OpenShift** (ykt1), which enforces NetworkPolicy natively | A second cluster in the demo, on VPN, and §"Measured" already warns Kind and cluster timings are not comparable | 🟡 Viable as a **separately recorded clip**, labelled as a different cluster | +| **Cut D3 and substitute the #888 asymmetry** | A few hours, no new infrastructure | ✅ **Recommended — see M3** | + +If the beat is built at all, two things still apply: - **Make the failure legible.** A `NetworkPolicy` denial is a connection timeout, which on stage is indistinguishable from a slow network. Either surface the agent's own error @@ -308,11 +332,13 @@ call outside it and show the block. "trust-scope drift" seam the abstract says is the *problem*. Demoing the static version and naming the drift is honest and still makes the point. Claiming it is scope-aware at wake time would not be. -- If Option B lands later, the same beat upgrades to sidecar-reading-SVID without - changing the demo script. -Exit criteria: a wrong-scope call is blocked, visibly, within a few seconds, in under -60 s of demo time. +If Option B (§3) lands later, the same beat upgrades to sidecar-reading-SVID without +changing the demo script. + +Exit criteria, if built: a wrong-scope call is blocked, visibly, within a few seconds, in +under 60 s of demo time, **on a CNI that enforces policy** — verified by checking that a +deliberately-denied call actually fails before relying on it. ### W4 — Fix #887 and #889 @@ -354,13 +380,150 @@ pick one for the recorded numbers and say which. --- -## 5. The 30-minute shape +## 5. The MVP cut: M1-M4 + +§4 is the full plan. This is the subset that makes the talk **honest and demo-backed**, +which is a smaller target than finishing Phase 2. Roughly **two days of focused work.** + +The line the MVP buys: + +> **Three seams delivered and demo-backed. One named as the upgrade path, with its +> obstacles.** + +That is achievable. Four seams delivered is not, and attempting it is what puts the talk +at risk. + +### M1 — Fix #888, then #885 *(~0.5 day)* + +Both call sites were read; both are small and localised. + +**#888** — `kafka_in.py:93`. The decision is unreachable without a keyset: + +```python +ok, why = True, "not checked" +if self._keyset is not None: # <- `require` is never consulted +``` + +Fix: when `require_signature` is on and `keyset is None`, fail closed. **The startup +banner matters as much as the behaviour** — `__main__.py:76` prints +`response verification OFF` in exactly that configuration, so an operator who asked for +enforcement is told it is off and refuses nothing. The silent wrong state is the bug; +the missing refusal is only half of it. + +**#885** — `kafka_in.py:136` routes on `is_group_event(evt)`, the **original** event, so +the rejection rewrite reaches `on_group_event` → `complete_group()`. The rewrite +deliberately preserves `type` and `groupid` for forensics (`insert_response` derives both +`data_json` and `raw_json` from the one dict), so the fix is not to strip them: it is to +not route a **rejected** event to the group handler at all. The forensic record already +lives under `data["rejected"]`, so nothing is lost. + +Tests must **assert the outcome, not the verdict** — §8.6's rule. +`response_decision` returning `False` is not the same as the batch not completing, and +that gap *is* #885. + +**Out of scope for the MVP:** verifying the group mirror's replay path. It is the right +fix, but it changes restart behaviour for existing unsigned topic data, so it needs a +flag — and it does not appear in the demo. + +Exit criteria: a forged `group.completed` with enforcement on does **not** complete the +batch and does **not** notify; enforcement with no keyset refuses and the banner says +why. + +### M2 — Run `kind-signed` end to end *(~1 day, depends on M1)* + +**The item that must not slip.** It is what converts "we wrote a verifier" into "we ran +one", and §2.6 is the argument: there is currently no evidence the signed path survives a +real broker, 575 passing tests did not catch #885 or #888, and §4.4 step 2 records that +the one comparable bug was found against a live broker and by no unit test. + +- Add `kind-signed` to `k8s_demo_flow.py --overlay`, which today accepts only + `("test", "kind", "demo")` and therefore **cannot drive the signed overlay at all**. +- Two new stages: **a signed wake that verifies**, and **a forged event that is refused**. + The second is the money shot and has no script today. +- Capture the on-cluster signing cost. ~150-200 ms/op in pure Python has to be inside the + wake-latency number on the slide, not omitted from it. +- Update `IMPLEMENTATION_REPORT1.md` §8, which still says `ER_REQUIRE_SIGNATURE=true` has + no end-to-end run. + +Exit criteria: one command drives zero → signed wake → verified completion → forged event +refused → zero, with assertions and timings. + +### M3 — Substitute the egress beat *(a decision, not a build)* + +Per §4/W3, the `NetworkPolicy` MVP does not enforce on Kind's default CNI. Rather than +change CNI or add a second cluster, **cut D3 and substitute the #888 asymmetry**: + +> Enforcement on, no keyset → refuses **nothing**, banner says `OFF`. +> The request side, no key → refuses **everything**. +> Both were documented as "refusal is on". One was true. + +This is a real, reproducible, measured trust-scope failure, it needs no new +infrastructure, and it fits the talk's thesis better than a static allowlist would have — +a deploy-time allowlist is the seam the abstract calls a *problem*, so demoing it +straight always required an apology. Demo the fix landing. + +If D3 is wanted for real, record it on ykt1 as a separate clip and say on-slide that it +is a different cluster. + +### M4 — #889 and the honesty pass *(~2 hours, parallel)* + +- **#889** — a stale pidfile naming PID 1 crash-loops the pod after an abrupt exit. Low + glamour, high demo risk: this is the bug that bites when a pod is restarted between + rehearsal and the live run. +- **Docs:** correct `keyset.py`'s "the upgrade path changes no verification logic" claim + (false per §8.3 — EdDSA-only against SPIRE's EC/RSA, and `keyset.load` rejects JWKS), + and `DESIGN_PHASE2.md` §6's `490/5` test count. Small, but in a repository whose §8 is + about claims that quietly stopped being true, a test count is the cheapest one to keep + honest. + +**Deferred: #887.** Widest blast radius in principle — an unauthenticated +`PUT /transcript` changes what the agent resumes from — but the HMAC capability-key fix is +real design work and invisible on stage. Mitigate operationally for the conference +instead: do not expose the Route, or set `EB_AUTH_TOKENS`. + +### What the MVP cuts, explicitly + +| Cut | Why | +|---|---| +| **SPIRE** (§3 Option B) | Needs an issuer-agnostic verifier — EC/RSA added to hand-rolled pure-Python crypto, or relaxing §1.1 for `cryptography`. Dominant cost of the whole plan | +| **D3 as a live beat** | kindnetd does not enforce NetworkPolicy (§4/W3) | +| **Phase 3 tenancy** | Partly implemented, but owner-scoped reads are not. Good "what's next" material; do not demo it | +| **Group-mirror replay verification** | Correct, but needs a flag and is not on stage | +| **#887 code fix** | Mitigate operationally | + +### Sequencing + +``` +M1 (0.5d) ──▶ M2 (1d) ──▶ record the demo +M4 (2h, parallel) +M3 = a decision, not a build +``` + +M1 → M2 is strictly ordered: M2 is not worth running until M1's fixes are in. Rehearse on +Kind, and note that Kind sets `group.initial.rebalance.delay.ms: 0` where the shared +cluster leaves it at 3000 ms — so pick one for the recorded numbers and say which. + +### The talk the MVP supports + +| Seam | After M1-M4 | +|---|---| +| Unbound audit | ✅ closed — `ce_causationid` | +| Unverified delivery | ✅ closed — signed, enforced, **and run on a cluster** | +| Cold-start identity gap | ✅ demoed as the #888 asymmetry, named honestly | +| Trust-scope drift | 🟡 named, with the deploy-time-allowlist limit stated and §8.3's SPIRE obstacles given | + +Plus the §8 section (§6.3 below), which is the differentiator and needs no engineering at +all. + +--- + +## 6. The 30-minute shape 30 minutes is roughly **22–24 minutes of content, 4 minutes of demo, 3–4 minutes of questions.** The demo is the constraint: D1–D5 as written cannot be driven live in four minutes, because POST → pod Ready alone is ~14 s and a full cycle to zero is 22 s. -### 5.1 Demo strategy: pre-record, narrate live +### 6.1 Demo strategy: pre-record, narrate live **Record the demo. Do not run it live.** Reasons, in order of how much they would hurt: @@ -383,7 +546,7 @@ It costs no tokens, is deterministic, and is auto-selected when no credential is The one thing it cannot show is *retained context* across a scale-to-zero (stage 2c), so either accept that or record that one beat separately against a funded key. -### 5.2 Running order +### 6.2 Running order | Min | Section | Notes | |---|---|---| @@ -397,7 +560,7 @@ either accept that or record that one beat separately against a funded key. | 25–27 | What's next, honestly | SPIRE upgrade path + obstacles; Phase 3 tenancy as design | | 27–30 | Questions | | -### 5.3 The §8 section is the differentiator — give it 3–4 minutes +### 6.3 The §8 section is the differentiator — give it 3–4 minutes `DESIGN_PHASE2.md` §8 — thirteen documented claims that do not hold in code, each with a filed issue, plus the five-question checklist that came out of them — is the most valuable @@ -425,7 +588,7 @@ not the verdict**. **This section also inoculates you.** Saying the thirteen out loud turns the open bugs from something a sharp attendee catches into evidence of rigor. Go first. -### 5.4 Slide-by-slide sketch +### 6.4 Slide-by-slide sketch | # | Slide | Content | |---|---|---| @@ -447,7 +610,7 @@ from something a sharp attendee catches into evidence of rigor. Go first. | 16 | What's next | SPIRE upgrade + the EdDSA/JWKS obstacles; Phase 3 tenancy as design | | 17 | Links | Repo, `agentdocs/`, the five issues | -### 5.5 Things to say out loud, verbatim +### 6.5 Things to say out loud, verbatim Lifted from the code's own docstrings, which are better than a paraphrase: @@ -462,7 +625,7 @@ any user."** That is standard PEP design and the alternative is worse (spreading GitHub credential across every runner), but somebody will ask. Phase 2 §2.7 has the answer ready. -### 5.6 Likely questions, with answers that exist +### 6.6 Likely questions, with answers that exist | Question | Where the answer is | |---|---| @@ -472,17 +635,18 @@ answer ready. | Why not HMAC — it's 170,000× faster? | Phase 2 §4.3: symmetric, so EventBridge could forge any agent's response | | Kafka ACLs? | Phase 2 §3.3: the authorizer is global, not per-listener, so the demo is either broken or vacuous | | Multi-tenancy? | `DESIGN_PHASE3.md` §2–§3, **design only** | -| What's the signing overhead? | ~150–200 ms/op. Get the on-cluster number in W2 | +| What's the signing overhead? | ~150–200 ms/op. Get the on-cluster number in M2 | --- -## 6. Risks +## 7. Risks | Risk | Mitigation | |---|---| -| D3 (egress block) never gets built | W3 is the smallest honest version; if it slips, **cut the beat from the demo list and say so**, rather than narrating something that didn't happen | -| A reviewer finds #888 from the slides | Fix in W1; present it in the §8 section regardless | -| Live demo fails | Pre-record (§5.1) | +| D3 (egress block) cannot be built on Kind's default CNI | Resolved in the MVP: M3 cuts the beat and substitutes the #888 asymmetry. If D3 is wanted anyway, record it on ykt1 as a separate, separately-labelled clip (§4/W3) | +| Something else promised in the demo list goes unbuilt | **Cut the beat and say so**, rather than narrating something that did not happen. §1.1's D1-D5 numbering exists so a cut is explicit rather than silent | +| A reviewer finds #888 from the slides | Fix in M1; present it in the §8 section regardless — and after M3 it *is* a demo beat | +| Live demo fails | Pre-record (§6.1) | | Real-agent path still budget-blocked | Mock mode for the recording; state it on-slide | | SPIRE question dominates Q&A | Prepare §3's two options as a 30-second answer | | Talk runs long | The §8 section is the one thing not to cut. Compress slides 6–8 | @@ -490,9 +654,9 @@ answer ready. --- -## 7. Observations worth carrying forward +## 8. Observations worth carrying forward -Five things from the 2026-10-05 review that are not obvious from any single document: +Six things from the 2026-10-05 review that are not obvious from any single document: 1. **The docs are more rigorous than the code, and that is unusual and valuable.** §8 exists because somebody restated every claim for a reader who could not see the code, @@ -518,16 +682,33 @@ Five things from the 2026-10-05 review that are not obvious from any single docu conclusion for group events and pins them; it does not for member answers. If anyone in the room works on authorization, this is the question they will ask. +6. **§8.5.1 applies to infrastructure, not just to code — and this document broke the + rule while citing it.** The first version of §4/W3 recommended a `NetworkPolicy` as + the no-SPIRE MVP for the egress beat, written without checking which CNI the Kind + config creates. It is kindnetd, which accepts NetworkPolicy objects and enforces + none of them, so the recommendation would have produced a demo that appeared to work + and blocked nothing — the exact failure the talk is about, and the same shape as the + thirteen: a control claim that was true of the configuration imagined rather than the + one that exists. "Which code path, by name?" has to include "on which CNI, which + broker, which cluster". + ### Caveats on the review this document is based on The code was read and the decision functions were run, but there was **no cluster access** — everything stated about cluster behaviour comes from `IMPLEMENTATION_REPORT1.md` -rather than from observation. Cluster-dependent claims in §2 should be re-confirmed in W2. +rather than from observation. Cluster-dependent claims in §2 should be re-confirmed in M2. + +The CNI finding in §8.6 was reached by reading `k8s/kind/kind-cluster.yaml`, not by +applying a NetworkPolicy to a running Kind cluster and watching it fail to block. That +is the right direction for the conclusion — kindnetd's lack of NetworkPolicy support is +documented upstream — but by this document's own §8.6 rule it is still a claim that has +been read rather than run. Confirm it before spending anything on either CNI option. --- -## 8. Change log +## 9. Change log | Date | Change | |---|---| | 2026-10-05 | Created. Captures the accepted abstract, status against it as of `main`, the SPIRE decision, the W1–W6 plan, and the 30-minute slide/demo shape | +| 2026-10-05 | Added §5, the M1-M4 MVP cut (~2 days) with sequencing, explicit cuts and the talk it supports. Corrected §4/W3: Kind's default CNI is kindnetd and does not enforce NetworkPolicy, so the `NetworkPolicy` MVP was unbuildable as written — three options tabulated, M3 recommends substituting the #888 asymmetry for the egress beat. §7's risk table and §8's observations updated; the 30-minute shape renumbered §5→§6, risks §6→§7, observations §7→§8 | From ea4f7f54d04931e4ae8b6b5dfdb3cecf1726fc68 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Tue, 6 Oct 2026 10:02:55 -0400 Subject: [PATCH 3/4] =?UTF-8?q?docs(eventing):=20KubeCon=20NA=20=E2=80=94?= =?UTF-8?q?=20link=20M1=20and=20M2=20to=20their=20issues?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M1 and M2 are filed as #895 and #896. Section 5 gains a tracking table so the MVP's status is visible without reading all four subsections, and the two subsections that now have issues link to them. M3 has no issue by design -- it is a decision about what to demo, not work to schedule. M4 tracks against the existing #889. Assisted-By: Claude Code Signed-off-by: Mariusz Sabath --- eventing/agentdocs/KUBECON_NA_2026.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/eventing/agentdocs/KUBECON_NA_2026.md b/eventing/agentdocs/KUBECON_NA_2026.md index dcb3cf01..2cb375ee 100644 --- a/eventing/agentdocs/KUBECON_NA_2026.md +++ b/eventing/agentdocs/KUBECON_NA_2026.md @@ -393,7 +393,16 @@ The line the MVP buys: That is achievable. Four seams delivered is not, and attempting it is what puts the talk at risk. -### M1 — Fix #888, then #885 *(~0.5 day)* +| | Item | Effort | Tracked | Blocks | +|---|---|---|---|---| +| **M1** | Fix #888, then #885 | ~0.5 d | [#895](https://github.com/rossoctl/examples/issues/895) | M2 | +| **M2** | Run `kind-signed` end to end | ~1 d | [#896](https://github.com/rossoctl/examples/issues/896) | the recording | +| **M3** | Substitute the egress beat | a decision | — | the demo list | +| **M4** | #889 + the honesty pass | ~2 h | #889 | — | + +M2 is the one that must not slip. M3 is a call to make, not work to schedule. + +### M1 — Fix #888, then #885 *(~0.5 day)* — [#895](https://github.com/rossoctl/examples/issues/895) Both call sites were read; both are small and localised. @@ -429,7 +438,7 @@ Exit criteria: a forged `group.completed` with enforcement on does **not** compl batch and does **not** notify; enforcement with no keyset refuses and the banner says why. -### M2 — Run `kind-signed` end to end *(~1 day, depends on M1)* +### M2 — Run `kind-signed` end to end *(~1 day, depends on M1)* — [#896](https://github.com/rossoctl/examples/issues/896) **The item that must not slip.** It is what converts "we wrote a verifier" into "we ran one", and §2.6 is the argument: there is currently no evidence the signed path survives a @@ -711,4 +720,5 @@ been read rather than run. Confirm it before spending anything on either CNI opt | Date | Change | |---|---| | 2026-10-05 | Created. Captures the accepted abstract, status against it as of `main`, the SPIRE decision, the W1–W6 plan, and the 30-minute slide/demo shape | +| 2026-10-06 | M1 and M2 filed as #895 and #896; §5 gains a tracking table | | 2026-10-05 | Added §5, the M1-M4 MVP cut (~2 days) with sequencing, explicit cuts and the talk it supports. Corrected §4/W3: Kind's default CNI is kindnetd and does not enforce NetworkPolicy, so the `NetworkPolicy` MVP was unbuildable as written — three options tabulated, M3 recommends substituting the #888 asymmetry for the egress beat. §7's risk table and §8's observations updated; the 30-minute shape renumbered §5→§6, risks §6→§7, observations §7→§8 | From 8846e9f3fb2d84c67dc6451431e70d63726d51e9 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Tue, 6 Oct 2026 10:31:16 -0400 Subject: [PATCH 4/4] =?UTF-8?q?docs(eventing):=20KubeCon=20NA=20=E2=80=94?= =?UTF-8?q?=20re-derive=20the=20figures=20and=20citations=20against=20the?= =?UTF-8?q?=20base?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All eight findings from @aslom's #893 review. Every figure in this file was either re-measured or traced to the document that records it; none was carried over from the draft. **The code citations were derived against the pre-#883 tree.** `kafka_in.py:93`, `:94`, `:122`, `:136` and `__main__.py:76` all match `3264e8d`, the commit before #883 — this PR's base is `0667270`. Re-derived at the base: 109/110, 138, 152 and 80. The findings hold at the new lines; only the pointers were wrong, and `:93` landed inside a comment block. §5/M1 now states the commit the pointers are against and says to re-derive rather than adjust them after a rebase. **The test count was stale and the replacement is environment-dependent.** 575/5 was also a pre-#883 number. Running the suite at this base gives **899 passed, 5 skipped** on a laptop, where CI reports 896/8 — the three extra CI skips are `test_agentspec.py`'s pinned-CLI check (needs `claude` on `PATH`) and `test_graceful_shutdown.py` (needs a reachable Kafka); the five constant ones are `test_manifests.py`'s `no reachable cluster` cases. The document now quotes the collected count, 904, plus the environment, since a bare pass figure is not reproducible. The two load-bearing uses ("575 passing tests did not catch #885 or #888") read ~900, which strengthens the argument. **Three cluster timings were unsourced.** `1.7–3.0 s`, `~14 s` and `22 s` appear in no other file in the repository, while §8's own caveat says cluster claims come from `IMPLEMENTATION_REPORT1.md`. Replaced with §3's measured figures — wake 1.0–5.0 s on ykt1, 2.2–4.3 s on Kind, 30–31 s back to zero at `cooldownPeriod: 30`, image pull 5.1 s cold / 454 ms–2.45 s warm — each attributed to its cluster, per §4's own note that the two are not comparable. §6's demo budget and §6.1's pull figure follow. The superseded numbers are named so they do not reappear. **An estimate was quoted where a measurement exists.** `~150–200 ms` is the docstring estimate; `IMPLEMENTATION_REPORT1.md` §9 measured 222 ms / 227 ms. Re-running it here gave 190/198 ms median, so the band is ~190–230 ms and the hardware is the variable — worth saying, because the review framed the estimate as understating the measurement and on this machine it does not. All four sites now cite the measurement, and §6.6's prepared answer no longer reads as though none exists. M2 is framed as confirming on cluster hardware, not first contact. **The SPIFFE claim was scoped to the tree.** True of `eventing/` (8 mentions across 4 files, all prose); repo-wide it is ~95, because `mcp/wiki_memory_tool/` ships header-simulated SPIFFE identity as executable code. Scoped, with the sibling component named so the question is answered before an attendee greps. **The file prescribed `` markers it did not carry.** Added at the five places that expire when #885–#889 land: §2.4, §2.5, §5/M1, §5/M3 and §6.3. §5/M3 is the sharp one — it substitutes the #888 asymmetry as a demo beat, and §5/M1 fixes #888, so the marker says to reframe rather than delete. Moved from W5 (after the talk) into M4. **Overwriting `DESIGN_PHASE2.md` §6's test count would have broken it.** `490/5` is Phase 2's delta measurement — "adds 40 over a 450/5 baseline on the same tree" — not a live claim, and the `agentdocs` README says a phase record is superseded explicitly rather than edited in place. W5 and M4 now call for a footnote. Also: `§8.6` meant this document's §8 observation 6 in one sentence and Phase 2's §8.6 in the next; every cross-document reference is now qualified. The `agentdocs/README.md` row says this is a living document with a deadline, unlike the phase records around it. 899 passed, 5 skipped at this base, unchanged. Docs only. Assisted-By: Claude Code Signed-off-by: Mariusz Sabath --- eventing/agentdocs/KUBECON_NA_2026.md | 126 ++++++++++++++++++-------- eventing/agentdocs/README.md | 2 +- 2 files changed, 91 insertions(+), 37 deletions(-) diff --git a/eventing/agentdocs/KUBECON_NA_2026.md b/eventing/agentdocs/KUBECON_NA_2026.md index 2cb375ee..f0c22ff0 100644 --- a/eventing/agentdocs/KUBECON_NA_2026.md +++ b/eventing/agentdocs/KUBECON_NA_2026.md @@ -67,7 +67,7 @@ tracks each one separately. | D2 | signed task event → agent wake | 🟡 JWS real, never run on a cluster | | D3 | blocked egress on a wrong-scope call | ❌ **no implementation at all** | | D4 | completion bound to the wake event | ✅ `ce_causationid` | -| D5 | scale to zero when agents are idle | ✅ measured, 22 s full cycle | +| D5 | scale to zero when agents are idle | ✅ measured, 30–31 s back to zero with `cooldownPeriod: 30` | --- @@ -75,24 +75,34 @@ tracks each one separately. Assessed 2026-10-05 against `main`. Method: read all nine `agentdocs/` files and the full source, then **ran the identity decision functions** rather than trusting -docstrings — which is this repository's own §8.6 rule, and it changed two conclusions. +docstrings — which is `DESIGN_PHASE2.md` §8.6's rule, and it changed two conclusions. -Test suite at review time: **575 passed, 5 skipped**. (`DESIGN_PHASE2.md` §6 says -490/5 — the suite has grown; the doc is stale, not wrong.) +Test suite at review time: **904 collected**. On CI, 896 passed and 8 skipped; on a +laptop with the `claude` CLI on `PATH`, 899 passed and 5 skipped — the three extra CI +skips are `test_agentspec.py`'s pinned-CLI check plus `test_graceful_shutdown.py`, which +needs a reachable Kafka. The five constant skips are `test_manifests.py`'s +`no reachable cluster` cases. Quote the collected count and the environment, not a bare +pass figure. (`DESIGN_PHASE2.md` §6 says 490/5; that is Phase 2's own delta measurement, +not a live count — see §5/M4 before touching it.) ### 2.1 The four seams | Seam | Named CNCF integration | Code status | |---|---|---| -| Cold-start identity gap | SPIRE → KEDA | ❌ **No SPIRE/SPIFFE/SVID in the implementation.** 7 mentions total, all prose, all aspirational | +| Cold-start identity gap | SPIRE → KEDA | ❌ **No SPIRE/SPIFFE/SVID in `eventing/`.** 8 mentions across 4 files, all prose, all aspirational. (`mcp/wiki_memory_tool/` does ship header-simulated SPIFFE identity — a different component, and its own README notes there is no real SVID verification) | | Trust-scope drift | SPIRE → egress sidecar | ❌ **No NetworkPolicy, no Istio, no AuthorizationPolicy, no sidecar** | | Unverified delivery | CloudEvents verifier → Knative trigger | 🟡 Verification exists and is real; **no Knative** (Strimzi instead, which the abstract permits) | | Unbound audit | event bus → work record | 🟡 `ce_causationid` binds response to request; **the replay path verifies nothing** | ### 2.2 What is genuinely done and demo-ready -- **KEDA scale-to-zero, measured on two clusters.** OpenShift and Kind. POST → KEDA - decision 1.7–3.0 s; POST → pod Ready ~14 s; full cycle back to zero 22 s. +- **KEDA scale-to-zero, measured on two clusters.** OpenShift (ykt1) and Kind. The only + sourced figures are `IMPLEMENTATION_REPORT1.md` §3's: wake latency **1.0–5.0 s** on + ykt1 and **2.2–4.3 s** on Kind; scale back to zero **30–31 s** with + `cooldownPeriod: 30`; image pull **5.1 s** cold (245 MB), 454 ms–2.45 s warm. Earlier + drafts of this file quoted 1.7–3.0 s, ~14 s and 22 s, which appear in no other document + in the repository — do not put them on a slide. Per §4's own sequencing note, ykt1 and + Kind timings are not comparable, so say which cluster a number is from. - **Detached JWS over the CloudEvent envelope.** Ed25519 implemented from RFC 8032 in pure Python, checked against the RFC's own test vectors. - **The canonicalization is injective.** Length-prefixed (netstring-style) fields, after @@ -119,6 +129,11 @@ a challenge; the code has it. ### 2.4 Confirmed by running the code + + Not from docstrings. These five lines are the honest state of response verification: ``` @@ -130,12 +145,14 @@ E) forged group.completed, enforced -> refused by verifier… then applied anywa ``` **(D) is a deliberate policy, not a bug.** `emit()` signs terminal events only, because -it runs per `stdout` frame and Ed25519 costs ~150–200 ms here; signing every frame would +it runs per `stdout` frame and Ed25519 costs **222 ms to sign, 227 ms to verify** here +(`IMPLEMENTATION_REPORT1.md` §9, measured — not the ~150–200 ms the docstrings +estimate); signing every frame would add minutes to a chatty run. The verifier matches that policy. The honest limit: this proves *who finished a run*, not *what it said along the way*. Say that on stage. **(E) is the one that will bite on stage.** `signing.response_decision` correctly refuses -the forged `group.completed`. But `kafka_in.py:122` rewrites only `phase` and `data` — it +the forged `group.completed`. But `kafka_in.py:138` rewrites only `phase` and `data` — it preserves `type` and `groupid`, then routes on `is_group_event(evt)` using the **original** event. `group_service.on_group_event` reads only those two fields, so it calls `complete_group()` anyway. A forged event the verifier *correctly rejected* still @@ -147,6 +164,10 @@ replayable, but the replay trusts it unconditionally. ### 2.5 Open bugs, all still open + + From the `DESIGN_PHASE2.md` §8 docs review. None fixed as of 2026-10-05. | Issue | What | @@ -294,8 +315,11 @@ repository's own precedent says unit tests do not catch this class of bug. refused** — the latter is the money shot and currently has no script. - Record the result in `IMPLEMENTATION_REPORT1.md` §8, which still says `ER_REQUIRE_SIGNATURE=true` has no end-to-end run. -- Measure the signing cost on-cluster. Ed25519 here is ~150–200 ms/op in pure Python; - the wake-path latency number on the slides must include it. +- Confirm the signing cost on-cluster. `IMPLEMENTATION_REPORT1.md` §9 already measured + **222 ms / 227 ms** on a laptop — re-running it here gave 190/198 ms median, so treat + ~190–230 ms as the pure-Python band and the hardware as the variable. This is a + confirmation on cluster hardware, not a first measurement, and the wake-path latency + number on the slides must include it. Exit criteria: one command drives zero → signed wake → verified completion → forged event refused → zero, with assertions and timings. @@ -356,11 +380,14 @@ deliberately-denied call actually fails before relying on it. The talk will send people to this repository. Three fixes: - `keyset.py`'s "upgrade path changes no verification logic" claim — correct it per - §8.3, in place, since the module docstring is what a reader lands on. -- `DESIGN_PHASE2.md` §6's test count (490/5 → current). Small, but the talk's thesis is - about claims that quietly stopped being true. -- Add the `` comments §8.5.5 prescribes to any statement in this file that - depends on #885–#889 staying open. That is this document eating its own cooking. + `DESIGN_PHASE2.md` §8.3, in place, since the module docstring is what a reader lands on. +- A footnote on `DESIGN_PHASE2.md` §6's test count pointing at §2's current figure. + **Not** an overwrite: `490/5` is Phase 2's delta measurement and its "adds 40 over + 450/5" arithmetic depends on it. Small, but the talk's thesis is about claims that + quietly stopped being true, and a historical measurement never stopped being true. +- The `` comments `DESIGN_PHASE2.md` §8.5.5 prescribes are now in this + file (§2.4, §2.5, §5/M1, §5/M3, §6.3) as of M4 — W5 is after the talk, which was too + late for a document whose §5/M3 depends on #888 staying open. Re-check them here. ### W6 — Stretch, only if W1–W4 land early @@ -404,9 +431,13 @@ M2 is the one that must not slip. M3 is a call to make, not work to schedule. ### M1 — Fix #888, then #885 *(~0.5 day)* — [#895](https://github.com/rossoctl/examples/issues/895) -Both call sites were read; both are small and localised. +Both call sites were read; both are small and localised. Every `path:line` below is +against **`0667270`**, this document's base — the first draft cited `3264e8d`, the commit +before #883, and the rebase moved `kafka_in.py` by +16 lines and `__main__.py` by +4 +without the pointers being re-derived. Re-derive them, do not adjust them, after any +rebase. -**#888** — `kafka_in.py:93`. The decision is unreachable without a keyset: +**#888** — `kafka_in.py:109`. The decision is unreachable without a keyset: ```python ok, why = True, "not checked" @@ -414,19 +445,19 @@ if self._keyset is not None: # <- `require` is never consulted ``` Fix: when `require_signature` is on and `keyset is None`, fail closed. **The startup -banner matters as much as the behaviour** — `__main__.py:76` prints +banner matters as much as the behaviour** — `__main__.py:80` prints `response verification OFF` in exactly that configuration, so an operator who asked for enforcement is told it is off and refuses nothing. The silent wrong state is the bug; the missing refusal is only half of it. -**#885** — `kafka_in.py:136` routes on `is_group_event(evt)`, the **original** event, so +**#885** — `kafka_in.py:152` routes on `is_group_event(evt)`, the **original** event, so the rejection rewrite reaches `on_group_event` → `complete_group()`. The rewrite deliberately preserves `type` and `groupid` for forensics (`insert_response` derives both `data_json` and `raw_json` from the one dict), so the fix is not to strip them: it is to not route a **rejected** event to the group handler at all. The forensic record already lives under `data["rejected"]`, so nothing is lost. -Tests must **assert the outcome, not the verdict** — §8.6's rule. +Tests must **assert the outcome, not the verdict** — `DESIGN_PHASE2.md` §8.6's rule. `response_decision` returning `False` is not the same as the batch not completing, and that gap *is* #885. @@ -442,15 +473,16 @@ why. **The item that must not slip.** It is what converts "we wrote a verifier" into "we ran one", and §2.6 is the argument: there is currently no evidence the signed path survives a -real broker, 575 passing tests did not catch #885 or #888, and §4.4 step 2 records that +real broker, ~900 passing tests did not catch #885 or #888, and §4.4 step 2 records that the one comparable bug was found against a live broker and by no unit test. - Add `kind-signed` to `k8s_demo_flow.py --overlay`, which today accepts only `("test", "kind", "demo")` and therefore **cannot drive the signed overlay at all**. - Two new stages: **a signed wake that verifies**, and **a forged event that is refused**. The second is the money shot and has no script today. -- Capture the on-cluster signing cost. ~150-200 ms/op in pure Python has to be inside the - wake-latency number on the slide, not omitted from it. +- Capture the on-cluster signing cost. The measured laptop figure is 222 ms / 227 ms + (`IMPLEMENTATION_REPORT1.md` §9), ~190–230 ms across re-runs; whatever the cluster + gives has to be inside the wake-latency number on the slide, not omitted from it. - Update `IMPLEMENTATION_REPORT1.md` §8, which still says `ER_REQUIRE_SIGNATURE=true` has no end-to-end run. @@ -459,6 +491,12 @@ refused → zero, with assertions and timings. ### M3 — Substitute the egress beat *(a decision, not a build)* + + Per §4/W3, the `NetworkPolicy` MVP does not enforce on Kind's default CNI. Rather than change CNI or add a second cluster, **cut D3 and substitute the #888 asymmetry**: @@ -480,10 +518,17 @@ is a different cluster. glamour, high demo risk: this is the bug that bites when a pod is restarted between rehearsal and the live run. - **Docs:** correct `keyset.py`'s "the upgrade path changes no verification logic" claim - (false per §8.3 — EdDSA-only against SPIRE's EC/RSA, and `keyset.load` rejects JWKS), - and `DESIGN_PHASE2.md` §6's `490/5` test count. Small, but in a repository whose §8 is - about claims that quietly stopped being true, a test count is the cheapest one to keep - honest. + (false per §8.3 — EdDSA-only against SPIRE's EC/RSA, and `keyset.load` rejects JWKS). + **Leave `DESIGN_PHASE2.md` §6's `490/5` alone:** it is Phase 2's own delta measurement + — "adds 40 over a 450/5 baseline on the same tree" — not a live claim about the suite, + and overwriting it breaks that arithmetic. The `agentdocs` README says a phase record + is superseded explicitly rather than edited in place. Add a footnote pointing at §2's + current count instead. +- Add the `` comments `DESIGN_PHASE2.md` §8.5.5 prescribes to any statement + in this file that depends on #885–#889 staying open — §2.4's table, §2.5, §5/M1 and + §6.3's slide, and in particular **§5/M3**, whose substituted demo beat stops reproducing + once M1 lands. Naming the release, the issue and what should change is the rule; this + document prescribing markers it does not carry is the same inversion §8 is about. **Deferred: #887.** Widest blast radius in principle — an unauthenticated `PUT /transcript` changes what the agent resumes from — but the HMAC capability-key fix is @@ -530,7 +575,9 @@ all. 30 minutes is roughly **22–24 minutes of content, 4 minutes of demo, 3–4 minutes of questions.** The demo is the constraint: D1–D5 as written cannot be driven live in four -minutes, because POST → pod Ready alone is ~14 s and a full cycle to zero is 22 s. +minutes: wake alone is 1.0–5.0 s on ykt1 and the cycle back to zero is 30–31 s with +`cooldownPeriod: 30` (`IMPLEMENTATION_REPORT1.md` §3), and that is before the agent +does any work. ### 6.1 Demo strategy: pre-record, narrate live @@ -539,8 +586,9 @@ minutes, because POST → pod Ready alone is ~14 s and a full cycle to zero is 2 1. The full four-stage loop is ~6 minutes in mock mode (`k8s_demo_flow.py`), and the real-agent path has been blocked on an exhausted LiteLLM team budget (`IMPLEMENTATION_REPORT1.md` §7) — a 429 on stage is unrecoverable. -2. Image pull is ~12 s of the ~14 s wake. Pre-pull on the node and the number improves, - but it is still dead air. +2. A cold image pull is 5.1 s of the wake (245 MB); warm it is 454 ms–2.45 s + (`IMPLEMENTATION_REPORT1.md` §3). Pre-pull on the node and the number improves, but + it is still dead air. 3. Conference wifi. Phase 2 §2.5 already lists offline operation as a design requirement for exactly this reason. 4. #889 crash-loops a pod after an abrupt exit. @@ -571,6 +619,11 @@ either accept that or record that one beat separately against a funded key. ### 6.3 The §8 section is the differentiator — give it 3–4 minutes + + `DESIGN_PHASE2.md` §8 — thirteen documented claims that do not hold in code, each with a filed issue, plus the five-question checklist that came out of them — is the most valuable artifact in this repository, and it is more on-theme than it first appears. @@ -590,7 +643,8 @@ will be presenting one. The five questions (§8.5) compress to one slide: 4. Can an attacker choose the input the check reads? 5. What does the claim become one release from now? -Plus the two rules from §8.6 that generalize best: **call the real function** (a +Plus the two rules from `DESIGN_PHASE2.md` §8.6 that generalize best: **call the real +function** (a reimplementation of the guard proves nothing about the guard), and **assert the outcome, not the verdict**. @@ -644,7 +698,7 @@ answer ready. | Why not HMAC — it's 170,000× faster? | Phase 2 §4.3: symmetric, so EventBridge could forge any agent's response | | Kafka ACLs? | Phase 2 §3.3: the authorizer is global, not per-listener, so the demo is either broken or vacuous | | Multi-tenancy? | `DESIGN_PHASE3.md` §2–§3, **design only** | -| What's the signing overhead? | ~150–200 ms/op. Get the on-cluster number in M2 | +| What's the signing overhead? | 222 ms sign / 227 ms verify, measured (`IMPLEMENTATION_REPORT1.md` §9). Pure Python; `cryptography` does it in microseconds. Confirm on-cluster in M2 | --- @@ -673,7 +727,7 @@ Six things from the 2026-10-05 review that are not obvious from any single docum 2. **Reading the code is not enough, and neither is the test suite.** All thirteen findings coexisted with a passing suite, because the tests exercised the configuration - each claim was true in. 575 passing tests did not catch #885 or #888. + each claim was true in. ~900 passing tests did not catch #885 or #888. 3. **My own first attempt at the group-forgery test reported no issue** — because I used wrong event-type constants (`io.rossoctl.…` instead of `dev.rossoctl.agent.group. @@ -707,10 +761,10 @@ The code was read and the decision functions were run, but there was **no cluste access** — everything stated about cluster behaviour comes from `IMPLEMENTATION_REPORT1.md` rather than from observation. Cluster-dependent claims in §2 should be re-confirmed in M2. -The CNI finding in §8.6 was reached by reading `k8s/kind/kind-cluster.yaml`, not by +The CNI finding in §8 observation 6 was reached by reading `k8s/kind/kind-cluster.yaml`, not by applying a NetworkPolicy to a running Kind cluster and watching it fail to block. That is the right direction for the conclusion — kindnetd's lack of NetworkPolicy support is -documented upstream — but by this document's own §8.6 rule it is still a claim that has +documented upstream — but by `DESIGN_PHASE2.md` §8.6's rule it is still a claim that has been read rather than run. Confirm it before spending anything on either CNI option. --- diff --git a/eventing/agentdocs/README.md b/eventing/agentdocs/README.md index 2a72c072..ca71f9d5 100644 --- a/eventing/agentdocs/README.md +++ b/eventing/agentdocs/README.md @@ -19,7 +19,7 @@ and what is *not* verified. | [`IMPLEMENTATION_REPORT1.md`](IMPLEMENTATION_REPORT1.md) | What Phase 1 built, the measured results on both clusters, 28 findings including two real bugs only a scale-to-zero deployment could expose, and an honest account of what is still blocked and why. | | [`README_PHASE1.md`](README_PHASE1.md) | The Phase 1 runbook in full detail, covering all four ways to run it: locally without containers, under Docker, on a Kind cluster, and on OpenShift. | | [`DESIGN_PHASE2.md`](DESIGN_PHASE2.md) | The Phase 2 design — identity on the event path. Why GitHub's opaque user token rules out local verification and forces a `GET /user` lookup plus a load-bearing cache; why `401` and `403` are kept distinct; what `ce_submitter` is and is not worth, and what a signature over it does and does not prove; how the `kid` and the approved-key set prove *which agent* answered, why group events are pinned to EventBridge's own key, and why a rejected response is stored as an error rather than dropped. **§8 records the thirteen claims across §§2.6, 3, 4.3 and 4.4 that a docs review found do not hold in the code, with the issue filed for each — and §8.5-§8.8 the procedure that came out of them: the five questions to ask of any control claim, in any phase, before it ships.** | -| [`KUBECON_NA_2026.md`](KUBECON_NA_2026.md) | **The KubeCon NA knowledge base.** The accepted abstract and the five demo beats it commits to; the status of each against the code as of 2026-10-05, including what was confirmed by *running* the decision functions rather than reading them; the SPIRE decision that shapes everything else (two of the four seams name it and there is no SPIRE); the W1-W6 plan and the **M1-M4 MVP** cut of it; and the 30-minute slide and demo shape, with the recommendation to pre-record. Cites the other documents rather than restating them. | +| [`KUBECON_NA_2026.md`](KUBECON_NA_2026.md) | **The KubeCon NA knowledge base.** The accepted abstract and the five demo beats it commits to; the status of each against the code as of 2026-10-05, including what was confirmed by *running* the decision functions rather than reading them; the SPIRE decision that shapes everything else (two of the four seams name it and there is no SPIRE); the W1-W6 plan and the **M1-M4 MVP** cut of it; and the 30-minute slide and demo shape, with the recommendation to pre-record. Cites the other documents rather than restating them. **A living document with a deadline, unlike the phase records above — its status figures and its "all still open" bug list expire at the talk.** | | [`DESIGN_PHASE3.md`](DESIGN_PHASE3.md) | The Phase 3 design — per-user isolation, declarative agents, and event triggers. Why a raw `submitter` cannot be a tenancy key and why the derived `userkey` ends in a hash; per-user Kafka topics (`{prefix}-{userkey}-requests`), one EventRunner Deployment and ScaledObject per user, and the honest difference between separately-named topics and ACL-enforced isolation; why a per-user ntfy topic must *not* contain the user's name, and the option of running ntfy as a service in the cluster; the `AgentSpec` — an agent baked into the image or assembled from skills fetched from GitHub, S3 or a URL, and the five gates a fetched bundle passes; authenticating and scoping the session transcript; and a Knative-inspired trigger API with the four loop controls that keep an event-driven agent system from becoming a cost incident. **Partly implemented: §9's steps 1-4 (T1, T2, T3, T4, T13, T22) plus T5 (per-user stores and the global ownership index) are in the tree; `EB_TENANCY_MODE` still defaults to `single`, which reproduces Phase 2 exactly. Owner-scoped reads (T7), transcript auth (T8), `k8s_tenant.py` (T12), ntfy isolation, triggers, fetched skills and Kafka ACLs are design only — so `multi` mode routes each tenant to its own store and topics, but does not yet check that a reader is the owner.** | | [`IMPLEMENTATION_REPORT3.md`](IMPLEMENTATION_REPORT3.md) | What Phase 3 has built so far (§9 steps 1–4 plus T5), measured rather than estimated: the per-tenant file-descriptor cost against §6.1's estimate, the ownership index's per-operation cost, and the 270 ms that settles §2.6's objection to seeding. Seven deltas from the design recorded as decisions, and the findings with their root causes — including a `single`-mode regression that shipped behind a headline claiming it could not happen, an eviction contract whose first fix addressed only the visible case, and `except Exception` converting three separate programming errors into log lines. **Partial: it will be extended as later steps land rather than rewritten.** |