feat: drop a NodeInventory whose publishedAt trails the fleet's newest publish by more than --inventory-stale-after

[CMGS]

Head: 6b9cb2e8a1d1e2ec3888c0aada593371d671f582, 4 commits on master 6da4f6f7. Commit 835f250c reworks the first two to CMGS's K1 decision, and 6b9cb2e8 corrects the docs from the hardware results.

K1: NodeInventory freshness on the Kubernetes path, as section 9 of the e2b drop-in plan was rewritten after the decision:

1. A relative reference clock.
2. No exit delete; option 3 is dropped.

The vk-sandbox follow-up, which only stamps publishedAt, follows once this merges and is pinned to it.

What

• publishedAt: NodeInventory gains publishedAt (metav1.Time, omitempty). The CRD under helm/crds and docs/api.md are regenerated.
• The reference: kubeinventory.Options{StaleAfter} defaults to 90 s, and the constructor is now New(reader, opts). In the pass ListNodes already makes, it computes reference = min(newest publishedAt, now) and stores it on the Source as atomic unix nanos.
• The filter: ListNodes drops an inventory whose stamp trails the reference by more than StaleAfter. NodeInventory and NodeCapacity compare against the last stored reference and answer a wrapped NotFound for a stale node.
• Fresh by default: a zero reference (before the first list), and a missing or unparsable stamp, both count as fresh.
• What the reference buys:
  • A control-plane write outage freezes every stamp together, so it never empties the fleet.
  • A publisher clock running ahead is capped by now and cannot evict the others.
  • The reader's own clock no longer decides a node's age.
  • A fleet whose nodes all stop at once, or a one-node fleet, keeps its last inventory.
• No store change: every consumer reads through the source, so a stale node leaves all of them at once, and the watch emits Deleted for its entries.
• Flag: --inventory-stale-after (default 90s) on sandbox-apiserver and sandbox-envd-proxy, through kubeinventory.Options.AddFlags.
• Option 3 dropped: InventoryApplier stays Apply-only, and StaticInventorySource.Remove is unchanged from master.
• Docs:
  • configuration.md: flag rows; the reference semantics; that a dead node leaves between the window and one publish interval later, because the reference only advances when a live node publishes; the CRD-plus-vk-sandbox upgrade condition.
  • e2b-compat.md: the limit.
  • scaling-design.md: the dead-node row, and a control-plane outage row.
  • helm/README.md: publishedAt named among the pruned fields.

Closure

new exported identifier	consumer
NodeInventory.PublishedAt	kubeinventory.Source; vk-sandbox's publisher in the follow-up
kubeinventory.Options, .StaleAfter, .AddFlags	cmd/sandbox-apiserver, cmd/sandbox-envd-proxy

Package edges: none added. kubeinventory's new imports are pflag, apimachinery api/errors and runtime/schema, all external, plus stdlib sync/atomic.

The exported scale surface is unchanged from master.

Comment delta

file	+	−
api/v1beta1/nodeinventory_types.go	2	0
pkg/scale/kubeinventory/source.go	3	0

The additions are the field doc with its +optional marker, and godocs on Options, StaleAfter and AddFlags.

LOC

	+	−	net
prod	86	17	69
test	124	1	123
generated (CRD, deepcopy, api.md)	5	0	5
docs	39	3	36

Hot path

ListNodes parses each node's RFC3339 stamp once and takes one min against the clock. It stores one atomic value and allocates one stamp slice per call. A point lookup costs one parse and one atomic load.

A/B on BenchmarkClientInventoryWarmCandidates/*/nocopy, the informer-fed production read path with stamped inventories. Base and the rework head ran interleaved over 4 rounds with the order swapped:

fleet	base median	branch median	allocs
26 nodes × 100	48.9 µs	51.8 µs	682 / 683
200 nodes × 2000	271.9 µs	286.9 µs	5032 / 5033

Hardware

The kube kit on the test hosts runs:

• the apiserver and envd-proxy built from 835f250c (6b9cb2e8 changes docs only);
• vk-sandbox from the follow-up 76ec64d, on this branch;
• node k1-a, a vk-sandbox serving host A's sandboxd, and node k1-b, a vk-sandbox serving host B's;
• a SandboxWarmPool of 4, the e2b API, and default windows of 30 s publish and 90 s stale.

Lane (a), hard kill, kill -9 of B's sandboxd and B's vk-sandbox, with 7 fresh sandboxes on B:

PASS K1A-01  node B holds sandboxes that the e2b list and the kube list both show
PASS K1A-02  baseline: 30 e2b creates over both nodes
PASS K1A-03  B's sandboxes leave both lists   (120.771 s after B's last publish: A's first publish past the window)
PASS K1A-04  the kube watch emitted DELETED for each B sandbox
PASS K1A-05  e2b GET and DELETE on a B sandbox answer 404
PASS K1A-06  30 creates after the window never sample B; p50 matches the baseline and none stalls
PASS K1A-07  envd-proxy probes for 20 unknown ids
PASS K1A-08  no packet toward B's sandboxd after the window   (tcpdump dst B:27777: 25 before window+2s, 0 after)
PASS K1A-09  restarting B brings its inventory back on the next publish, and the sandboxes sandboxd kept reappear

Lane (b), reworked: SIGTERM and restart of B's vk-sandbox only, inside the window:

PASS K1B2-01  vk-sandbox down 6.9 s; the inventory present throughout; both lists unchanged (7 B sandboxes);
              0 watch events after the watch's opening burst; 0 apiserver log lines naming B

Lane (c), an old publisher, vk-sandbox 7d15e45, which never stamps:

PASS K1C-01  k1-b published without publishedAt; after 100 s the inventory is present and all B sandboxes are listed

Outage lane: kube-apiserver killed for 138 s, with two apiserver replicas as the chart ships. Replica 1 held the warm-pool lease; replica 2 had e2b on its own port.

t=15s:  replica1 up,   replica2 up, proxy up; r2 lists B/all: 7 16
t=45s:  replica1 up,   replica2 up, proxy up; r2 lists B/all: 7 16
t=75s:  replica1 down, replica2 up, proxy up; r2 lists B/all: 7 16
t=135s: replica1 down, replica2 up, proxy up; r2 lists B/all: 7 16
claim on replica 2 at t=136s: http 201 … "clientID":"k1-a"
sandbox-apiserver: warm-pool manager: leader election lost
PASS K1OUT-01

• What the outage lane shows: every node stays listed through the outage, and a claim lands.
• A pre-existing issue it surfaced, not introduced here: the replica holding the warm-pool driver's lease exits when it cannot renew the lease. startWarmPoolDriver calls fail when mgr.Start returns "leader election lost". With the chart's 2 replicas, the other keeps serving. A single-replica deployment loses its e2b API for the outage. The outage row in scaling-design.md says so. Whether the driver should survive a lost lease is a separate decision.

Harness notes:

• The first rework run was voided. B's sandboxes from the earlier round had passed their 30-minute leases, so lane (a) ran with none on B. And the first outage lane's claim line read a stale response file.
• Both are fixed: lane (a) now claims fresh sandboxes on B, and the claim is judged by its HTTP status.
• The watch assertion now counts events after the aggregated watch's opening burst, which sends one ADDED per existing sandbox.

Review

• The first round, on 25d0e940: /simplify, altitude and /code are in the history of this PR.
• The rework: the reference logic got mutation testing. It covers the boundary (>=, +1 s), the min against now (removed either way), the store, the list filter, the get filter, the zero stamp, the parse error, and max vs min. Every mutation is caught.
• Code removed by mutation evidence: a redundant zero-reference guard. A zero reference already yields a negative age.
• New tests: dead node against a live publisher (90 s kept, 91 s dropped), a 10-minute publish outage, a rogue stamp 1 h ahead, a lookup before any list, a custom StaleAfter, and the stamp parser.

Gates

• GOWORK=off make lint: 8 × 0 issues.
• make fmt-check: clean.
• asl -forwarder=false on both GOOS: clean.
• go mod tidy -diff: clean.
• make generate: leaves no diff.
• go test -race -count=1 ./...: ok.
• vk-sandbox follow-up built against this branch: go vet clean, go test -race ./... ok.