Skip to content

feat: cmd/sandbox-e2b, the e2b surface on a sandboxd mesh with no Kubernetes - #57

Merged
CMGS merged 3 commits into
masterfrom
feat/mesh-sandbox-e2b
Sep 27, 2026
Merged

CMGS merged 3 commits into
masterfrom
feat/mesh-sandbox-e2b

Conversation

@CMGS

@CMGS CMGS commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Summary

PR 5 of 5 for mesh mode: cmd/sandbox-e2b serves the e2b surface on a sandboxd mesh with no Kubernetes. That means no kubeconfig, no NodeInventory and no APIService.

Wiring.

  • meshinventory.New runs over the seeds, dialing sandboxd clients that share one HTTP client.
  • One scatter-gather store with claim routing sits over it, and the e2b server serves from that store.
  • With --envd-proxy-bind-address, the envd data plane runs in the same process. It shares the poller, the store and the index, and its domain is --e2b-domain.
  • The startup log shows the first tick:
mesh inventory up seeds=[<A>:27777] nodes=[<B>:27777 <A>:27777] poll=10s

Flags:

Flag Default Meaning
--sandboxd-seeds — Comma-separated seed addresses, each naming one node.
--sandboxd-token, --sandboxd-token-file — The fleet root token.
--inventory-poll 10s The tick between polls.
--e2b-* The same flags as the apiserver's e2b surface.
--envd-proxy-bind-address — Serve the envd data plane in the same process.
--envd-proxy-tls-cert-file, --envd-proxy-tls-private-key-file, --envd-proxy-guest-http2 — The proxy's TLS and guest options.

Shared pieces, so the three mains share one definition of each.

  • e2bcompat: Flags holds the --e2b-* block, including --e2b-template-aliases. ServerOptions(inv) reads the key and alias files. Server.Serve(ctx, addr) covers the listener start and drain, moved from the apiserver main.
  • envdproxy: Flags holds the listener options, and AddFlags(fs, prefix) registers them. The envd-proxy binary keeps its names with prefix "", and sandbox-e2b uses envd-proxy-. Flags.Serve(ctx, ln, h) is moved from the envd-proxy main, and its drain test moves with it.
  • sandboxd.TokenFrom(literal, file): the token-file read used by the apiserver and sandbox-e2b.

The store logs a followed claim redirect at debug: claim followed a redirect from=<node> node=<node> id=<claim>. The line sits on the hop, not on the warm hit.

Registration: the Makefile BINARIES, .goreleaser.yml (a build plus versioned and unversioned archives, goreleaser check ok), the docker.yml image matrix, the build.yml artifacts, the README, and docs/index.md (plus the two inventory packages).

Docs:

  • docs/e2b-compat.md gains a mesh-mode section. It covers the flags, and that the node key is /v1/info advertise_addr, which is the client origin when client_advertise is set. It names the minimum sandboxd as the first release carrying that field (sandbox main acfca8b today). It also covers startup failures, the silent-node and unnamed-peer rules, warm pools through SetPoolsCluster or sandboxd config, and redirects.
  • docs/scaling-design.md describes the redirect hop on the claim path.

Closure and budget

Head to merge: 595780eba5b3273f2a26d6e39523e8a0ccf5b850, on master 2ef6a7c.

New exported identifiers and their consumers:

identifier consumers at this head
e2bcompat.Flags, NewFlags, AddFlags, ServerOptions cmd/sandbox-apiserver, cmd/sandbox-e2b
e2bcompat.Server.Serve the same two
envdproxy.Flags, AddFlags, Flags.Serve cmd/sandbox-envd-proxy, cmd/sandbox-e2b
sandboxd.TokenFrom cmd/sandbox-apiserver, cmd/sandbox-e2b

Removed from the mains: startE2BServer's listener code, e2bFileLines, resolveSandboxdToken, serveOn, and their consts.

Comment lines per file (added / removed):

file + −
cmd/sandbox-apiserver/main.go 0 4
cmd/sandbox-envd-proxy/main.go 0 2
cmd/sandbox-e2b/main.go 1 0
pkg/e2bcompat/flags.go 4 0
pkg/e2bcompat/server.go 1 0
pkg/envdproxy/flags.go 4 0
pkg/sandboxd/client.go 1 0
others 0 0
  • The additions are the package comment and one-line godocs on the new exported identifiers.
  • The exception is envdproxy's one WHY line on shutdownTimeout. It moved with the const from the envd-proxy main, which drops two lines for it.

Boundaries

GOWORK=off go list -f '{{join .Imports "\n"}}', cocoonstack and pflag imports only:

pkg/e2bcompat:   pkg/scale, github.com/spf13/pflag
pkg/envdproxy:   pkg/e2bcompat, pkg/sandboxd, pkg/scale, github.com/spf13/pflag   (the e2bcompat edge is pre-existing, for sandbox id rendering)
cmd/sandbox-e2b: pkg/e2bcompat, pkg/envdproxy, pkg/sandboxd, pkg/scale, pkg/scale/meshinventory, version, github.com/spf13/pflag
  • e2bcompat imports neither envdproxy nor any cmd.
  • cmd/sandbox-e2b never imports kubeinventory.

Acceptance 1:

  • go list -deps ./cmd/sandbox-e2b | grep -E 'sigs.k8s.io/controller-runtime|k8s.io/client-go|kubeinventory' prints nothing (rc 1).
  • The same query on ./cmd/sandbox-apiserver counts 387.

Hot path

  • Apiserver: the e2b start is the same code, moved; the startup and request paths are unchanged.
  • envd-proxy: the same serve loop, moved. The TLS-pair check now runs in Serve, still before any request.
  • sandbox-e2b: claims, reads and watches are routed from the snapshot, with no per-request inventory I/O. The background cost is 2 requests per node per tick, all at once, bounded by the tick. A followed redirect adds one debug line. A warm hit logs nothing new.

Hardware

Setup:

  • Two testbed hosts.
  • sandboxd main acfca8b on both mesh nodes, joined by memberlist, with an e2b-rt:24.04 pool.
  • The PR5 binaries built at 8aaf421. This head differs from 8aaf421 only in the docs/e2b-compat.md minimum-sandboxd wording.
  • JS ran from a workstation through an ssh tunnel to the sandbox-e2b e2b listener and its embedded envd-proxy listener. Python and curl ran on the node.

Acceptance 5, mesh mode. sandbox-e2b was seeded with node A only and poll 10s, with the key and alias files.

row JS 2.51.0 JS 2.50.0 Python 2.51.0
base matrix (sdk-run: create, commands, files, info, list after the poll, pause, connect from a fresh handle, exec after resume, second key refused, default lane, network rules 400, kill) PASS PASS PASS
paging (page-run, pages of 2 over 5) PASS PASS PASS
aliases (alias-run: Sandbox.create() → base → e2b-rt; templateId = image, name = base; alias check) PASS PASS PASS
legacy resume (py-resume) — — PASS
row result
e2b CLI 2.20.0 sandbox logs rc 0
PA-mesh-01..04, curl (resume, logs, page params, aliases) 4/4 PASS
M5-01 forced warm miss See below.

M5-01 in detail:

  • Setup: sandbox-e2b restarted with poll 60s while A was warm and B cold. B was then filled and A drained within the window, so the snapshot still named only A warm.
  • Result: the create answered 201 in 2.6 ms, with clientID = <B>:27777, templateID = the e2b-rt image and alias = base.
  • Counters: A's sandboxd_claims_total stayed flat at warm=14 clone=0 cold=0, and B's warm tier went from 19 to 20.
  • Log:
{"level":"debug","func":"scale.claimRedirected",...,"message":"claim followed a redirect from=<A>:27777 node=<B>:27777 id=sb_..."}

Acceptance 6, kube regression on the PR5 sandbox-apiserver and sandbox-envd-proxy.

  • The unchanged SBX lane: 28/28 PASS (SBX-01..12, SBL-01..08, SBE-01..04, SBP-01..03).
  • On the PR5 apiserver arm: JS 2.51.0 sdk-run, page-run and alias-run PASS; Python 2.51.0 sdk-run PASS; PA-p5-01..04 PASS.

Review and gates

/simplify, all four lenses:

  • The hoists are minimal, and the prefix on the proxy flags is the smallest way to share them.
  • Kept in each main: stderrIsTerminal and the log setup. Sharing them would take a package for trivial code.
  • Kept: two HTTP clients in sandbox-e2b, one for mesh polls and one for claim routing, so the poll burst never touches the claim path's connections.
  • Applied in review::
    • Server.Serve moves beside Handler.
    • The envd-proxy start log carries domain again.
    • The docs quote the real advertise_addr error, name the unnamed-peer drop, say claims are routed from the snapshot, and name no unreleased tag.
    • The --e2b-domain help reads for both binaries.

/code, every touched file read in full:

  • Applied: the own-line WHY comment is uppercase.
  • Rejected: moving TokenFrom above the Client methods, because methods precede standalone functions.

Gates:

  • make lint: 8/8 0 issues. on darwin and on linux.
  • make fmt-check: ok.
  • make vet: ok, with the three tagged harnesses.
  • make build: builds all three binaries.
  • asl: 0 findings on both platforms.
  • go mod tidy -diff: clean.
  • go test -race: ok.

…ernetes

sandbox-e2b builds the mesh inventory source over the seeds, one store
with claim routing over it, and serves the e2b surface from that store;
with --envd-proxy-bind-address the envd data plane shares the same
poller, store and index in the process. The --e2b-* flag block and the
listener start and drain move into e2bcompat (Flags, ServerOptions,
Server.Serve), the proxy's listener flags and serve loop into envdproxy
(Flags with a name prefix, Flags.Serve), and the token-file read into
sandboxd.TokenFrom, so the three mains share one definition of each.
The binary is registered in make, goreleaser, the docker and build
workflows, and the README and docs, with a mesh-mode section in the
e2b guide and the redirect hop in the scaling design.
A delivered redirect hop left no trace, so an operator could not tell a hop from a direct warm hit.
…s start log, mesh docs match the code

Serve moves from flags.go to server.go with its timeouts; the hoist had dropped domain from the envd-proxy log; the mesh section quotes the real advertise_addr error, names the unnamed-peer drop, and routes claims rather than serving them from the snapshot; --e2b-domain help reads for both binaries.
@CMGS
CMGS merged commit 4e10f5c into master Sep 27, 2026
2 checks passed
@CMGS
CMGS deleted the feat/mesh-sandbox-e2b branch September 27, 2026 08:29
CMGS added a commit that referenced this pull request Sep 27, 2026
Repository-scope review round at the end of phase A (#50 to #57). Fixes with regression tests: the fleet HTTP client dials with a one-second timeout so a dead node is a capacity miss instead of a ten-second 500; AddressIPs cuts a client origin's scheme; an unnamed mesh member leaves only after MaxStale silent ticks so a graceful restart no longer hides a node, with the fewest-fails publish and seed-wins dedupe rules pinned; the chart passes the template alias file; the v2 page cursor keys on the raw claim time; the alias lookup answers for an image the fleet advertises. Simplifications: one logs handler, warmByKey over PoolCapacity, the list paths trimmed, two tests reuse pageOfList. Comments cut to one line each in the driver, the client and the mains. The flag is --e2b-template-alias-file. Docs: snapshot-placement follows #52, the package doc names both inventory sources, the mesh section states the corrected membership rule, reconnect by id needs SDK 2.6. Prod lines net -42, comments +33/-82, all gates green on darwin and linux.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant