Add backend="fabric": cells in worker processes that can be killed

[seanwevans]

This is the one that makes the repo's description true. Stacked on #297 (a one-file lint fix for currently-red main); merge that first and this retargets to main cleanly.

The gap this closes

#295 landed real sub-interpreter cells but left the one thing a multi-tenant deployment cannot live without: a running cell cannot be reclaimed. close() refuses while the guest executes, there is no kill, and an async exception aimed at the thread does not reach the interpreter running in it. In-process, a runaway guest is permanent — the cell is abandoned and its thread stays pinned for the life of the process.

The fabric puts cells somewhere killable:

 Supervisor
   |
   +-- Worker process   <- the kill domain
   |     +-- cell  cell  cell      (one CellPool, many cells)
   |
   +-- Worker process
         +-- cell  cell

Level	Isolates	Reclaimable
cell	sys.modules, builtins, globals	no
worker	the kill domain, and where a memory cap applies	yes — SIGKILL
fabric	which worker a tenant lands in	n/a

It actually works

Measured end to end through the public API on free-threaded 3.14:

acme  -> 1.4142135623730951
globex-> 3.0
placement: [{'id': 1, 'pid': 7821, 'tenant': 'acme',   'cells': 1, 'alive': True},
            {'id': 2, 'pid': 7824, 'tenant': 'globex', 'cells': 1, 'alive': True}]
acme wedged, reclaimed: cell c1 exceeded 0.5s and did not return. A running sub-inte ...
globex-> still here
acme  -> recovered
stats: {'workers_started': 3, 'workers_killed': 1, 'cells_live': 2, 'tenant_isolation': True}

The runaway is reclaimed at its deadline, the worker process is genuinely gone, the other tenant never noticed, and the wedged tenant is serving again on a fresh worker.

There's a tell in the test suite: #295's runaway tests have to shell out to a fresh interpreter, because a stranded cell hangs pytest at exit. tests/test_fabric.py runs its runaway cases inline — the fabric reclaims them.

Placement is the blast-radius decision, so it is explicit

tenant_isolation=True (default) means a worker only ever hosts one tenant's cells, so killing it costs that tenant alone. Turning it off packs tenants together for density and makes them share a fate.

pool = WorkerPool(
    max_workers=8,
    cells_per_worker=16,
    tenant_isolation=True,
    worker_mem_bytes=2 << 30,   # RLIMIT_AS per worker
)
pool.prewarm("acme", 2)         # pay the ~160 ms spawn before traffic arrives

worker_mem_bytes answers a gap #295 recorded: sys.getallocatedblocks() is process-global on both free-threaded and GIL builds, so there is no per-cell figure to cap. Worker sizing is the control that exists.

What it costs

Operation	p50
worker spawn (fresh interpreter + pyisolate import + pool)	160.8 ms
first cell for a tenant (spawns its worker)	190.3 ms
further cells in that worker	29.8 ms
exec + recv round trip	1.20 ms

The kill domain costs ~0.4 ms per round trip over an in-process cell (1.20 ms vs 0.82 ms), plus one worker spawn per tenant that prewarm moves off the request path.

Two defects found and fixed while building it

Zombie workers. A worker killed from outside stayed a zombie until something happened to poll it, so a fabric recycling under load would accumulate one per recycle. The reader thread now reaps on EOF — the earliest reliable signal the process is finished. (Surfaced because a test helper's os.kill(pid, 0) probe kept reporting a killed worker as alive.)

A dead worker reporting is_alive(). Popen.poll() returns None when it cannot take the waitpid lock, so while the reader thread sat inside wait() a dead worker still looked alive — long enough for placement to hand it a new cell. The worker is now marked unusable when its channel hits EOF, before the reap.

Design notes worth a reviewer's attention

• Workers are spawned, never forked. Forking a process that already hosts sub-interpreters and their threads segfaults — the same reason #295's tests use a fresh interpreter.
• Exceptions are rebuilt from a fixed name map, not a dynamic lookup. A worker runs guest code; letting it name an arbitrary supervisor class and have it constructed here would be a hole. There's a test for it.
• Cell operations run on their own thread inside the worker, so one wedged guest cannot stall the control loop and make a busy worker indistinguishable from a wedged one.
• A pump forwards cell output every 5 ms, so a guest that posts and then blocks is not silent until its operation returns.

What it is still not

A guest that escapes its cell owns its worker, and a worker is an ordinary process holding the supervisor's privileges. The fabric is a fault boundary, not a security one. backend="process" remains the boundary mode.

Kernel confinement of workers is now a roadmap item with a plan rather than a gap: tenant_isolation makes a worker's policy unambiguous, so the process backend's seccomp/Landlock/cgroup layers could be applied at spawn. Also on the roadmap: broker request execution (capability-scoped per cell, not per worker) and deriving placement from policy.

Testing

tests/test_fabric.py — 30 tests: the cell ABI through a worker, placement (tenant isolation on/off, capacity, packing), the kill domain (runaway recycle, bystander unaffected, tenant recovery, external kill surfaces instead of hanging, dead-worker reaping), pre-warming, observability, and the delegated sandbox surface. Plus 4 supervisor-level tests for backend="fabric" and fabric_report().

• 3.14t free-threaded: 628 passed, 19 skipped
• 3.13: 574 passed, 73 skipped
• CI's 3.14t job (renamed cells + fabric / py3.14t) gains a fabric step; I ran it against a venv with only pytest pyyaml platformdirs, matching the job's dependency shape: 30 passed, 1 skipped.
• pre-commit run --all-files passes.

test_apply_confinement_installs_seccomp_and_allows_normal_syscalls fails in my container on every branch including unmodified main — seccomp is unavailable here. I also saw test_cpu_quota_is_debug_telemetry_without_watchdog fail in isolation on 3.14t; it does that on the parent branch too and passes in a full run, so it is order-dependent and unrelated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fs1hTtmF4Hm9h617AG9Gse

---

Generated by Claude Code