Add backend="fabric": cells in worker processes that can be killed

.github/workflows/ci.yml

@@ -102,7 +102,7 @@ jobs:
         run: pytest -q tests/test_matrix_hardening.py -m "race and no_gil"
 
   tests-subinterpreter:
-    name: sub-interpreter cells / py3.14t
+    name: cells + fabric / py3.14t
     runs-on: ubuntu-24.04
     steps:
       - uses: actions/checkout@v4
@@ -132,6 +132,10 @@ jobs:
           PY
       - name: Run the sub-interpreter backend suite
         run: pytest -q tests/test_subinterpreter_backend.py tests/test_supervisor.py
+      - name: Run the fabric suite
+        # Spawns worker processes and kills them; the runaway cases run inline
+        # here because the fabric reclaims them, unlike the in-process backend.
+        run: pytest -q tests/test_fabric.py
       - name: Validate the no-GIL readiness axis on 3.14t
         run: pytest -q tests/test_nogil.py
 

CHANGELOG.md

@@ -8,6 +8,16 @@ guarantees; **no release should be treated as a hardened security boundary**.
 ## [Unreleased]
 
 ### Added
+- `backend="fabric"`: the multi-tenant mode. Sub-interpreter cells hosted in
+  worker processes the supervisor can kill, which is the only reclaim that
+  works on a running cell. A deadline that expires kills the worker and the
+  tenant keeps running on a fresh one; other tenants are untouched. Includes
+  tenant-aware placement (`tenant_isolation=True` by default, so a kill costs
+  one tenant), per-worker `RLIMIT_AS` caps, `prewarm` to move the ~160 ms
+  worker spawn off the request path, worker recycling with counters, and
+  `Supervisor.fabric_report()` for placement visibility. Needs CPython 3.14+
+  and fails closed below it.
+
 - `backend="subinterpreter"`: real CPython sub-interpreter cells on 3.14+, via
   `concurrent.interpreters`, with a pre-warmed `CellPool`. Each guest gets its
   own `sys.modules` and its own `builtins`, so the import allow-list is a
@@ -63,11 +73,17 @@ guarantees; **no release should be treated as a hardened security boundary**.
 - The eBPF programs compile to loadable objects and are covered by ELF-level
   tests, but load/attach against a live verifier is still only exercised by
   the root-gated `PYISOLATE_LIVE_BPF_TESTS=1` tests, not by CI.
-- A running sub-interpreter cell cannot be reclaimed: one that overruns its
-  wall-time deadline is abandoned, and its thread stays pinned until the
-  process exits. Cells enforce a wall-time deadline and no other quota;
-  `sys.getallocatedblocks()` is process-global, so per-cell memory
-  accounting needs a worker-process layer that does not exist yet.
+- In `backend="subinterpreter"` a running cell still cannot be reclaimed: one
+  that overruns is abandoned and its thread stays pinned until the process
+  exits. Use `backend="fabric"`, where the worker is the kill domain.
+- A fabric worker is an ordinary process with the supervisor's privileges: the
+  fabric bounds faults, not hostile Python. Kernel confinement of workers is
+  not implemented.
+- The broker `request` op is surfaced by the fabric but, as with the other
+  backends, nothing executes it.
+- Memory is capped per worker (`RLIMIT_AS`), not per cell:
+  `sys.getallocatedblocks()` is process-global on both free-threaded and GIL
+  builds, so there is no per-cell figure to limit.
 - Process-backed sandboxes are not attached to cgroups or watched by the
   resource watchdog (they get `rlimit` only).
 - `backend="microvm"` fails closed: the guest agent and vsock cell transport are

README.md

@@ -263,6 +263,95 @@ domain.
 
 ---
 
+## The fabric
+
+`backend="fabric"` is the multi-tenant mode: sub-interpreter cells hosted in
+**worker processes the supervisor can kill**.
+
+```
+ Supervisor
+   |
+   +-- Worker process   <- the kill domain
+   |     +-- cell  cell  cell      (one CellPool, many cells)
+   |
+   +-- Worker process
+         +-- cell  cell
+```
+
+It exists because of one limitation the in-process cell backend cannot fix: a
+running sub-interpreter **cannot be reclaimed**. `close()` refuses while the
+guest executes, there is no `kill`, and an async exception aimed at the thread
+does not reach the interpreter running in it. In-process, a runaway guest is
+permanent. Putting cells in a worker makes the process the unit of reclaim:
+
+```python
+import pyisolate as iso
+
+sb = iso.spawn("report", backend="fabric", tenant="acme", wall_time_ms=500)
+sb.exec("while True: pass")
+# WallTimeExceeded: cell c1 exceeded 0.5s and did not return. A running
+# sub-interpreter cannot be reclaimed, so the worker process hosting it was
+# killed; cells sharing that worker were lost with it.
+```
+
+Each of the three levels is a different kind of boundary:
+
+| Level | Isolates | Reclaimable |
+| --- | --- | --- |
+| cell | `sys.modules`, `builtins`, globals | no |
+| worker | the kill domain, and where a memory cap applies | **yes — SIGKILL** |
+| fabric | decides which worker a tenant lands in | n/a |
+
+**Placement is the blast-radius decision.** With `tenant_isolation=True` (the
+default) a worker only ever hosts one tenant's cells, so killing it for a
+runaway costs that tenant and nobody else. Turning it off packs tenants
+together for density and makes them share a fate. The fabric makes callers
+state which they want rather than picking silently.
+
+```python
+from pyisolate.runtime.fabric import WorkerPool
+
+pool = WorkerPool(
+    max_workers=8,
+    cells_per_worker=16,
+    tenant_isolation=True,     # one tenant per worker
+    worker_mem_bytes=2 << 30,  # RLIMIT_AS per worker
+)
+pool.prewarm("acme", 2)        # pay the ~160 ms spawn before traffic arrives
+```
+
+`worker_mem_bytes` is where a memory limit can actually be enforced:
+`sys.getallocatedblocks()` is process-global rather than per-interpreter on
+both free-threaded and GIL builds, so there is no per-cell figure to cap.
+Worker sizing is the control.
+
+### What it costs
+
+Measured on the same 4-core container as the figures above, free-threaded
+CPython 3.14:
+
+| Operation | p50 |
+| --- | --- |
+| worker spawn (fresh interpreter + pyisolate import + pool) | 160.8 ms |
+| first cell for a tenant (spawns its worker) | 190.3 ms |
+| further cells in that worker | 29.8 ms |
+| `exec` + `recv` round trip | 1.20 ms |
+
+So the kill domain costs about **0.4 ms per round trip** over an in-process
+cell (1.20 ms against 0.82 ms) plus one worker spawn per tenant, which
+`prewarm` moves off the request path.
+
+### What it is still not
+
+A guest that escapes its cell owns its worker, and a worker is an ordinary
+process holding the supervisor's privileges. The fabric is **not** a boundary
+against hostile Python. For untrusted code use `backend="process"` — one
+confined process per sandbox — or a microVM. What the fabric buys is that
+trusted-but-independent tenants cannot wedge each other, and that a tenant
+which wedges itself is recoverable.
+
+---
+
 ## Canonical execution model
 
 A cell is intentionally limited to seven operations: `exec`, `call`, `post`, `recv`, `log`, `metric`, and `request`.

ROADMAP.md

@@ -10,7 +10,9 @@ normative statement.
 - **Backends** — `backend="thread"` (execution cell; a dedicated thread of the
   supervisor process, previously spelled `subinterpreter`),
   `backend="subinterpreter"` (real CPython sub-interpreter cells on 3.14+, with
-  a pre-warmed pool; an execution cell, not a boundary), and
+  a pre-warmed pool; an execution cell, not a boundary), `backend="fabric"`
+  (those cells hosted in worker processes, with tenant-aware placement, worker
+  memory caps, and kill-and-replace recovery — the multi-tenant mode), and
   `backend="process"` (the boundary mode): a real separate-process boundary with
   `no_new_privs` + a seccomp deny-list, Landlock filesystem rules, Landlock
   TCP-egress rules (ABI ≥ 4), a coarse per-cgroup eBPF/LSM deny-mask, and
@@ -33,17 +35,21 @@ normative statement.
 
 ## Now / next
 
-- **A kill domain for cells** — a running sub-interpreter cannot be reclaimed:
-  `close()` refuses while the guest is executing and there is no `kill`, so a
-  cell that overruns is abandoned and its thread is pinned for the life of the
-  process. The fix is not in-process. Add a layer of pre-forked worker
-  processes between the supervisor and the cells, size them by tenant, and make
-  the worker the unit that gets killed and replaced. This is what turns the cell
-  pool into something that survives a hostile-by-accident tenant.
-- **Per-cell resource accounting** — there is none today.
-  `sys.getallocatedblocks()` is process-global on both free-threaded and GIL
-  builds, so memory has to be capped at the worker/cgroup level rather than per
-  cell. Cells currently enforce a wall-time deadline and nothing else.
+- **Broker request execution for the fabric** — a fabric cell's `request` op
+  surfaces to the supervisor like the other backends', and still nothing
+  executes it. The fabric is where mediation matters most, because a worker
+  holds many tenants' cells: the handler has to be capability-scoped per cell,
+  not per worker.
+- **Kernel confinement of fabric workers** — a worker is currently an ordinary
+  process with the supervisor's privileges. With `tenant_isolation=True` a
+  worker serves one tenant, so its policy is unambiguous and the process
+  backend's seccomp/Landlock/cgroup layers could be applied to it at spawn.
+  That would make the fabric a defence-in-depth boundary rather than only a
+  fault boundary.
+- **Fabric admission from policy** — routing is explicit today
+  (`backend="fabric"`, `tenant=...`). Deriving placement and backend choice
+  from the policy's import list would let a tenant that needs numpy land on
+  `process` automatically instead of failing at import time.
 - **Broker request execution** — the `request` op currently surfaces a
   `BrokerRequest` to the host but nothing executes it or returns a result. Add a
   request/response round-trip and a pluggable, capability-scoped handler so the