Rename backend="subinterpreter" to backend="thread" - #294
Merged
Merged
Conversation
pyisolate/runtime/thread.py has always run guests in a threading.Thread, exec'ing guest source against a restricted __builtins__ mapping. The backend that selects it was named for an implementation it does not have, which makes the mechanism impossible to read off the API: nothing in the name tells you that sys.modules is shared with the supervisor rather than per-interpreter, which is the difference that decides what isolation you are getting. The runtime gets its real name. "subinterpreter" still resolves, with a DeprecationWarning, through a DEPRECATED_BACKEND_ALIASES map exported next to SUPPORTED_BACKENDS. The alias is deliberately not a permanent synonym. The warning tells callers who want today's behaviour to pass "thread", because the name is reserved for a real CPython sub-interpreter backend, and reassigning it later must not silently change what existing callers get. No behaviour changes: the default backend selects the same runtime it did before, and the boundary claim is untouched. A thread and a sub-interpreter are equally not a boundary against hostile Python, which the docs already said; what they no longer say is that the backend is named for an implementation it is waiting on. The README section is rewritten around the rename, and the ROADMAP item that proposed this rename now owns the real implementation instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fs1hTtmF4Hm9h617AG9Gse
backend="subinterpreter" now runs each guest in its own CPython interpreter via concurrent.interpreters, rather than naming a runtime it did not have. Each cell gets its own sys.modules and its own builtins, which is the point: the import allow-list stops being thread-local bookkeeping inside one shared interpreter and becomes a property of the interpreter, so one tenant's imports, monkey-patches and globals cannot be observed or clobbered by another. It is still not a boundary against hostile Python -- a cell shares the supervisor's address space and ctypes imports cleanly inside one -- and the docs say so in the same places they always have. Three measured properties of the CPython primitive shape the design: * Creating a cell costs 10-57ms and 3.5-13MiB depending on its import surface, against 0.8ms to dispatch onto a warm one. So cells are pooled and pre-warmed by CellPool, keyed by a CellSpec, and a request never pays for creation when a cell warmed the same way exists. * An interpreter cannot be reset. Releasing a cell therefore retires it and warms a replacement in the background; returning it to the pool would carry one tenant's globals into the next. * A running interpreter cannot be reclaimed. close() refuses while the guest is executing, and an async exception aimed at the thread does not reach it. A cell that overruns its deadline is abandoned, not killed: the sandbox raises WallTimeExceeded saying so, the pool counts it, and the thread stays pinned until the process exits. The API does not pretend otherwise -- kill() returns False rather than claiming a stop it cannot perform. Requires 3.14+ and fails closed below it with a diagnostic naming the alternatives, rather than degrading to the thread backend, which isolates differently. The private _interpreters on 3.12/3.13 is not used as a fallback: destroying interpreters that imported http.client or email.message aborts the process there, which is exactly the workload a pool generates. Two hazards found while building this, both fixed here rather than documented as quirks: * Messages cross as JSON in a single str. A cross-interpreter queue falls back to pickle for anything not natively shareable, which would have the supervisor unpickling bytes the guest produced -- the thing the process backend explicitly refuses to do. A str is natively shareable, so the fallback never engages. It also means a guest posting something unserialisable gets a TypeError at the call rather than an opaque NotShareableError from the runtime. * retire() asks the runtime whether the interpreter is running before closing it. Losing that race is not a catchable error: CPython aborts the process with "Py_EndInterpreter: not the last thread", which would take every other tenant down with the lost cell. The import hook also has to let CPython's own machinery import pickle and traceback. Blocking those does not produce a policy denial; it produces NotShareableError from two layers down. The allow-list gates guest imports; starving the interpreter's plumbing is not a security control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fs1hTtmF4Hm9h617AG9Gse
The sub-interpreter backend needs 3.14+, and nothing in the matrix ran it: unit tests stopped at 3.13 and the only free-threaded job was 3.13t, which cannot create a cell at all. Add 3.14 to the unit matrix, and a `sub-interpreter cells / py3.14t` job that runs the backend suite on a free-threaded build. That job asserts up front that the interpreter really is a free-threaded 3.14 before running any tests. Every sub-interpreter test skips itself when the build cannot run it -- which is correct for the 3.11-3.13 matrix, but it means a runner that silently resolved to an older Python would skip everything and report the job green having tested nothing. Failing loudly is the difference between a job that covers the backend and a job that looks like it does. Dependency installation follows the existing 3.13t job: install without the runtime deps that may not have free-threaded wheels, then re-add the pure-Python ones the import path needs. Nothing in this job's scope needs real crypto, and the suite's stub covers it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fs1hTtmF4Hm9h617AG9Gse
…-ci-freethreaded Cover CPython 3.14 and the sub-interpreter backend in CI
…-subinterpreter-backend Add a real sub-interpreter backend with a pre-warmed cell pool
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of a series of independent PRs. Touches different files from #292 and #293 except for
CHANGELOG.md, where the entries land in different sections.Why
pyisolate/runtime/thread.pyhas always run guests in athreading.Thread,execing guest source against a restricted__builtins__mapping. The backend that selects it was named for an implementation it does not have.The repo is already candid about this — the README,
SECURITY.mdand the threat model all say so in prose. But prose is the wrong place for it. Nothing inbackend="subinterpreter"tells a caller thatsys.modulesis shared with the supervisor rather than per-interpreter, and that is exactly the difference that decides what isolation they are getting.What changes
backend="thread", and it is the default."subinterpreter"still resolves, with aDeprecationWarning, through aDEPRECATED_BACKEND_ALIASESmap exported next toSUPPORTED_BACKENDS.The alias is deliberately not a permanent synonym. The warning tells callers who want today's behaviour to pass
"thread":That matters for the next PR in the series: reassigning the name later must not silently change what existing callers get.
What does not change
No behaviour. The default backend selects the same runtime it did before, and the boundary claim is untouched — a thread and a sub-interpreter are equally not a boundary against hostile Python, which the docs already said. What they no longer say is that the backend is named for an implementation it is waiting on.
The ROADMAP item that proposed this rename ("…or rename the backend to
threadand let this item own the real thing") now owns the real implementation instead, and points at 3.14'sconcurrent.interpretersrather than the private_interpreters.Testing
Four new tests in
tests/test_supervisor.py:SUPPORTED_BACKENDS/IMPLEMENTED_BACKENDS;ValueErrorwithout warning.test_spawn_rejects_unknown_backendusedbackend="thread"as its stand-in for an unknown backend, which is now the default one — switched to a name that is genuinely not a backend.Full suite: 534 passed, 14 skipped. The one failure in my container,
test_apply_confinement_installs_seccomp_and_allows_normal_syscalls, is pre-existing and environmental (seccomp unavailable) and reproduces on unmodifiedmain.pre-commit run --all-filespasses.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fs1hTtmF4Hm9h617AG9Gse
Generated by Claude Code