Add streaming read/write API for Sandbox#135
Open
scotttrinh wants to merge 9 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
This PR adds a streaming file I/O API to the unstable Sandbox filesystem via box.fs.open(...), enabling chunked reads/writes for binary and text data (async + sync), and switches uploads to a bounded-memory tar+gzip streaming encoder with transfer validation and configurable request timeouts.
Changes:
- Introduces concrete streaming file-handle types (async + sync) with context-manager semantics and
open()overloads. - Implements streaming tar+gzip archive generation for uploads plus size/mode/chunk validation and new transfer-focused error types.
- Adds unit/live tests and updates unstable docs + example demonstrating streaming transfers.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unstable/test_sandbox_transfer_validation.py | Adds tests for transfer validation helpers, exported handle/error types, and timeout defaulting. |
| tests/unstable/test_sandbox_streaming_archive.py | Adds coverage for the bounded-memory tar+gzip encoder and async/sync archive iterators. |
| tests/unstable/test_sandbox_filesystem.py | Extends filesystem tests for streaming read/write behavior, response closing, and newline buffering. |
| tests/unstable/test_sandbox_api_client.py | Updates API client construction to include the new file-transfer timeout parameter. |
| tests/live/test_unstable_sandbox_live.py | Adds a live scenario test validating streaming upload/download parity across async/sync drivers. |
| tests/live/_unstable_scenarios.py | Implements streaming transfer scenario helpers using the new fs.open API. |
| src/vercel/unstable/sandbox/sync.py | Exposes new streaming handle types and transfer errors in the sync public unstable API. |
| src/vercel/unstable/sandbox/init.py | Exposes new streaming handle types and transfer errors in the async public unstable API. |
| src/vercel/unstable/README.md | Documents the new open() streaming API and provides examples. |
| src/vercel/_internal/unstable/sandbox/sync_runtime.py | Adds sync fs.open, switches reads to streamed responses, and writes to streamed archive uploads. |
| src/vercel/_internal/unstable/sandbox/sync_filesystem_handle.py | Implements sync streaming reader/writer handle classes and publish plumbing. |
| src/vercel/_internal/unstable/sandbox/streaming_archive.py | Adds bounded-memory tar+gzip streaming encoder and async/sync body generators with size checks. |
| src/vercel/_internal/unstable/sandbox/service.py | Adds write_archive + open_read_response service methods and wires file-transfer timeouts into the API client. |
| src/vercel/_internal/unstable/sandbox/runtime_common.py | Introduces upload entry model + tar-path normalization and validation helpers for sizes/modes/chunks. |
| src/vercel/_internal/unstable/sandbox/options.py | Adds file_transfer_timeout to service options with a 5-minute default. |
| src/vercel/_internal/unstable/sandbox/filesystem_handle_common.py | Adds shared handle helpers (option validation, read-size validation, text codecs). |
| src/vercel/_internal/unstable/sandbox/errors.py | Adds transfer-focused filesystem error types (size mismatch, transfer base error). |
| src/vercel/_internal/unstable/sandbox/async_runtime.py | Adds async fs.open, switches reads to streamed responses, and writes to streamed archive uploads. |
| src/vercel/_internal/unstable/sandbox/async_filesystem_handle.py | Implements async streaming reader/writer handle classes and publish plumbing. |
| src/vercel/_internal/unstable/sandbox/api_client.py | Adds streamed read responses, streamed upload bodies, and request timeouts for file transfers. |
| examples/unstable/sandbox_06_streaming_files.py | Adds an end-to-end example demonstrating streaming upload and download. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
elprans
reviewed
Jun 22, 2026
Reduces the amount of branching and makes it easier to later remove or reduce the functionality of the explicit-sized branch once we have a direct streaming backend.
9d0d664 to
3c19bbf
Compare
41dd901 to
5a060bf
Compare
Move byte-source normalization and staging-file ownership into shared runtime capabilities so Sandbox and upcoming services can reuse async-shaped business logic from both public runtimes. Update transport streaming request and response lifecycles with bounded backpressure, cleanup, and non-suspending sync adapters. Refactor Sandbox file handles, archive uploads, text readers, and process logs onto the new primitives.
5a060bf to
f8c451f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds streaming file access to the unstable Sandbox filesystem API through
box.fs.open. Both async and sync runtimes support reading and writing binary or text data without buffering entire files in memory. The implementation includes bounded-memory tar/gzip encoding for uploads, transfer validation, configurable timeouts and chunk sizes, and concrete file-handle types with familiar context-manager semantics.Note: since we currently must know the full size of the file before sending the data to the backend, we spool the contents to a
TemporaryFile, read the size, and then stream the data from that temp file to the backend. If you know the size of the content, you can skip this spooling by providing the exact byte size inopen(..., size=byte_size)which will stream directly to the server. We hope this requirement will go away soon, so we're mostly documenting the size-less API for now.Edit:
When looking at the implementation and anticipating what we'll need when adding Blob into the mix here, I decided that this work should also add an
iter_coroutinestyle byte streaming utility.transport.pynow supports incrementally produced request bodies and owned streaming responses.A new session-owned byte-stream "runtime" normalizes various byte sources behind one internal interface. It also provides disk-backed temporary byte files (using
tempfile.TemporaryFilefor sync sessions andanyio.TemporaryFilefor async sessions) for workflows that need seeking, size discovery, or mutation without buffering the complete object in memory which we'll also need in our upcoming Blob redesign.