Chunked media uploads to bypass Vercel's 4.5 MB request limit#408
Open
carlosjdelgado wants to merge 12 commits into
Open
Chunked media uploads to bypass Vercel's 4.5 MB request limit#408carlosjdelgado wants to merge 12 commits into
carlosjdelgado wants to merge 12 commits into
Conversation
carlosjdelgado
changed the title
carlosjdelgado
force-pushed
the
feat/chunked-media-upload
branch
from
2d16dab to
9614425
Compare
carlosjdelgado
changed the title
carlosjdelgado
changed the title
Vercel serverless functions cap request bodies at 4.5 MB, which after base64 overhead limited media uploads to ~3.3 MB. The browser now slices files into ~3 MB chunks, POSTs each to /api/upload/chunk (multipart), and then calls /api/upload/finalize which reassembles, pushes to GitHub, and deletes the chunks. Chunks are staged in a new upload_chunk table (Postgres text, base64). Stale chunks are reaped opportunistically on each insert. Max file size is 50 MB / 50 chunks, configurable in both endpoints and the client. githubSaveFile is extracted to lib/utils/github-save-file.ts so both the existing files endpoint and the new finalize endpoint share rename-on- conflict logic.
Client now uploads chunks in batches of 4 instead of sequentially. Server moves opportunistic stale-chunk cleanup (chunk endpoint) and per-upload chunk deletion (finalize endpoint) into next/server `after`, so neither blocks the response.
When the file fits in Vercel's 4.5 MB request body (after base64 overhead and JSON envelope), upload directly via the existing files endpoint instead of the chunked path. Cuts DB write/read traffic by the share of small uploads, which is most of them in practice.
The last chunk is sent inline with the finalize metadata (multipart) instead of going through the DB, saving one INSERT+SELECT per upload. For a 4 MB file (2 chunks) this halves DB writes; for larger files the ratio drops but every saved chunk still helps under Neon Free quotas.
Eliminates the +33% base64 overhead in the upload_chunk table. The
INSERT/SELECT bandwidth per chunk drops by ~25% with no client change
and no CPU cost. Migration uses decode('base64') in USING so any chunks
in flight at upgrade time are converted instead of dropped.
For a 20 MB upload, total DB bandwidth goes from ~48 MB to ~36 MB
(2.4x -> 1.8x file size).
Multipart carries the chunk as raw binary, not base64, so 4 MB fits in Vercel's 4.5 MB body with about 500 KB of headroom. For a 20 MB upload this drops the chunk count from 7 to 5 (4 staged in DB, 1 inline) and total DB bandwidth from ~36 MB to ~32 MB.
The first chunk is always CHUNK_BYTES (4 MB) for any multi-chunk upload; the last one can be smaller. Sending the first inline keeps the largest chunk out of the DB. For files whose size is a multiple of CHUNK_BYTES there is no change; for the rest, DB bandwidth drops by up to (CHUNK_BYTES - lastChunkSize) * 2.
Limits per upload to 22 MB of DB bandwidth in the worst case, sized to fit Neon Free quotas with margin for other traffic.
Merges the old 0013 (CREATE TABLE with text data) and 0014 (ALTER to bytea) into a single 0013 that creates the table with bytea directly. Also drops the unused chunk-assembly self-check script.
Files <=3 MB previously skipped the chunked path and posted base64+JSON to the files endpoint. Removed because the chunked path already short-circuits to inline-only when totalChunks=1, no DB rows are created, and the request body is smaller (binary multipart vs base64 JSON). One code path now handles every size up to 15 MB.
carlosjdelgado
force-pushed
the
feat/chunked-media-upload
branch
from
ad2b11f to
543c2d1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Vercel serverless functions cap request bodies at 4.5 MB. After base64 encoding and the JSON envelope, media uploads through
/api/[owner]/[repo]/[branch]/files/[path]hitFUNCTION_PAYLOAD_TOO_LARGEat ~3.3 MB of real file size. Users get a hard wall on anything bigger.Solution
A single new upload path that slices the file into 4 MB binary chunks:
/api/upload/finalize./api/upload/chunk, which stages them in a newupload_chunktable./api/upload/finalizereads any staged chunks back, concatenates them with the inline chunk, and pushes the file to GitHub via the existing token flow.Files ≤ 4 MB use the same code path but with zero staged chunks: the entire file rides inline and the DB is never touched. Files > 4 MB stage
N-1chunks in the DB. The GitHub token never leaves the server, and no new infrastructure is required — the existing Postgres database is the buffer.Why first chunk inline (not last)
chunk 0is always exactly 4 MB; the last chunk may be smaller. Sending the largest chunk inline keeps the most bytes out of the DB. For files whose size is a multiple of 4 it makes no difference; for the rest, DB bandwidth drops by up to(FILE_MB - 4) × 2.Storage details
datacolumn isbytea(no base64 overhead in the table).next/server'safter()so the response is not blocked:/api/upload/finalizedeletes the chunks of the upload it just consumed and all the stale chunks older than 10 minutes.Limits
Refactor
githubSaveFile(with its rename-on-conflict logic) was extracted fromapp/api/[owner]/[repo]/[branch]/files/[path]/route.tsintolib/utils/github-save-file.tsso the existing files endpoint and the new finalize endpoint share the same write path.Migration
A single new migration (
0013_upload_chunks.sql) creates theupload_chunktable directly withbytea. It runs on Vercel'spostbuildhook with the rest.Test plan
upload_chunkrows created; file lands on the target branchupload_chunkrows created; whole file rides inline in the finalize request