Offload traffic to static workers and use node as the proxy by freddyaboulton · Pull Request #13366 · gradio-app/gradio

freddyaboulton · 2026-05-05T21:57:57Z

Description

Similar to #13351 but using the built-in node server for ssr mode as the proxy

In SSR Mode, node will now launch in front of the python server and not the other way around
node will proxy requests for static routes to the background workers. node will continue serving the requests for the ssr-related assets as well
This bypasses the Node proxy in the python server and offers huge speedups
Working with gr.Server + ZeroGPU (space) and gr.Blocks + ZeroGPU (space)

Architecture

  flowchart TD
      Client[Client / Browser]
      Client --> Node["<b>node :7860</b><br/>reverse proxy"]

      Node -- "/upload, /file=, /static,<br/>/assets, /svelte, /favicon.ico" --> Pool
      Node -- "/queue/*, /api/*, SSE" --> Main
      Node -- "/, /_app/*" --> node

     subgraph Pool["Static Workers (round-robin)"]
          W1["Worker 1 :7862<br/>uploads, downloads, static assets"]
          W2["Worker 2 :7863<br/>uploads, downloads, static assets"]
      end

      Main["<b>Main Server :7861</b><br/>queue, SSE, API, session state, ML inference"]

      Pool --> FS
      Main --> FS
      FS[("Shared filesystem<br/>/tmp/gradio/")]

AI Disclosure

We encourage the use of AI tooling in creating PRs, but the any non-trivial use of AI needs be disclosed. E.g. if you used Claude to write a first draft, you should mention that. Trivial tab-completion doesn't need to be disclosed. You should self-review all PRs, especially if they were generated with AI.

I used AI to... [fill here]
I did not use AI

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Testing and Formatting Your Code

PRs will only be merged if tests pass on CI. We recommend at least running the backend tests locally, please set up your Gradio environment locally and run the backed tests: bash scripts/run_backend_tests.sh
Please run these bash scripts to automatically format your code: bash scripts/format_backend.sh, and (if you made any changes to non-Python files) bash scripts/format_frontend.sh

Note

High Risk
High risk because it changes the SSR launch/network architecture (Node becomes the user-facing reverse proxy) and introduces multi-process static/upload serving, which can affect routing, ports, and file security behavior.

Overview
In SSR mode, switches to a new Node-as-front-proxy architecture: Node now binds the user-facing port and proxies requests to an internal Python port, with optional round-robin routing of static/upload/file routes to a background worker pool.

Adds num_workers/GRADIO_NUM_WORKERS to start per-process static servers (StaticWorkerPool) that handle /static, /assets, /svelte, /upload, and /file* traffic, and refactors shared file/upload helpers into route_utils so both the main app and workers reuse the same safe path/mimetype/range handling.

Updates the Node build/runtime to include a custom proxy entrypoint (via http-proxy) with routing logic + unit tests, adjusts SSR startup/health checks/logging and shutdown cleanup, expands server mode to propagate SSR/proxy env, and removes legacy Python→Node proxy middleware/caching. Also tweaks profiling summaries to separate upload timing from prediction traces and deletes the old benchmark tooling directory.

^{Reviewed by Cursor Bugbot for commit 6c71719. Bugbot is set up for automated code reviews on this repo. Configure here.}

gradio-pr-bot · 2026-05-05T21:58:45Z

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
`@self/app`	`minor`
`@self/spa`	`minor`
`gradio`	`minor`

Offload traffic to static workers and use node as the proxy

✅ Changeset approved by @freddyaboulton

Maintainers can remove approval by unchecking this checkbox.

Something isn't right?

Maintainers can change the version label to modify the version bump.
If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

freddyaboulton · 2026-05-06T20:53:49Z


        import numpy as np

+        prediction_traces = [


Needed for the benchmarking. Need to separate the upload trace since it doesn't have the rest of the lifecycle stages (pre/postprocess e.g)

freddyaboulton · 2026-05-06T20:55:33Z

 import functools
 import hashlib
 import hmac
+import importlib.resources


There is a lot of code dupe between server.py and static_server.py. Consolidated the overlap into util functions here. The util functions cover file i/o (like serving assets from the various build dirs)

freddyaboulton · 2026-05-06T20:59:02Z

@@ -0,0 +1,101 @@
+import http from "node:http";


This is the node proxy. it gets copied to the build directory to replace the index.js that is generated by sveltekit. The handler is generated by sveltekit and its how we serve the sveltekit page.

freddyaboulton · 2026-05-06T20:59:36Z

 		"ts:check": "svelte-check --tsconfig tsconfig.json --threshold error",
 		"test": "pnpm --filter @gradio/client build && vitest dev --config .config/vitest.config.ts",
-		"test:run": "pnpm --filter @gradio/client build && vitest run --config .config/vitest.config.ts",
+		"test:run": "pnpm --filter @gradio/client build && vitest run --config .config/vitest.config.ts && node --test js/app/proxy_routes.test.js",


Added the proxy unit tests to test:run command

freddyaboulton · 2026-05-06T21:00:04Z

@@ -1,24 +0,0 @@
-"""Audio generation app — exercises file caching and downloading on output."""


Moving everything here to gradio-app/hf-perftest repo

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 483df89. Configure here.}

cursor · 2026-05-06T21:05:28Z

+        max_fields=1000,
+        max_file_size=max_file_size,
+        upload_id=upload_id,
+    )


Upload progress tracking completely broken after refactoring

High Severity

The refactored upload_fn does not pass upload_progress to GradioMultiPartParser, and the calling code in routes.py no longer calls file_upload_statuses.track(upload_id) before invoking it. The GradioMultiPartParser uses upload_progress to report chunk-level progress, and file_upload_statuses.track() is required for the SSE-based /upload_progress endpoint to work. This breaks client-side upload progress bars entirely.

Additional Locations (1)

gradio/routes.py#L1733-L1740

^{Reviewed by Cursor Bugbot for commit 483df89. Configure here.}

cursor · 2026-05-06T21:05:28Z

+        except MultiPartException as exc:
+            code = 413 if "maximum allowed size" in exc.message else 400
+            return PlainTextResponse(exc.message, status_code=code)
+        return output_files


Static worker silently loses files on cross-filesystem uploads

Medium Severity

The static worker's upload handler calls upload_fn with force_move=False but discards the files_to_copy and locations return values (using _, _). When os.rename fails (e.g., cross-filesystem temp-to-upload move), files remain at the temp location while the returned paths point to destinations that were never written. Subsequent file downloads will 404.

^{Reviewed by Cursor Bugbot for commit 483df89. Configure here.}

abidlabs · 2026-05-12T13:14:33Z

Very cool @freddyaboulton! Will review this.

Would it be possible to include benchmarks that compare an app built with gr.Server + new workers approach to the most analogous vanilla FastAPI app? My understanding is we should see performance improvements even compared to FastAPI app, which would be a big win

freddyaboulton · 2026-05-14T14:24:02Z

@abidlabs I'm working on a FastAPI comparison now. Main thing is ensuring the comparison is apples-to-apples since gradio does a lot of things FastAPI doesnt like SSR and queueing.

freddyaboulton and others added 30 commits March 30, 2026 17:45

Profile the upload route

35dc855

add changeset

f4d197c

Fix

993a5d7

Add max_threads + async

723bcfd

add changeset

8bfb618

use decorator for smaller diff

5b9653c

Add static servers

5762ba9

Streaming Proxy

90a515d

Kill servers

7ff4f8f

Fix

d62254b

Add code

8a6ab2e

better cascading error

8b1c83c

use a redirect again

db33d61

Runner file

4ce3b68

Fix code

6fb9976

Disable redirect middleware

6fcc62c

merge main

693d0c7

Merge branch 'main' into multiprocess-gradio-test

1951b8b

profiling-fix

e62a43f

Merge branch 'main' into multiprocess-gradio-test

ab6dedd

Merge branch 'main' into multiprocess-gradio-test

2edf008

add changeset

76e6caa

Lint

1c59f9b

Proxy-to-Node

92051f0

Addc code

ccce0e7

Add code

c19f0d3

logging

649fb79

Remove old code

fade5f7

Refactor

9d7f637

Add code

2e4c248

freddyaboulton and others added 11 commits May 6, 2026 09:34

Fix code

e812658

Fix

edc0538

Some formatting

c28d712

Remove benchmarking

e1433f7

Docs

79afb79

Add python unit test

3dec4b4

Lint + tests

c174a99

add changeset

656a1c7

exclude

20a422a

Add code

b64c8d4

Fix

73f5db1

freddyaboulton marked this pull request as ready for review May 6, 2026 17:22

freddyaboulton requested review from abidlabs, cbensimon, dawoodkhan82, hannahblair and pngwn May 6, 2026 17:22

Fix close event

dbf36d5

freddyaboulton commented May 6, 2026

View reviewed changes

cursor Bot reviewed May 6, 2026

View reviewed changes

Comment thread gradio/routes.py

Comment thread gradio/blocks.py

freddyaboulton commented May 6, 2026

View reviewed changes

remove logger

483df89

cursor Bot reviewed May 6, 2026

View reviewed changes

Merge branch 'main' into proxy-to-node

6c71719

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offload traffic to static workers and use node as the proxy#13366

Offload traffic to static workers and use node as the proxy#13366
freddyaboulton wants to merge 45 commits into
mainfrom
proxy-to-node

freddyaboulton commented May 5, 2026 •

edited by cursor Bot

Loading

Uh oh!

gradio-pr-bot commented May 5, 2026 •

edited

Loading

Something isn't right?

Uh oh!

freddyaboulton May 6, 2026

Uh oh!

Uh oh!

Uh oh!

freddyaboulton May 6, 2026

Uh oh!

freddyaboulton May 6, 2026

Uh oh!

freddyaboulton May 6, 2026

Uh oh!

freddyaboulton May 6, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot May 6, 2026

Uh oh!

cursor Bot May 6, 2026

Uh oh!

abidlabs commented May 12, 2026

Uh oh!

freddyaboulton commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1,24 +0,0 @@
		"""Audio generation app — exercises file caching and downloading on output."""

Conversation

freddyaboulton commented May 5, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Architecture

AI Disclosure

🎯 PRs Should Target Issues

Testing and Formatting Your Code

Uh oh!

gradio-pr-bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦄 change detected

This Pull Request includes changes to the following packages.

Something isn't right?

Uh oh!

freddyaboulton May 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

freddyaboulton May 6, 2026

Choose a reason for hiding this comment

Uh oh!

freddyaboulton May 6, 2026

Choose a reason for hiding this comment

Uh oh!

freddyaboulton May 6, 2026

Choose a reason for hiding this comment

Uh oh!

freddyaboulton May 6, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot May 6, 2026

Choose a reason for hiding this comment

Upload progress tracking completely broken after refactoring

Uh oh!

cursor Bot May 6, 2026

Choose a reason for hiding this comment

Static worker silently loses files on cross-filesystem uploads

Uh oh!

abidlabs commented May 12, 2026

Uh oh!

freddyaboulton commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

freddyaboulton commented May 5, 2026 •

edited by cursor Bot

Loading

gradio-pr-bot commented May 5, 2026 •

edited

Loading