Skip to content

Add ComfyStream live-runner for go-livepeer orchestrators - #594

Draft
eliteprox wants to merge 28 commits into
mainfrom
feat/comfypeer-liverunner
Draft

eliteprox wants to merge 28 commits into
mainfrom
feat/comfypeer-liverunner

Conversation

@eliteprox

Copy link
Copy Markdown
Collaborator

Summary

  • Add a Livepeer live-runner path that registers ComfyStream as app comfystream (capacity 1) against a go-livepeer orchestrator with -useLiveRunners, driving comfystream.Pipeline in-process (no BYOC/pytrickle subprocess).
  • Expose an agent-shaped HTTP surface for video-in→text-out and live trickle sessions: POST /analyze, POST /start_stream, POST /update_stream, GET /text, GET /healthz.
  • Ship Docker packaging (Dockerfile.live-runner, docker-compose.live-runner.yml, --live-runner entrypoint), optional .[live-runner] deps, a smoke client, and a stub analyze workflow.
  • Also includes supporting workspace/ComfyUI cwd cleanup (--workspace / --cwd), constraint/override-based node installs, and CI workflow fork-restriction removals needed to operate from this branch.

Motivation

ComfyPeer / live-runner work wants ComfyStream sellable as a session wall-clock / latency runner on an existing orchestrator—same registration + trickle path as the transcode live-runner—rather than a batch $/image race or legacy BYOC.

What's included

Live-runner

Piece Role
server/live_runner.py Register runner, create trickle channels, run Pipeline sessions
server/live_runner_client.py Smoke client: reserve → analyze/stream → update → stop
docker/Dockerfile.live-runner Overlay image on livepeer/comfystream + livepeer-gateway
docker-compose.live-runner.yml Host-network attach to an already-running orchestrator
docker/entrypoint.sh --live-runner → python server/live_runner.py …
workflows/comfystream/analyze-stub-api.json Minimal video→text stub for bring-up
pyproject.toml Optional live-runner extra

Supporting (earlier on branch)

  • Normalize ComfyUI workspace handling (--workspace, config passthrough, client init)
  • Script/dependency hygiene (constraints.txt, overrides.txt, setup_nodes / build_trt)
  • Remove GitHub Actions repo-owner guards so fork CI can run

Legacy BYOC (server/byoc.py) is unchanged.

Quick test

# Attach to an orchestrator that already has -useLiveRunners
docker compose -f docker-compose.live-runner.yml up -d --build
curl -sk "$LIVEPEER_ORCH_URL/discovery" | jq '.[].runners[].app'

pip install '.[live-runner]'
python server/live_runner_client.py sample.mp4 \
  --workflow workflows/comfystream/analyze-stub-api.json \
  --discovery "$LIVEPEER_ORCH_URL/discovery"

Test plan

  • Compose build succeeds; runner appears in orchestrator discovery as comfystream
  • Smoke client analyze against stub workflow returns text
  • start_stream + frame publish produces trickle video (and optional text)
  • update_stream applies mid-session workflow/prompt change
  • Orchestrator session release cleans up active session (capacity returns to 1)
  • /healthz healthy while idle and during a session
  • Confirm secrets/host defaults in docker-compose.live-runner.yml are overridden via env (no committed orch secrets in final merge)
  • Existing WebRTC / BYOC paths still work; workspace --config still preserves --workspace

Notes / follow-ups

  • Depends on livepeer-gateway branch ja/live-runner until that lands on a tagged release.
  • Compose file currently carries host-specific GPU UUID / orch URL defaults—should be env-only before merge.
  • Protobuf: live-runner image installs protobuf>=6.31.1 for gateway pb2; call out any ComfyUI pin tension in review.

Copilot AI and others added 14 commits January 10, 2026 06:42
Co-authored-by: eliteprox <16746274+eliteprox@users.noreply.github.com>
Remove repository restrictions from GitHub workflows to enable fork operation
- Standardize on --workspace (with --cwd alias) so ComfyStream can launch from any directory while mapping to the correct ComfyUI cwd.
- Pass the workspace/CWD and logging settings cleanly through server args → pipeline → ComfyStreamClient so ComfyUI’s configuration is honored.
- Add config passthrough support: when a ComfyUI config file is provided, forward it directly and skip overriding flags to match ComfyUI’s precedence rules.
- Ensure ComfyUI packages load correctly via the client initialization (__init__.py/ComfyStreamClient), removing legacy env handling and relying on proper cwd/config inputs.
- Update Docker, supervisord, launch configs, and docs to use the unified workspace flag and defaults.
This reverts commit 6853675.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: eliteprox <16746274+eliteprox@users.noreply.github.com>
Preserve workspace cwd when using --config in frame_processor
- Cleaned up __init__.py by removing unused imports and defining __all__.
- Introduced overrides.txt for managing package dependencies with specific versions and extra index URLs.
- Updated setup_models.py and setup_nodes.py to improve configuration handling and error reporting during node installations.
- Enhanced utils.py with new functions for workspace management and improved config path resolution.
…ng and dependency management

- Refactored build_trt.py to include a setup_comfy function for better workspace initialization and module imports.
- Added a workspace argument to build_trt_engine for flexibility in specifying the ComfyUI workspace.
- Updated constraints.txt to pin specific versions of onnxruntime and huggingface-hub, and added a new dependency for transformers.
- Modified setup_nodes.py to utilize a centralized constraints file for pip installations, ensuring consistent dependency management across custom nodes.
… dependency management and module imports

- Added import for comfy.sd in build_trt.py to streamline module access.
- Updated constraints.txt to include a new dependency on nvidia-modelopt with a minimum version requirement.
- Modified setup_nodes.py to handle nvidia-modelopt version conflicts by stripping unnecessary extras from the package line.
- Introduced docker-compose.live-runner.yml to facilitate running ComfyStream as a live-runner connected to an existing Livepeer orchestrator.
- Created Dockerfile.live-runner to build the necessary environment for the live-runner, including dependencies for livepeer-gateway.
- Updated entrypoint.sh to handle the new --live-runner flag for starting the service.
- Added server/live_runner.py to implement the live-runner functionality, enabling video analysis and live streaming.
- Developed server/live_runner_client.py as a smoke client for testing the live-runner features.
- Enhanced README.md with instructions for using the live-runner and added a new workflow JSON for analysis.
- Updated pyproject.toml to include live-runner dependencies.
- Ensured proper integration with Livepeer's API for session management and media handling.
Comment thread server/live_runner.py Fixed
Comment thread server/live_runner.py Fixed
Comment thread server/live_runner.py Fixed
The room can attach to the orchestrator, drive ws_stream, and load the passthrough workflows without going through the fal batch routes.
Commit the route index, schema bytes, and operator overlay so schema hashes stay stable without copying OpenAPI documents into the image.
Batch jobs run FalSubmit and FalCollect in an isolated process pool, and SaveFalResult gives ComfyUI the output node those graphs were missing.
Install stream-pack and fal-api only, keep a GCC 14 libstdc++ so av can load, and fail the image if node setup does not finish.
Comment thread server/webrtc_trickle_bridge.py Fixed
Comment thread server/webrtc_trickle_bridge.py Fixed
Log the traceback server-side, accept payment-ticket headers up to 256KB, and pin setuptools below 82 so ComfyUI tests can import pkg_resources.
@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.26667% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.79070%. Comparing base (06b2a27) to head (5443744).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
server/live_runner_batch_client.py 27.47253% 66 Missing ⚠️
test/conftest.py 50.00000% 7 Missing ⚠️
test/test_batch_http.py 99.30556% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                  Coverage Diff                  @@
##                 main        #594          +/-   ##
=====================================================
- Coverage   100.00000%   82.79070%   -17.20930%     
=====================================================
  Files               1           6           +5     
  Lines              55         430         +375     
=====================================================
+ Hits               55         356         +301     
- Misses              0          74          +74     
Files with missing lines Coverage Δ
test/test_fal_catalog.py 100.00000% <100.00000%> (ø)
test/test_job_pool.py 100.00000% <100.00000%> (ø)
test/test_batch_http.py 99.30556% <99.30556%> (ø)
test/conftest.py 50.00000% <50.00000%> (ø)
server/live_runner_batch_client.py 27.47253% <27.47253%> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e9dace...5443744. Read the comment docs.

Files with missing lines Coverage Δ
test/test_fal_catalog.py 100.00000% <100.00000%> (ø)
test/test_job_pool.py 100.00000% <100.00000%> (ø)
test/test_batch_http.py 99.30556% <99.30556%> (ø)
test/conftest.py 50.00000% <50.00000%> (ø)
server/live_runner_batch_client.py 27.47253% <27.47253%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- Added support for isolated realtime pipelines in the live-runner, allowing concurrent processing of video streams with dedicated GPU resources.
- Introduced a new configuration file `configs/realtime.yaml` to define pipeline specifications for `flux-klein` and `sd-turbo`.
- Updated `docker-compose.live-runner.yml` to expose necessary GPU configurations and environment variables for realtime processing.
- Enhanced `live_runner.py` to load and manage realtime pipeline specifications, integrating them with the existing live-runner functionality.
- Created `realtime_client.py` for sending video files to the realtime apps, supporting dynamic prompts and presets.
- Added new backend implementations for realtime processing, including `flux_klein_backend.py` and `comfy_backend.py`, to handle specific model requirements.
- Implemented fallback visuals for paused or failing sessions to maintain user experience during processing interruptions.
- Updated Dockerfile to include new dependencies and configurations for the realtime features.
- Added new launch configurations for debugging `sd-turbo` and `flux-klein` in VS Code.
- Created `realtime-integration.md` to document the integration process, including usage instructions and parameter details for both models.
- Updated `realtime_client.py` to allow the `TOKEN` to be set via environment variable, improving usability.
- Ensured that the client raises an error if the `TOKEN` is not provided, enhancing error handling.
- Removed outdated launch configurations for `sd-turbo` and `flux-klein` from VS Code settings.
- Added `diffusers==0.39.0` dependency to the Dockerfile for enhanced model support.
- Cleaned up Dockerfile by removing unnecessary Python import checks.
- Deleted `realtime_client.py` and `realtime-integration.md` as part of the refactor, streamlining the codebase.
- Improved error handling in `PipelineWorker` to manage session release failures more gracefully.
- Updated `docker-compose.live-runner.yml` to reflect changes in the registration of `flux-klein` and `sd-turbo` apps.
- Modified `README.md` to clarify the removal of static entries from the orchestrator's configuration.
- Introduced a new `capacity.py` module to manage GPU resource allocation for realtime pipelines, enforcing limits on concurrent loads.
- Updated `realtime.yaml` to include capacity settings, allowing for better management of GPU resources.
- Refactored `load_realtime_specs` to return both pipeline specifications and capacity policies.
- Added `flux_klein_stream.py` to implement the FLUX.2 Klein feedback loop as a ComfyStream node.
- Enhanced `PipelineWorker` and `RealtimeSupervisor` to utilize the new capacity management features, improving resource handling during pipeline execution.
- Updated tests to validate the new capacity management logic and ensure proper functionality of the realtime pipelines.
- Adjusted `realtime.yaml` to set a default `session_idle_timeout_s` of 15 seconds for improved session management.
- Modified `api.py` to include additional parameters in the `validate_params` function for better validation.
- Updated `spec.py` to reflect the new default value for `session_idle_timeout_s`.
- Enhanced `worker.py` to synchronize session capacity management and handle session timeouts more effectively.
- Added tests in `test_realtime.py` to verify the correct handling of session expirations and capacity updates.
- Every realtime pipeline registers. Warm pipelines that fit capacity load at
  boot; the rest start cold. A stream for a cold pipeline asks the supervisor
  for a slot, which evicts loaded pipelines without sessions and waits for
  their processes to exit before the requester loads.
- Workers report state and session counts to the supervisor, which advertises
  "busy" for any pipeline that could not load now (hidden from discovery and
  routing) and "ready" once its blocker goes idle.
- Pipelines may hold several sessions up to their backend's SESSION_LIMITS;
  the worker keeps per-session state and eviction skips any active session.
- The 3090 Ti holds one pipeline: flux-klein warm, sd-turbo cold. The RTX 4000
  Ada is left to go-livepeer transcoding.
- Install livepeer-gateway with registration update(), note_session_ended()
  and status (v1.0.0 has neither of the first two).
- Updated `docker-compose.live-runner.yml` to allow FAL_KEY to be sourced from environment variables, sibling `.env` files, or a specified fleet file.
- Modified `live_runner.py` to improve FAL_KEY loading logic, providing clearer feedback on the source of the key.
- Introduced a new `fal_key.py` module to encapsulate FAL_KEY resolution and application logic, enhancing maintainability.
- Updated `job_pool.py` to utilize the new FAL_KEY handling mechanism.
- Added comprehensive tests for the FAL_KEY functionality to ensure robust key resolution and application.
Serve convert, compress, trim, resize, gif, extract-audio, mute, and rotate from a process pool that does not share the live stream pipeline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants