Skip to content

README.md

Latest commit

 

History

History
365 lines (295 loc) · 14.7 KB

README.md

File metadata and controls

365 lines (295 loc) · 14.7 KB
Raw

Realtime Motion-to-Music (Web)

Browser-based real-time motion-to-music demo. A Python backend runs the GPU pipeline behind an HTTP + WebSocket server; a Next.js front-end renders the live UI:

  • Upload a source audio file, get a live ACE-Step stream back
  • Live-editable prompt
  • Every knob visible at once in stacked Core / Groups / Keystones sections (no bank tab switching)
  • Optional hardware MIDI input via the Web MIDI API with per-knob MIDI learn: click CC ? next to any knob, wiggle a physical control, and it rebinds live. Mappings persist per option-profile in localStorage; click Reset MIDI map to restore the auto-map.
  • Optional webcam motion input (frame-diff driving denoise)
  • HUD canvas with waveform background, history trails, playhead, SDE curve, and live stats
  • Zstd-compressed delta slices decoded in the browser

Requirements

  • Server: the full ACE-Step install (CUDA GPU, uv sync, prebuilt TensorRT engines). No extra dependencies beyond the main project.
  • Client: any modern Chromium or Firefox. Web MIDI and webcam support are optional.

Run

A single launcher starts the Python backend on :1318 and a Next.js dev server on :6660 with combined output:

uv run python -u -m demos.realtime_motion_graph_web.run
# forward backend args after `--`:
uv run python -u -m demos.realtime_motion_graph_web.run -- --accel eager

First run installs web/node_modules automatically (Node.js 20+ required). Open http://localhost:6660. Next.js rewrites /api/*, /fixtures/*, /loras/*, and /videos/* to the backend at :1318; the WebSocket URL comes from NEXT_PUBLIC_POD_BASE_URL (set by the launcher).

The full UI lives under web/ (React + zustand, mirrored from the internal daydreamlive/demon-react package). See web/components/, web/engine/, web/hooks/, and web/store/ for the source.

Backend args

Anything after -- on the launcher is forwarded to the backend.

--accel {tensorrt,compile,eager} sets BOTH decoder_backend and vae_backend on the underlying Session. Default is tensorrt.

--decoder-accel and --vae-accel override --accel for one component at a time. Useful when, for example, only one of the two TRT engines exists for a given checkpoint, or when you want to debug one component in eager while the other stays on TRT:

# Mix-and-match: TRT decoder, eager VAE.
uv run python -u -m demos.realtime_motion_graph_web.run -- \
    --accel tensorrt --vae-accel eager

The text encoder stays resident in VRAM by default so live prompt edits do not pay CPU/GPU transfer cost. Add --offload-text-encoder on lower-VRAM GPUs to restore the previous lower-memory behavior.

--checkpoint <name> selects which DiT checkpoint to load. The name must match a directory under <checkpoints_dir>/. Full TensorRT mode is registered for acestep-v15-turbo (default, 2B) and acestep-v15-xl-turbo (XL). XL TRT uses dynamic-batch b8 decoder profiles; build one first, then launch with:

uv run python -u -m demos.realtime_motion_graph_web \
    --accel tensorrt --checkpoint acestep-v15-xl-turbo

Headless Performance Benchmark

The browser HUD receives tick_ms and dec_ms over WebSocket, but the server does not print structured telemetry. To compare accelerators, checkpoints, TRT profiles, or VAE settings without browser/audio-device overhead, run the headless benchmark:

uv run python -u -m demos.realtime_motion_graph_web.benchmark \
    --accel tensorrt --checkpoint acestep-v15-xl-turbo

The benchmark mirrors the backend inference path: fixture/config defaults, TRT engine selection, Session(...), prepare_source, encode_text, session.stream(...), repeated stream.tick(...), and optional VAE decode. It reports setup timings, per-generation tick, decode, tick+decode mean/P50/P90/P95/min/max, skip counts, and peak CUDA memory.

Useful variants:

# Compare compile mode against the same workload.
uv run python -u -m demos.realtime_motion_graph_web.benchmark --accel compile

# Mixed backend, same style as the server flags.
uv run python -u -m demos.realtime_motion_graph_web.benchmark \
    --decoder-accel tensorrt --vae-accel eager

# Persist raw samples and summary stats.
uv run python -u -m demos.realtime_motion_graph_web.benchmark \
    --accel tensorrt --checkpoint acestep-v15-xl-turbo \
    --json runs/xl-trt-bench.json

# Mirror PipelineRunner's decode-skip behavior.
uv run python -u -m demos.realtime_motion_graph_web.benchmark \
    --accel tensorrt --skip-threshold 1e-3

By default, --skip-threshold -1 disables decode skipping so VAE decode latency is measured on every completed generation. Set --no-decode for decoder-only throughput.

Once it's running:

  1. Open http://localhost:6660/
  2. Click Play — the demo loads the default fixture (inside_confusion_loop_60s_gsm.wav). Fixtures stream from the daydreamlive/demon-fixtures-v2 Hugging Face dataset on first request and are cached locally.
  3. Switch fixtures any time using the selector at the top of the Advanced drawer; switching tears down the session and restarts with the new audio.
  4. Cold start takes ~15 s while the server loads the model + TRT engines (or longer on --accel compile); once it's ready the UI switches to the live HUD view.

Audio source vs. video

Audio is the primary source: the demo always loads from the canonical fixture set (daydreamlive/demon-fixtures-v2 on Hugging Face, listed in acestep.fixtures.KNOWN_FIXTURES), served by the backend at /fixtures/<name> via lazy HF download. Video is optional and secondary — drop any .mp4/.webm/.mov into videos/ (sibling of web/) to attach the audio-reactive shader pipeline. With no videos present the demo runs audio-only (graph mode is the default and looks the same).

Layout

demos/realtime_motion_graph_web/
├── README.md
├── __init__.py
├── __main__.py               # `python -m demos.realtime_motion_graph_web`
├── run.py                    # launcher: backend + Next.js dev server
├── server.py                 # HTTP API + WebSocket multiplex on one port
├── ws_adapter.py             # per-WebSocket coroutine; wraps StreamingSession
├── audio_codec.py            # per-subscriber slice codec + stem payload
├── protocol.py               # wire format (Python source of truth)
├── videos/                   # user-supplied .mp4/.webm/.mov drop-in (optional)
└── web/                      # Next.js front-end (React + zustand)

Protocol

The WebSocket protocol is defined in protocol.py (the Python source of truth that web/engine/protocol.ts mirrors):

  • Init: JSON config -> binary audio upload (<uint32 channels><uint32 samples> + float32 PCM)
  • Server init: JSON ready + binary float16 initial buffer
  • Streaming: JSON params/prompt out, binary slice (raw float16 or zstd-compressed float16 delta) + params_update / prompt_applied JSON messages in

server.py multiplexes the JSON HTTP API, fixture/video file serving, and the WebSocket upgrade onto one TCP port; the WS handshake hands off to ws_adapter.handle_client, which wraps a acestep.streaming.session.StreamingSession.

Audio-reactive video

The video is rendered through a small WebGL2 shader pipeline so it visually responds to the music in real time. Two effects:

  • Color parallax — saturated regions drift horizontally with a slow sway plus a punch on every kick.
  • Bloom on kick — luminance-thresholded bloom that brightens with the bass envelope.

The same kick amplitude is exposed to CSS as --bloom-amount, so the perimeter HUD bars and the cursor halo glow in lockstep with the shader bloom on the video.

Curator setup: nothing. Color parallax is saturation-driven, not depth-driven, so there is no preprocessing step and no depth map sidecars to generate. Drop the source video into videos/ and run the launcher as usual. If WebGL2 is unavailable the canvas is hidden and the plain video plays as fallback.

Test fixtures

The eight files in acestep.fixtures.KNOWN_FIXTURES ship with sidecar files in the daydreamlive/demon-fixtures-v2 HF dataset:

<track-id>/
  source.wav
  track.json                      # editable BPM/key/time-signature + asset manifest
  stems/
    vocals.wav                    # optional user-facing vocal stem
    instruments.wav               # optional user-facing instrumental stem
  sidecars/
    full.json
    full.safetensors              # full-track source latent + context_latent
    vocals.json
    vocals.safetensors            # optional pre-encoded vocal source
    instruments.json
    instruments.safetensors       # optional pre-encoded instrumental source

When the client sends fixture_name for a known fixture, the server loads the cached source latent + context latent and reads BPM / key from the JSON, skipping librosa beat tracking, the CNN key classifier, and Session.prepare_source. Session.encode_text still runs live every connect (it depends on the prompt and the demo's blended-prompt UI typically diverges from any baked tags within seconds of connecting; the ~60ms warm cost isn't worth the cache complication). For ad-hoc uploads (no fixture_name), the full live path runs as before.

The runtime checks MODELS_DIR/fixtures/ first (so local edits are tested without an upload round-trip) and falls through to the dataset. User uploads use the same layout under MODELS_DIR/user_uploads/; fixtures are effectively system-provided tracks.

If you want to override the BPM or key for a fixture, edit the <track-id>/track.json and re-run the precompute script. Editing the JSON's bpm / key / time_signature fields and re-running preserves them (the script only re-derives values that aren't already pinned). To forcibly re-derive everything from scratch, pass --force:

uv run python -m scripts.calibration.precompute_fixture_sidecars
uv run python -m scripts.calibration.precompute_fixture_sidecars --with-stems
uv run python -m scripts.calibration.precompute_fixture_sidecars --force
uv run python -m scripts.calibration.precompute_fixture_sidecars --only \
    inside_confusion_loop_60s_gsm.wav

After editing, upload the regenerated track JSON, WAV stem assets, and all sidecar JSON/safetensors pairs back to daydreamlive/demon-fixtures-v2. The runtime still falls back to the legacy daydreamlive/demon-fixtures dataset for old audio and full-track sidecar files when a v2 asset is not present.

Fixture dataset v2 plan:

  1. Create daydreamlive/demon-fixtures-v2 as a Hugging Face dataset repo.
  2. Run uv run python -m scripts.calibration.precompute_fixture_sidecars --with-stems.
  3. Upload each fixture directory with source.wav, track.json, stems/, and sidecars/.
  4. Keep daydreamlive/demon-fixtures available until deployed pods have warmed or downloaded the v2 layout.

Onboard MCP server (drive the demo from an LLM)

mcp_server.py is a stdio MCP server that exposes every user-facing demo action — prompt, knobs, LoRA enable/disable, timbre/structure refs, source swap — as an MCP tool. Useful for letting Claude Code (or any MCP client) drive the demo for automated testing.

How it works:

  1. The user opens the demo in their browser as usual. The backend registers that session in a process-global registry and starts the control bus, a small HTTP server on 127.0.0.1:1319.
  2. The MCP server speaks HTTP to the control bus. It does not open its own WebSocket and does not start a separate GPU session, so running it alongside a browser tab is free.
  3. Every command the MCP sends is dispatched through the same handler as the browser's own WebSocket frames. Acks (prompt_applied, lora_catalog, timbre_set, swap_ready, etc.) flow back to the browser's WS, so the front-end UI mirrors MCP-driven changes automatically. The new params_echo message handles knob changes (see useMcpMirror).

Run the backend the usual way:

uv run python -u -m demos.realtime_motion_graph_web.run

then point Claude Code at the MCP server:

{
  "mcpServers": {
    "demon": {
      "command": "uv",
      "args": [
        "run", "python", "-u",
        "-m", "demos.realtime_motion_graph_web.mcp_server"
      ],
      "cwd": "C:/_dev/projects/DEMON"
    }
  }
}

Env-var overrides:

  • DEMON_HOST / DEMON_PORT — backend's main HTTP+WS port (default 127.0.0.1:1318).
  • DEMON_CONTROL_HOST / DEMON_CONTROL_PORT — control bus (default 127.0.0.1:1319).

Server-side flags:

  • --control-host <host> — bind the control bus to a non-localhost interface (use sparingly: this lets remote clients write into a live session).
  • --control-port <port> — override the control port (default 1319).
  • --no-control — disable the control bus entirely.

Available tools: list_sessions, session_state, list_fixtures, list_loras, list_knobs, set_prompt, set_prompt_blend, set_knob, set_knobs, get_knob, enable_lora, disable_lora, set_timbre_strength, set_timbre_fixture, set_timbre_audio, clear_timbre, set_structure_fixture, set_structure_audio, clear_structure, swap_to_fixture, swap_to_audio. Most tools take an optional session_id parameter; they default to the most-recently-started session if you omit it.

Open the browser tab first — the MCP will refuse to act when there's no live session to attach to.

Browser notes

  • Web Audio: an AudioWorkletNode drives a shared PCM buffer that the main thread patches in place on each slice. Same crossfade logic as the native AudioEngine (50 ms on swap, in-place delta add otherwise).
  • Web MIDI: auto-attaches the first input. Values use the endless-encoder two's-complement CC semantics from the native client so existing controllers just work.
  • Webcam: getUserMedia with a low-res capture canvas and a simple abs-diff detector, smoothed like the OpenCV version.

Troubleshooting

  • "WebSocket connection failed": verify the backend is reachable on :1318 (firewall, reverse proxy). The launcher logs [backend] output if the Python side crashed.
  • Audio plays silent on first connect: browsers gate audio on a user gesture; Connect & start counts as one, so this should "just work" but if it doesn't, click anywhere in the HUD view.
  • No MIDI devices listed: Web MIDI requires localhost or HTTPS in Chromium. Use the local http://localhost:6660 URL, or run behind a reverse proxy with TLS for remote access.
  • Webcam permission denied: same-origin constraint as MIDI. Switch to on-screen knobs mode if the browser blocks the camera.
  • Cold start long: the GPU server rebuilds the pipeline on every new connection, same as the native server. Reuse connections when possible.