Automated E2E test suite with containerized REAPER and real frontend

[conormkelly]

Summary

Currently the extension has unit tests (Zig comptime mock injection) and frontend has Jest tests, but there's no automated E2E pipeline that exercises the full stack: REAPER → extension → WebSocket → frontend. Regressions in the integration layer (state polling, command dispatch, subscription lifecycle) can only be caught manually.

Goal

A containerized test environment with a known REAPER project that runs a curated set of E2E tests against the real system. This guards against regressions and dramatically speeds up development confidence.

Architecture questions to investigate

REAPER in a container

• REAPER has a -nosplash CLI flag and can run headless on Linux
• Linux build exists and works (ReleaseSafe, validated in CI — 84ebbe1, 8559fcf)
• Need to determine: can REAPER load a project and run the extension without a display? Xvfb or similar virtual framebuffer may be needed
• SWELL (Simple Windows Emulation Layer) handles the UI on Linux - may need X11 even for "headless" operation
• Alternative: macOS runner with REAPER installed (less portable but more realistic)

Known test project

• Curate a small .rpp with:
  • A few tracks with known names, colors, volumes, pans
  • Markers and regions at known positions
  • Items with takes (for Scrap/Retake/Keep testing)
  • FX chains with known parameters
  • Tempo map with changes
  • Pre-configured routing (sends/receives)
• Commit this fixture project to the repo under test-fixtures/

Test runner

• Browser automation (Playwright or Cypress) connecting to the served frontend
• Or: direct WebSocket client in TypeScript/Zig that sends commands and asserts on subscription state - avoids browser overhead for backend-focused tests
• Hybrid approach: WebSocket-level tests for command/state coverage, Playwright for UI interaction tests (touch instruments, timeline gestures)

Existing infrastructure to build on

• make test-e2e target already exists in the Makefile
• Extension test architecture uses comptime generics - E2E tests would complement this by testing the real REAPER API path that mocks can't cover
• CI already builds for Linux (GitHub Actions — 16a137d)

Suggested phased approach

Phase 1: WebSocket integration tests (no browser)

• Script that connects via WebSocket (websocat etc?), sends commands, asserts responses
• Run against a REAPER instance with a known project loaded
• Start with: transport state, track listing, command dispatch
• Can run locally first, containerize later

Phase 2: Containerized REAPER

• Docker image with REAPER + extension + test project pre-loaded
• Xvfb for display if needed
• GitHub Actions integration
• Smoke test: extension loads, WebSocket accepts connection, state is correct

Phase 3: Browser E2E

• Playwright tests against the served frontend
• Cover critical user flows: connect → see tracks → transport controls → Scrap/Retake/Keep → mixer adjustments
• Touch interaction simulation for instruments

Open questions

• Does REAPER's EULA allow running in CI containers?
• Can the extension auto-load a specific project on startup via CLI args?
  (reaper -nosplash test-fixtures/e2e-project.rpp - needs verification)
• What's the minimal REAPER Linux install size for a Docker image?

[m13v]

the phased approach makes a lot of sense. for Phase 1 (WebSocket integration tests), websocat works but has quirks with binary frames. we found that a small node script using ws is more reliable for asserting on structured JSON responses. the key advantage is you can do async assertion chains: send transport play, wait for the state update message, assert the state object has the right fields. for the containerized REAPER, Xvfb is almost certainly required since SWELL needs an X server even when nothing is drawn. the Docker image size concern is valid since REAPER itself is relatively small (~50MB) but the SWELL X11 dependencies add up. consider using a multi-stage build that only copies the REAPER binary and its runtime deps. one thing on the EULA question: REAPER's license allows evaluation use and the portable install doesn't require activation, so CI usage should be fine for testing your own extension.

[m13v]

for Phase 3 (browser E2E), we open sourced a test framework that handles the browser automation layer. the Playwright orchestration script for parallel test workers without session conflicts: https://github.com/m13v/browser-lock/blob/main/playwright-lock.sh and the test scenario discovery engine: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts