fix(deploy): pre-flight port check + safer GPU default; align E2E demo with init --flowmesh

Doctest agent flagged that `lumilake deploy init` without `--flowmesh`
silently produces a broken stack (no workers), and that the `"all"`
CUDA default conflicts with other GPU stacks on shared hosts. Port
collisions only surface at docker bind time today — too late to give a
useful error.

* `lumilake deploy up` now runs `_check_port_collisions` before any
  docker compose call, covering both the lumilake server port and the
  FlowMesh stack's ports. Failed pre-flight raises `DeployError` with
  the offending ports listed.
* Promoted `flowmesh._env_ports` → `flowmesh.env_ports` for the new
  caller.
* `CUDA_VISIBLE_DEVICES` default reverts to blank (skip GPU workers);
  the `.env.example` comment tells the user to set a free index on
  their host rather than `"all"`.
* `docs/E2E_DEMO.md`: `init --flowmesh` is mandatory; load script
  documented twice (explicit creds before init, auto-detect after);
  demo input bumped to three stocks; the port-conflict troubleshooting
  bullet points at the pre-flight rather than naming specific ports.

Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>

Author: timzsu

README.md

@@ -56,15 +56,29 @@ lumilake deploy -C ~/lumilake-deploy up                 # bring the stack up via
 
 Note: a real workflow run also requires running PostgreSQL and S3-compatible storage; agent-style retrievals (`DataRetrievalOp` with `type: agent`) additionally require `LUMID_DATA_URL`. See `docs/ENV.md` for the env contract. If you don't have your own data plane, the repo ships a bundled Postgres + MinIO at `scripts/dev/compose.data-plane.yml` — see `docs/E2E_DEMO.md` for the full three-step demo flow (data plane → load demo data → run a workflow).
 
-Submit and inspect a workflow:
+Submit and inspect a workflow. From a source checkout the example
+workflow file is at `examples/templates/yaml/trading-agent.yaml`;
+PyPI installs do not ship the templates, so pass an absolute path to a
+workflow file you have locally:
 
 ```bash
+# From a source checkout:
 uv run lumilake login http://127.0.0.1:9000
-uv run lumilake job submit examples/templates/yaml/trading-agent.yaml --format yaml --input 'Stock=NVDA,AAPL' --output-prefix demo/trading-agent
-uv run lumilake job list
-uv run lumilake job watch <job_id>
+uv run lumilake job submit examples/templates/yaml/trading-agent.yaml \
+    --format yaml --input 'Stock=NVDA,AAPL,MSFT' --output-prefix demo/trading-agent
+
+# From a PyPI install (lumilake on PATH; supply your own workflow file):
+lumilake login http://127.0.0.1:9000
+lumilake job submit /path/to/your/workflow.yaml \
+    --format yaml --input 'Stock=NVDA,AAPL,MSFT' --output-prefix demo/trading-agent
+
+lumilake job list
+lumilake job watch <job_id>
 ```
 
+See `docs/E2E_DEMO.md` for a full reproduction using the bundled demo
+workflows and dataset.
+
 ## Data Access
 
 - **SQL retrievals** connect directly to `DATABASE_URL`.

docs/ARCHITECTURE.md

@@ -35,7 +35,8 @@ Lumilake is organized as a small control plane around workflow parsing, scheduli
           v
 +-------------------+
 | Archive + results |
-| S3 or lumid.data  |
+| S3 (S3_ARCHIVE_   |
+|     PREFIX)       |
 +-------------------+
 ```
 
@@ -61,4 +62,9 @@ Lumilake is organized as a small control plane around workflow parsing, scheduli
 
 ## Storage Model
 
-Lumilake separates compute data from job archive data. Direct mode reads and writes compute data through configured PostgreSQL and S3-compatible services. lumid.data mode forwards SQL and storage operations to lumid.data instead. Job records and runtime artifacts use `S3_ARCHIVE_PREFIX` in both modes, with lumid.data handling the storage calls when configured.
+Lumilake separates compute data from job archive data.
+
+- **SQL `DataRetrievalOp`** connects directly to `DATABASE_URL`.
+- **S3 `DataRetrievalOp`** connects directly to `S3_URL`.
+- **Agent `DataRetrievalOp`** (`type: agent`) routes through lumid.data's `/agent/v1` endpoint and therefore requires `LUMID_DATA_URL`.
+- **Archive** (job records, runtime artifacts) is always written under `S3_ARCHIVE_PREFIX` using the same `S3_URL` connection.

docs/E2E_DEMO.md

@@ -32,13 +32,24 @@ image-generation pair also reads HTML files from
 - `pg_dump` / `pg_restore` (PostgreSQL 16+) on `PATH` — used by the
   demo-data scripts.
 - Disk: ~1 GB free in `~/.cache/lumilake-demo` for the bundle.
-- A model provider key for the LLM ops (`OPENAI_API_KEY` or equivalent).
 - For agent-retrieval only: a running lumid.data instance and
   `LUMID_DATA_URL` set.
 
+The bundled demo workflows use locally-served open-weight models
+(`Qwen/Qwen3-8B` for text, `llava-hf/llava-1.5-7b-hf` for vision,
+`Tongyi-MAI/Z-Image-Turbo` for image gen) — no `OPENAI_API_KEY` is
+required. Set one only if you author workflows that call hosted
+providers.
+
 The `mc` MinIO client is **not** required; the bundled scripts use the
 `minio` Python library that ships with `lumilake-sdk`.
 
+> **Working directory.** Step 1 and Step 2 commands use repo-relative
+> paths (`scripts/dev/...`). Run them from a Lumilake source checkout
+> root, or pass absolute paths. Step 3 commands use `lumilake deploy
+> -C <dir>` and `lumilake job submit <abs-path>` so they work from any
+> CWD once `lumilake` is installed.
+
 ---
 
 ## Step 1 — Bring up the data plane (optional)
@@ -100,10 +111,19 @@ defaults if you've overridden them.
 `lumilake_demo` schema, and uploads `news/{html,images}` into
 `s3://<bucket>/example-data/news/`.
 
-It reads database / S3 credentials from a `.env` file (auto-detected by
-walking up from the current directory). If you ran `lumilake deploy
-init` in step 3 first, the defaults already point at step 1's data
-plane.
+It reads database / S3 credentials from a `.env` file (auto-detected
+by walking up from the current directory). If you haven't run
+`lumilake deploy init` yet — which writes a `.env` — pass the data-plane
+URLs explicitly:
+
+```bash
+uv run python scripts/dev/load_demo_data.py \
+  --database-url postgresql://lumilake:lumilake_password@127.0.0.1:15432/lumilake \
+  --s3-url s3://lumilake:lumilake_password@127.0.0.1:19100/lumilake-demo
+```
+
+Once `lumilake deploy init` has written a `.env`, the no-flag form
+auto-detects everything:
 
 ```bash
 uv run python scripts/dev/load_demo_data.py
@@ -169,20 +189,30 @@ from a source checkout via `uv run lumilake ...`.
 
 ```bash
 mkdir -p ~/lumilake-deploy
-lumilake deploy -C ~/lumilake-deploy init     # writes .env from the bundled template
+lumilake deploy -C ~/lumilake-deploy init --flowmesh     # writes .env + .env.flowmesh
 ```
 
+`--flowmesh` is required for the bundled FlowMesh stack: without it
+`lumilake deploy up` only starts the lumilake-server container and
+every job submission fails with `FlowMeshConnectionError` because no
+workers are reachable.
+
 The shipped `.env.example` is **pre-pointed at step 1's data plane**
 (`postgresql://lumilake:lumilake_password@127.0.0.1:15432/lumilake`,
 `s3://lumilake:lumilake_password@127.0.0.1:19100/lumilake-demo`). Open
 `~/lumilake-deploy/.env` only if you need to:
 
-- Set a model provider key (`OPENAI_API_KEY`, etc.).
+- Set a model provider key (`OPENAI_API_KEY`, etc.) — only if you
+  author workflows that call hosted providers; the bundled demos run
+  on local open-weight models.
 - Point at your own Postgres / S3 (override `DATABASE_URL`, `S3_URL`,
   `S3_USER_DATA_PREFIX`).
 - Enable agent retrievals (`LUMID_DATA_URL=http://127.0.0.1:9102`).
-- Override `CUDA_VISIBLE_DEVICES` if the default `"all"` (one worker
-  per `nvidia-smi`-detected GPU) is too aggressive.
+- Set `LUMILAKE_GPU_DEVICES` to one or more free GPU indices on your
+  host (default is empty — no GPU workers). On a shared host, pick an
+  index that other stacks are not using rather than `"all"`. This is
+  distinct from `CUDA_VISIBLE_DEVICES` in `.env.flowmesh`, which only
+  scopes what the FlowMesh server container sees.
 
 Then bring up the stack:
 
@@ -201,7 +231,7 @@ WORKFLOW_DIR=/path/to/lumilake_OSS/examples/templates
 
 lumilake job submit "$WORKFLOW_DIR/yaml/trading-agent.yaml" \
   --format yaml \
-  --input 'Stock=NVDA,AAPL' \
+  --input 'Stock=NVDA,AAPL,MSFT' \
   --output-prefix demo/trading-agent
 
 # Watch progress.
@@ -215,10 +245,9 @@ The other workflow pairs follow the same shape — swap the YAML path
 (or JSON path + `--format n8n`).
 
 **Image-generation note.** Requires a GPU-equipped FlowMesh worker.
-`lumilake deploy up` reads `CUDA_VISIBLE_DEVICES` from `.env`: the
-shipped default `"all"` spawns one worker per `nvidia-smi`-detected
-GPU; set it to a comma-separated subset (`"0,1"`) for partial use, or
-blank to skip GPU worker creation.
+`lumilake deploy up` reads `LUMILAKE_GPU_DEVICES` from `.env`: the
+shipped default is blank (no GPU workers). Set it to a free GPU index
+on your host (e.g. `"0"`) or a comma-separated subset for partial use.
 
 **Agent-retrieval note.** Requires `LUMID_DATA_URL` set in your `.env`;
 the agent retrievals route through lumid.data's `/agent/v1` endpoint.
@@ -281,5 +310,13 @@ halves are loaded as a unit and versioned together.
   that at least one CPU worker is registered; image-generation also
   needs a GPU worker.
 - **`unresolved placeholder` on `{symbol}`** — empty input list. Pass
-  `--input 'Stock=NVDA,AAPL'` (the YAML's `inputs: Stock: []` is a
-  template slot, not a default value).
+  `--input 'Stock=NVDA,AAPL,MSFT'` (the YAML's `inputs: Stock: []` is a
+  template slot, not a default value).
+- **Port collision at `lumilake deploy up`** — the deploy CLI runs a
+  pre-flight check and prints the conflicting port and its role. Free
+  the port on the host (e.g. tear down a competing stack), or pick a
+  free value for `LUMILAKE_SERVER_PORT` in `.env` (server) and
+  `SERVER_HTTP_PORT` / `SERVER_GRPC_PORT` / `REDIS_CONTROL_PORT` /
+  `REDIS_TELEMETRY_PORT` in `.env.flowmesh` (orchestrator). If you
+  change `SERVER_HTTP_PORT`, update `LUMILAKE_RUNTIME_ORCHESTRATOR_URL`
+  in `.env` and `FLOWMESH_BASE_URL` in `.env.flowmesh` to match.

docs/WORKFLOWS.md

@@ -52,10 +52,12 @@ params:
 
 ## Examples
 
-Runnable examples live in `examples/templates/`:
+Runnable examples live in `examples/templates/`. Each is shipped as a
+YAML + n8n pair; see `docs/E2E_DEMO.md` for the full reproduction
+recipe (data plane, demo dataset, deploy).
 
-- `examples/templates/yaml/agent-retrieval-mini.yaml`
-- `examples/templates/yaml/image-generation.yaml`
-- `examples/templates/n8n/image-generation.json`
+- `examples/templates/yaml/trading-agent.yaml` + `examples/templates/n8n/trading-agent.json`
+- `examples/templates/yaml/agent-retrieval.yaml` + `examples/templates/n8n/agent-retrieval.json`
+- `examples/templates/yaml/image-generation.yaml` + `examples/templates/n8n/image-generation.json`
 
-Use `uv run lumilake job preview examples/templates/yaml/agent-retrieval-mini.yaml --format yaml --input Stock=AAPL` to validate and inspect a YAML schedule without dispatching runtime work.
+Use `lumilake job preview examples/templates/yaml/trading-agent.yaml --format yaml --input Stock=NVDA` to validate and inspect a YAML schedule without dispatching runtime work.

packages/deploy/src/lumilake_deploy/assets/.env.example

@@ -18,10 +18,13 @@ LUMILAKE_BATCH_ACCUMULATION_SECONDS="0"
 LUMILAKE_OPTIMIZER_SUBPROCESS_TIMEOUT_SECONDS="60"
 LUMILAKE_CPU_WORKER_GROUP_SIZE="1"
 LUMILAKE_GPU_WORKER_GROUP_SIZE="1"
-# GPU devices for FlowMesh workers (one worker per device). "all"
-# expands to every nvidia-smi-detected GPU; "0,2" selects a subset;
-# leave blank to skip GPU worker creation.
-CUDA_VISIBLE_DEVICES="all"
+# GPU devices Lumilake assigns to FlowMesh workers, one worker per
+# device. Distinct from ``.env.flowmesh``'s ``CUDA_VISIBLE_DEVICES``,
+# which scopes what the FlowMesh server container itself sees. Set to a
+# specific index (e.g. "0") or a comma-separated subset ("0,2") for the
+# free GPUs on your host; "all" expands to every nvidia-smi-detected
+# GPU. Leave blank to skip GPU worker creation.
+LUMILAKE_GPU_DEVICES=""
 
 # ── Runtime (FlowMesh) ────────────────────────────────────────────────────────
 # The orchestrator URL the server dispatches workflows to. When using the

packages/deploy/src/lumilake_deploy/doctor.py

@@ -103,7 +103,7 @@ def _add(self, level: FindingLevel, message: str) -> None:
     "HARDWARE_MEMORY_REQUIREMENT",
     "HARDWARE_GPU_REQUIREMENT",
     "HARDWARE_GPU_MEMORY_REQUIREMENT",
-    "CUDA_VISIBLE_DEVICES",
+    "LUMILAKE_GPU_DEVICES",
 )
 
 _KNOWN_KEYS: frozenset[str] = frozenset(

packages/deploy/src/lumilake_deploy/flowmesh.py

@@ -106,7 +106,7 @@ def _load_env(ef: Path) -> None:
 )
 
 
-def _env_ports(env_file: str | Path) -> dict[str, int]:
+def env_ports(env_file: str | Path) -> dict[str, int]:
     """Extract configured ports from env file for collision checking."""
     env = parse_env_file(Path(env_file))
     return {
@@ -163,7 +163,7 @@ def stack_up(env_file: str | Path) -> None:
     _make_workdir()
     env_path = Path(env_file).resolve()
 
-    errors = check_ports(_env_ports(env_file))
+    errors = check_ports(env_ports(env_file))
     if errors:
         for e in errors:
             _error(e)
@@ -308,7 +308,7 @@ def create_workers(env_file: str | Path, cpu_count: int, gpu_devices: str) -> No
         if not device_ids:
             _info(
                 "No GPUs detected; skipping GPU worker creation. "
-                "Set CUDA_VISIBLE_DEVICES to a comma-separated list "
+                "Set LUMILAKE_GPU_DEVICES to a comma-separated list "
                 "of indices to force-create."
             )
         else:
@@ -326,7 +326,7 @@ def create_workers(env_file: str | Path, cpu_count: int, gpu_devices: str) -> No
 
 
 def _resolve_gpu_device_ids(gpu_devices: str) -> list[str]:
-    """Resolve a CUDA_VISIBLE_DEVICES-style string into explicit device ids."""
+    """Resolve a LUMILAKE_GPU_DEVICES-style string into explicit device ids."""
     normalized = gpu_devices.strip()
     if normalized and normalized.lower() != "all":
         return [tok.strip() for tok in normalized.split(",") if tok.strip()]

packages/deploy/src/lumilake_deploy/setup.py

@@ -51,6 +51,25 @@ def load_project_env(project_root: Path) -> None:
         importlib.reload(envs)
 
 
+def _check_port_collisions(project_root: Path, layout: "_InfraLayout") -> None:
+    """Pre-flight: surface any host-port conflicts before docker compose starts."""
+    ports: dict[str, int] = {
+        "lumilake-server": int(envs.LUMILAKE_SERVER_PORT or 9000),
+    }
+    if layout.deploy_fm:
+        env_fm = project_root / FLOWMESH_ENV_FILE_NAME
+        if env_fm.is_file():
+            ports.update(fm_mod.env_ports(env_fm))
+    errors = fm_mod.check_ports(ports)
+    if errors:
+        for err in errors:
+            info(err)
+        raise DeployError(
+            "Port collision detected. Free the listed ports (or change them "
+            "in .env / .env.flowmesh) and retry."
+        )
+
+
 def _resolve_infra_layout(project_root: Path) -> _InfraLayout:
     """Decide which sibling infra to bring up alongside the server."""
     return _InfraLayout(deploy_fm=(project_root / FLOWMESH_ENV_FILE_NAME).is_file())
@@ -203,14 +222,16 @@ def run_setup(project_root: Path, options: SetupOptions) -> None:
     if options.reset:
         _reset_stack(project_root)
 
+    _check_port_collisions(project_root, layout)
+
     if layout.deploy_fm:
         env_fm = project_root / FLOWMESH_ENV_FILE_NAME
         fm_mod.stack_pull(env_fm)
         fm_mod.stack_up(env_fm)
         if not fm_mod.wait_healthy(env_fm, timeout=120):
             raise DeployError("FlowMesh stack did not become healthy.")
         cpu_count = int(envs.LUMILAKE_CPU_WORKER_GROUP_SIZE or 0)
-        gpu_devices = os.environ.get("CUDA_VISIBLE_DEVICES", "")
+        gpu_devices = os.environ.get("LUMILAKE_GPU_DEVICES", "")
         fm_mod.create_workers(env_fm, cpu_count=cpu_count, gpu_devices=gpu_devices)
 
     if options.no_server: