Skip to content

fix(runtime): worker GPU schema, profile InputOp params, chat metadata#4

Merged
timzsu merged 4 commits into
mainfrom
fix/e2e-runtime-trio
May 13, 2026
Merged

fix(runtime): worker GPU schema, profile InputOp params, chat metadata#4
timzsu merged 4 commits into
mainfrom
fix/e2e-runtime-trio

Conversation

@timzsu

@timzsu timzsu commented May 13, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Driving the OSS server end-to-end against the shared Postgres + MinIO turned up three silent breakages that affected every real workload, not just the demo I was running:

  • Workers were invisible to the scheduler. Our local WorkerInfo.hardware.gpu schema expected a gpus field, but the FlowMesh worker payload uses devices. The /workers route 500'd on response validation, and _has_gpu returned False for every worker — every GPU-needing workflow then sat in the "Waiting for worker group" check forever.
  • Every YAML workflow with input-bound SQL params failed data profiling. The data-profile builder kept only params with a literal data field, dropping anything bound via node: <InputOp> — the entire purpose of inputs: in the YAML format. The result was an "unresolved placeholder" error before the workflow could be optimized. The main runtime path already resolves these; profiling didn't.
  • Structural-output jobs crashed during result assembly. The chat-history aggregator did an unguarded item["metadata"]["prompt"]. FlowMesh workers include metadata.prompt for free-text generations but omit it for structural-output (and image) ops, so a fully-successful job would still 500 with KeyError: 'metadata' while collecting outputs.

The PR also lands a small demo workflow (examples/templates/yaml/trading-debate-demo.yaml) that drives all of these paths so the failures don't come back silently. It also exercises HALO planning over the typical mix the optimizer is designed for: 4 SQL retrievals + 6 LLM ops (bull/bear/risk/macro analysts → debate moderator → verdict normalizer).

Changes

Three behaviour fixes plus a schema refactor that prevents a class of future drift.

Design

A few calls a reviewer should weigh in on:

  • Worker schemas now re-export from flowmesh.models.workers instead of being declared locally. FlowMesh owns the worker wire format and we already depend on the SDK; mirroring its types here means the next worker-payload addition (more telemetry, a new hardware field) doesn't need a parallel schema change in lumilake. The local WorkerInfo's elastic_disabled field was unused dead and went with the rewrite — flagging in case there's an external consumer I didn't see.
  • The data-profile fix is shaped to match the existing main-runtime path. _build_data_profile_node_from_data_retrieval_op now receives inputs_dict and resolves InputOp-bound params into literal data: {type: list, items: [...]} — same transform the runtime build already does at runtime_graph.py:843-847. Other param kinds (op-output refs) still route to the constraints path as before. Extracted the resolver to _resolve_profile_param (try/except, no nested isinstance ladders).
  • Missing chat-history metadata now silently skips that entry rather than failing the job. chat_histories is informational; structural-output and image ops legitimately don't have a prompt-history shape to record. Open to making this stricter (e.g. an explicit per-op-type check) if there's a use case where a missing metadata.prompt should be a hard error.

Test Plan

Pre-existing CI runs (pytest, pre-commit) cover the code-level safety; the new behaviour needs an actual server + a worker + a database to exercise. The demo workflow drives the full path end-to-end through the CLI:

# After setting up `.env` for direct pg+s3 (LUMID_DATA_URL empty,
# DATABASE_URL + S3_URL pointing at PG + MinIO) and creating the `demo`
# schema described at the top of the workflow file:
uv run lumilake deploy up                 # FlowMesh + lumilake server
uv run lumilake login http://127.0.0.1:19000
uv run lumilake job submit \
    examples/templates/yaml/trading-debate-demo.yaml \
    --format yaml \
    --input "Stock=NVDA" \
    --output-type s3 \
    --output-prefix "<bucket>/<prefix>/"
uv run lumilake job result <job_id>

A workflow with input-bound SQL params (the demo, and any future inputs:-driven workflow) reaching execution at all is the data-profile-fix evidence. The same job completing without a KeyError: 'metadata' during result aggregation is the chat-history-fix evidence. The same job not stalling in "Waiting for worker group" is the worker-schema-fix evidence.

Test Result

Local run against the shared PG + MinIO + lumid.data backing stack (lumid.data data-plane URL left empty so the server talks directly to PG + S3):

  • pytest tests/: 235 passed in 1.49s.
  • pre-commit run --all-files: isort / black / ruff / codespell / mypy — all clean.
  • E2E (NVDA, then AAPL, both via the CLI): status: completed, error: null, 10/10 nodes succeeded, ~130 s wall time per run, final output {"verdict": "HOLD"}.

Pre-submission Checklist
  • I have read CONTRIBUTING.md.
  • I have run uv run pre-commit run --all-files and fixed any issues.
  • I have added or updated tests covering my changes (if applicable) — the new behaviour needs a live server + worker + PG to exercise, so it's covered by the demo workflow + the live e2e I ran rather than by tests/. Adding a unit test that doesn't materialize a real worker payload felt low-value; happy to add one if you want a minimum-viable regression check.
  • I have verified that uv run pytest tests/ passes locally.
  • If this is a breaking change, I have prefixed the PR title with [BREAKING] and described migration steps above. (Not breaking — public surfaces unchanged. The worker schema refactor removes the unused elastic_disabled field; flagging in the Design section.)
  • I have updated documentation if user-facing behavior changed — the new demo workflow file has the schema-setup SQL inline at the top.

timzsu and others added 4 commits May 13, 2026 15:31
Three runtime bugs that surfaced when the trading-debate demo workflow
(examples/templates/yaml/trading-debate-demo.yaml) was driven end-to-end
through the CLI. Each was already silently breaking real workloads:

GpuPlatformInfo expected `gpus: list[GpuInfo]`, but the FlowMesh worker
payload uses `devices: list[...]` (plus the optional `memory_is_unified`
and `shared_memory_total_bytes` fields). The mismatch made `/workers`
500 on response validation and caused `_has_gpu` to report False for
every worker — every GPU-needing workflow then hung indefinitely in the
"Waiting for worker group" check. Schema aligned to what workers
actually serialize; `_has_gpu` updated to match.

`_build_data_profile_node_from_data_retrieval_op` kept only params with
a literal `data` field, so any param bound to an InputOp via `node:` was
silently dropped from the data-profile path. With the param gone, the
template placeholder (`{symbol}` etc.) couldn't be resolved and profile
SQL failed with "unresolved placeholder". The main runtime build already
does this resolution at `runtime_graph.py:843-847`; profiling now does
the same: receives `inputs_dict` and inlines InputOp-bound params as
literal lists before the data-only filter runs.

The final-result aggregator unconditionally read `item["metadata"]
["prompt"]` when building chat histories. Worker payloads include
`metadata.prompt` for free-text generations but omit it for
structural-output (and image) ops — so a fully-successful job with N
structural-output outputs would 500 with `KeyError: 'metadata'` during
result assembly. Chat history is informational; missing entries now
skip silently instead of failing the job.

Verified end-to-end against postgres + minio on 10.10.40.2:
- workflow: 4 SQL retrievals + 6 LLM ops (Bull/Bear/Risk/Macro
  researchers → Debate Moderator → Verdict normalizer), 19 ops total
  after parser expansion
- input: Stock=NVDA, model: google/gemma-3-27b-it
- result: `{"verdict": "HOLD"}`, 10/10 nodes succeeded, ~2 min wall time
- pure pg+s3 mode (LUMID_DATA_URL empty)
- driven entirely through `lumilake job submit/info/result` CLI

235 tests pass; pre-commit (isort/black/ruff/codespell/mypy) clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Three follow-ups from the review of fix/e2e-runtime-trio:

Worker schemas re-export the FlowMesh SDK types instead of redefining
them. The local hierarchy was the same shape but kept drifting from the
worker wire format (the `gpus` vs `devices` mismatch this PR exists to
fix being the obvious example). `flowmesh.models.workers` is the source
of truth — re-exporting from there means future worker-payload
additions arrive automatically without another schema-skew bug.

The `_has_gpu` reader's "fallback ``count`` field" branch goes too —
FlowMesh has never emitted it; only the alias check was real, and
that's now a single isinstance + len.

The metadata chain access in the final-result aggregator uses
try/except instead of a stack of isinstance guards. Easier to read,
catches the same KeyError / TypeError cases.

The demo workflow now points at a dedicated `demo` schema (copied from
the experiment-data tables, NVDA/AAPL/TSLA/AMD/GOOGL only) so it
doesn't reach into project-internal `exp_data_1g` naming. Setup SQL is
in the file header as a copy-pasteable block.

End-to-end verified against the new schema: NVDA produced
``{"verdict": "HOLD"}`` in ~132 s, 10/10 nodes successful.

235 tests pass; pre-commit clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Inline comments and module docstrings on the e2e-fix files were too
prosey. Trimmed to the same shape the rest of the repo uses (single-
line where present, none where the code already reads).

`_build_data_profile_node_from_data_retrieval_op` had a five-line
nested conditional ladder. Extracted to `_resolve_profile_param` (try/
except) and turned the call site into a list comprehension.

235 tests pass; pre-commit clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
An empty list for a declared workflow input would parse cleanly through
the submission path, reach the runtime, and produce 0 rows per
per-symbol fan-out — wasting scheduling and crashing downstream paths
that assume at least one element. Now rejected with a 400 at shape
validation; the S3 / DB resolution path also gets a fallback guard for
the case where a remote resolver returns an empty set.

Verified end-to-end: raw POST with `inputs: {"Stock": []}` returns
``400 inputs['Stock']: expected a non-empty list of strings, got []``;
valid input still submits and runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
@timzsu
timzsu merged commit 05ea926 into main May 13, 2026
10 checks passed
@timzsu
timzsu deleted the fix/e2e-runtime-trio branch May 13, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant