fix(runtime): worker GPU schema, profile InputOp params, chat metadata#4
Merged
Conversation
Three runtime bugs that surfaced when the trading-debate demo workflow
(examples/templates/yaml/trading-debate-demo.yaml) was driven end-to-end
through the CLI. Each was already silently breaking real workloads:
GpuPlatformInfo expected `gpus: list[GpuInfo]`, but the FlowMesh worker
payload uses `devices: list[...]` (plus the optional `memory_is_unified`
and `shared_memory_total_bytes` fields). The mismatch made `/workers`
500 on response validation and caused `_has_gpu` to report False for
every worker — every GPU-needing workflow then hung indefinitely in the
"Waiting for worker group" check. Schema aligned to what workers
actually serialize; `_has_gpu` updated to match.
`_build_data_profile_node_from_data_retrieval_op` kept only params with
a literal `data` field, so any param bound to an InputOp via `node:` was
silently dropped from the data-profile path. With the param gone, the
template placeholder (`{symbol}` etc.) couldn't be resolved and profile
SQL failed with "unresolved placeholder". The main runtime build already
does this resolution at `runtime_graph.py:843-847`; profiling now does
the same: receives `inputs_dict` and inlines InputOp-bound params as
literal lists before the data-only filter runs.
The final-result aggregator unconditionally read `item["metadata"]
["prompt"]` when building chat histories. Worker payloads include
`metadata.prompt` for free-text generations but omit it for
structural-output (and image) ops — so a fully-successful job with N
structural-output outputs would 500 with `KeyError: 'metadata'` during
result assembly. Chat history is informational; missing entries now
skip silently instead of failing the job.
Verified end-to-end against postgres + minio on 10.10.40.2:
- workflow: 4 SQL retrievals + 6 LLM ops (Bull/Bear/Risk/Macro
researchers → Debate Moderator → Verdict normalizer), 19 ops total
after parser expansion
- input: Stock=NVDA, model: google/gemma-3-27b-it
- result: `{"verdict": "HOLD"}`, 10/10 nodes succeeded, ~2 min wall time
- pure pg+s3 mode (LUMID_DATA_URL empty)
- driven entirely through `lumilake job submit/info/result` CLI
235 tests pass; pre-commit (isort/black/ruff/codespell/mypy) clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Three follow-ups from the review of fix/e2e-runtime-trio:
Worker schemas re-export the FlowMesh SDK types instead of redefining
them. The local hierarchy was the same shape but kept drifting from the
worker wire format (the `gpus` vs `devices` mismatch this PR exists to
fix being the obvious example). `flowmesh.models.workers` is the source
of truth — re-exporting from there means future worker-payload
additions arrive automatically without another schema-skew bug.
The `_has_gpu` reader's "fallback ``count`` field" branch goes too —
FlowMesh has never emitted it; only the alias check was real, and
that's now a single isinstance + len.
The metadata chain access in the final-result aggregator uses
try/except instead of a stack of isinstance guards. Easier to read,
catches the same KeyError / TypeError cases.
The demo workflow now points at a dedicated `demo` schema (copied from
the experiment-data tables, NVDA/AAPL/TSLA/AMD/GOOGL only) so it
doesn't reach into project-internal `exp_data_1g` naming. Setup SQL is
in the file header as a copy-pasteable block.
End-to-end verified against the new schema: NVDA produced
``{"verdict": "HOLD"}`` in ~132 s, 10/10 nodes successful.
235 tests pass; pre-commit clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Inline comments and module docstrings on the e2e-fix files were too prosey. Trimmed to the same shape the rest of the repo uses (single- line where present, none where the code already reads). `_build_data_profile_node_from_data_retrieval_op` had a five-line nested conditional ladder. Extracted to `_resolve_profile_param` (try/ except) and turned the call site into a list comprehension. 235 tests pass; pre-commit clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
An empty list for a declared workflow input would parse cleanly through
the submission path, reach the runtime, and produce 0 rows per
per-symbol fan-out — wasting scheduling and crashing downstream paths
that assume at least one element. Now rejected with a 400 at shape
validation; the S3 / DB resolution path also gets a fallback guard for
the case where a remote resolver returns an empty set.
Verified end-to-end: raw POST with `inputs: {"Stock": []}` returns
``400 inputs['Stock']: expected a non-empty list of strings, got []``;
valid input still submits and runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Driving the OSS server end-to-end against the shared Postgres + MinIO turned up three silent breakages that affected every real workload, not just the demo I was running:
WorkerInfo.hardware.gpuschema expected agpusfield, but the FlowMesh worker payload usesdevices. The/workersroute 500'd on response validation, and_has_gpureturnedFalsefor every worker — every GPU-needing workflow then sat in the "Waiting for worker group" check forever.datafield, dropping anything bound vianode: <InputOp>— the entire purpose ofinputs:in the YAML format. The result was an "unresolved placeholder" error before the workflow could be optimized. The main runtime path already resolves these; profiling didn't.item["metadata"]["prompt"]. FlowMesh workers includemetadata.promptfor free-text generations but omit it for structural-output (and image) ops, so a fully-successful job would still 500 withKeyError: 'metadata'while collecting outputs.The PR also lands a small demo workflow (
examples/templates/yaml/trading-debate-demo.yaml) that drives all of these paths so the failures don't come back silently. It also exercises HALO planning over the typical mix the optimizer is designed for: 4 SQL retrievals + 6 LLM ops (bull/bear/risk/macro analysts → debate moderator → verdict normalizer).Changes
Three behaviour fixes plus a schema refactor that prevents a class of future drift.
Design
A few calls a reviewer should weigh in on:
flowmesh.models.workersinstead of being declared locally. FlowMesh owns the worker wire format and we already depend on the SDK; mirroring its types here means the next worker-payload addition (more telemetry, a new hardware field) doesn't need a parallel schema change in lumilake. The localWorkerInfo'selastic_disabledfield was unused dead and went with the rewrite — flagging in case there's an external consumer I didn't see._build_data_profile_node_from_data_retrieval_opnow receivesinputs_dictand resolves InputOp-bound params into literaldata: {type: list, items: [...]}— same transform the runtime build already does atruntime_graph.py:843-847. Other param kinds (op-output refs) still route to theconstraintspath as before. Extracted the resolver to_resolve_profile_param(try/except, no nested isinstance ladders).chat_historiesis informational; structural-output and image ops legitimately don't have a prompt-history shape to record. Open to making this stricter (e.g. an explicit per-op-type check) if there's a use case where a missingmetadata.promptshould be a hard error.Test Plan
Pre-existing CI runs (
pytest,pre-commit) cover the code-level safety; the new behaviour needs an actual server + a worker + a database to exercise. The demo workflow drives the full path end-to-end through the CLI:A workflow with input-bound SQL params (the demo, and any future
inputs:-driven workflow) reaching execution at all is the data-profile-fix evidence. The same job completing without aKeyError: 'metadata'during result aggregation is the chat-history-fix evidence. The same job not stalling in "Waiting for worker group" is the worker-schema-fix evidence.Test Result
Local run against the shared PG + MinIO + lumid.data backing stack (lumid.data data-plane URL left empty so the server talks directly to PG + S3):
pytest tests/: 235 passed in 1.49s.pre-commit run --all-files: isort / black / ruff / codespell / mypy — all clean.status: completed,error: null, 10/10 nodes succeeded, ~130 s wall time per run, final output{"verdict": "HOLD"}.Pre-submission Checklist
CONTRIBUTING.md.uv run pre-commit run --all-filesand fixed any issues.tests/. Adding a unit test that doesn't materialize a real worker payload felt low-value; happy to add one if you want a minimum-viable regression check.uv run pytest tests/passes locally.[BREAKING]and described migration steps above. (Not breaking — public surfaces unchanged. The worker schema refactor removes the unusedelastic_disabledfield; flagging in the Design section.)