runtime v2 flag for new captures and derivations#3219
Merged
Conversation
Adds a `--runtime-v2-new-materializations` flag (env `RUNTIME_V2_NEW_MATERIALIZATIONS`) to the agent. When enabled, any newly-created materialization that doesn't already set `enable-runtime-v2` is published with that shard flag set to `true`, placing it on the V2 runtime. Existing materializations are never touched, and an explicit per-task flag always takes precedence. This extends the existing `RuntimeV2Rollout` initializer (previously capture-only) with an independent per-task-type toggle, so materializations can be rolled out separately from captures. Captures and materializations share a single existence query, since catalog names are globally unique. This can be deployed inert: the flag defaults to false, so it only begins to take effect once the agent is configured to enable it.
Adds a `--runtime-v2-new-derivations` flag (env `RUNTIME_V2_NEW_DERIVATIONS`) to the agent. When enabled, any newly-created derivation that doesn't already set `enable-runtime-v2` is published with that shard flag set to `true`, placing it on the V2 runtime. Existing derivations are never touched, and an explicit per-task flag always takes precedence. A derivation is a collection carrying a `derive` block, so unlike captures and materializations its shards live at `derive.shards`, and plain collections are never candidates. This extends the existing `RuntimeV2Rollout` initializer with an independent per-task-type toggle, so derivations can be rolled out separately from captures and materializations. This can be deployed inert: the flag defaults to false, so it only begins to take effect once the agent is configured to enable it.
Sets RUNTIME_V2_NEW_CAPTURES / _DERIVATIONS / _MATERIALIZATIONS in the local control-plane agent env, so every local stack stamps `enable-runtime-v2` onto newly-created tasks of all three types at publication time. Local development and QA thus exercise the V2 runtime by default; existing tasks and explicit per-task flags are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Rolls newly-created materializations and derivations onto runtime-v2, mirroring the existing captures flag (b98d27c). Adds
RUNTIME_V2_NEW_MATERIALIZATIONSandRUNTIME_V2_NEW_DERIVATIONSagent flags, each an independent per-task-type toggle onRuntimeV2Rollout. New tasks getenable-runtime-v2stamped at publish; existing tasks and explicit flags are untouched. Defaults off (deploy-inert); the last commit turns all three on for local stacks.Tested: integration tests for both types + end-to-end on a local stack (new task gets the flag in model + built shard label and actually runs on the v2 sidecar).
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)