workflows/host: distinguish transient unavailable runner from missing runner#2259
Open
nadahalli wants to merge 3 commits into
Open
workflows/host: distinguish transient unavailable runner from missing runner#2259nadahalli wants to merge 3 commits into
nadahalli wants to merge 3 commits into
Conversation
… runner subscribe() previously returned a generic "cannot find a runner" error whenever no runner matched a trigger's requirements, conflating two cases: (1) no runner of the required type is configured at all (permanent misconfiguration), and (2) a runner of the right type exists but cannot currently satisfy the requirement, e.g. a capability that gates the runner is temporarily unavailable or not yet activated across the DON (transient). Case (2) now returns a typed ErrRunnerUnavailable so callers can hold and retry rather than failing workflow initialization outright; case (1) keeps the existing hard error. Adds a generated HandlesRequirements helper (a runner-type check) to tell the two apart. Lets the confidential-workflows engine degrade gracefully while the confidential-workflows capability rolls out DON-wide, instead of error-storming subscribe failures on nodes that don't yet have it.
Contributor
✅ API Diff Results -
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves workflow runner selection error signaling by distinguishing between (a) no runner type being configured vs (b) an appropriate runner type existing but currently being unable to satisfy requirements (transient), enabling callers to retry transient initialization failures instead of failing permanently.
Changes:
- Introduces
ErrRunnerUnavailableto represent the “runner type exists but can’t currently satisfy requirements” case and returns it fromsubscribe()when appropriate. - Adds generated
HandlesRequirementshelper to detect whether a module covers a requirement type without evaluating requirement values. - Updates subscription-related tests to assert
ErrRunnerUnavailablevs the existing hard error path.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/workflows/host/requirements_helper_gen.go | Adds generated HandlesRequirements helper used to detect runner-type presence without enforcing requirements. |
| pkg/workflows/host/requirements_gen/requirements_helper.go.tmpl | Updates generator template to emit HandlesRequirements for all sdk.Requirements fields. |
| pkg/workflows/host/requirement_selecting_module.go | Returns ErrRunnerUnavailable when a compatible runner type exists but no runner can currently satisfy the requirements. |
| pkg/workflows/host/requirement_selecting_module_test.go | Adjusts tests to validate transient vs permanent error behavior (ErrRunnerUnavailable vs generic failure). |
Files not reviewed (1)
- pkg/workflows/host/requirements_helper_gen.go: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+39
to
+43
| // HandlesRequirements reports whether handler covers every non-nil field in req (the | ||
| // right runner type), regardless of whether those callbacks pass. Distinguishes a | ||
| // present-but-unavailable runner from a missing one. | ||
| func HandlesRequirements(handler RequirementsHandler, req *sdk.Requirements) bool { | ||
| if req == nil { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
subscribe()inrequirement_selecting_modulereturned a genericcannot find a runnererror whenever no runner matched a trigger's requirements. That conflated two different cases:Case (2) now returns a typed
ErrRunnerUnavailable, so callers can treat it as transient (hold + retry) rather than failing workflow initialization outright. Case (1) keeps the existing hard error.Adds a generated
HandlesRequirementshelper (a runner-type check that mirrorsCheckRequirementsbut ignores the value) to distinguish the two.Why
On a mixed/rolling DON, the confidential-workflows capability (
confidential-workflows@1.0.0-alpha) is not present on every node yet. TheConfidentialModuleTEE runner exists on those nodes but reports zero regions, sosubscribefailed hard for every confidential workflow, error-storming engine-init on nodes that do not yet have the capability. This lets the engine (chainlink core, follow-up PR) degrade gracefully and self-heal once the capability is DON-wide, instead of hard-failing.Notes
ErrRunnerUnavailableas a transient hold) will consume this.HandlesRequirementsis generated viago generate ./pkg/workflows/host/(template + regen, no hand-edited generated code).ErrRunnerUnavailablevs permanent hard error).