feat(ci): group e2e tests into functional suites with opt-in extended PR runs#1197
Open
jasonmadigan wants to merge 1 commit into
Open
feat(ci): group e2e tests into functional suites with opt-in extended PR runs#1197jasonmadigan wants to merge 1 commit into
jasonmadigan wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughThe PR adds functional Ginkgo suite labels across E2E specs, introduces curated PR/extended/full/named suite targets, and routes PR and on-demand E2E execution through label- and comment-based selection. It also updates documentation and workflow comments. ChangesE2E suite routing and labelling
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 markdownlint-cli2 (0.22.1).claude/rules/e2e-tests.mdmarkdownlint-cli2 wrapper config was not available before execution docs/ci.mdmarkdownlint-cli2 wrapper config was not available before execution tests/e2e/CLAUDE.mdmarkdownlint-cli2 wrapper config was not available before execution Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jasonmadigan
force-pushed
the
1195-e2e-functional-suites
branch
2 times, most recently
from
ae349e1 to
9a092f8
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
.github/workflows/e2e-on-demand.yaml (1)
49-81: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winResolve comment-triggered suites through the shared router.
This step re-implements the aggregate-suite mapping that
build/suite-router.shalready owns. That duplication will drift the next time a suite or alias changes, and then/test-e2ecomments will stop matching PR routing behaviour.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/e2e-on-demand.yaml around lines 49 - 81, The suite-to-make-target mapping in the Resolve make target step is duplicated logic that should be routed through the shared suite router instead. Update the workflow to use build/suite-router.sh for resolving comment-triggered suites so aliases and suite names stay consistent with the existing routing logic, and keep the e2e-on-demand resolve step focused on consuming that shared output rather than reimplementing the mapping.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e.yaml:
- Around line 278-299: The sticky comment lookup in the github-script block only
calls issues.listComments once, so it can miss the existing `<!-- e2e-suite-bot
-->` comment when there are more than 30 comments. Update the comment retrieval
logic to use `github.paginate` with `github.rest.issues.listComments`, then run
the existing `comments.find(...)` check against the full result set so
`issues.updateComment` is used instead of creating duplicates. Keep the rest of
the marker-based update/create flow unchanged.
In `@build/e2e.mk`:
- Around line 8-10: The PR-suite size description is stale and no longer matches
the updated curated gate. Update the inline documentation around E2E_SUITE_PR in
build/e2e.mk so the `pr` suite is described as ~31 specs, and make sure the
neighboring `pr-extended`/suite help text stays consistent with the new count.
Use the existing suite labels and comments in this section as the source of
truth so `make help` and the inline docs agree.
In `@build/suite-router.sh`:
- Around line 76-106: The pattern order in suite selection is too general in two
places, causing first-match Bash semantics to skip the intended specific cases.
In the main case block in suite-router.sh, move the TLS-specific
internal/broker/upstream/tls* rule ahead of the broader
internal/broker/upstream/* rule so TLS files map correctly; in the tests/e2e/*
nested case, place *url_elicitation* before *elicitation* so URL-elicitation
tests are not captured by the generic match. Keep the rest of the suggestions
logic unchanged.
In `@docs/ci.md`:
- Around line 79-88: Update the “How the Suite Router Works” section to point
readers to the shared routing logic in suite-router.sh instead of the workflow
YAML, and make clear that the PR workflow invokes that script. Use the existing
suite-router flow description and the known symbols like E2E_KNOWN_SUITES and
the PR workflow entrypoint to keep the docs aligned with the actual
implementation.
---
Nitpick comments:
In @.github/workflows/e2e-on-demand.yaml:
- Around line 49-81: The suite-to-make-target mapping in the Resolve make target
step is duplicated logic that should be routed through the shared suite router
instead. Update the workflow to use build/suite-router.sh for resolving
comment-triggered suites so aliases and suite names stay consistent with the
existing routing logic, and keep the e2e-on-demand resolve step focused on
consuming that shared output rather than reimplementing the mapping.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 407ae73c-23da-45a6-aa34-9700fa889266
📒 Files selected for processing (20)
.claude/rules/e2e-tests.md.github/workflows/e2e-auth.yaml.github/workflows/e2e-nightly.yaml.github/workflows/e2e-on-demand.yaml.github/workflows/e2e.yamlbuild/e2e.mkbuild/known-suites.txtbuild/suite-router.shdocs/ci.mdtests/e2e/CLAUDE.mdtests/e2e/auth_policy_test.gotests/e2e/custom_tls_test.gotests/e2e/elicitation_test.gotests/e2e/happy_path_test.gotests/e2e/multi_gateway_test.gotests/e2e/oauth_protected_resource_test.gotests/e2e/tool_discovery_test.gotests/e2e/tool_validation_test.gotests/e2e/url_elicitation_test.gotests/e2e/user_specific_list_test.go
💤 Files with no reviewable changes (2)
- .github/workflows/e2e-auth.yaml
- .github/workflows/e2e-nightly.yaml
jasonmadigan
force-pushed
the
1195-e2e-functional-suites
branch
2 times, most recently
from
9c17867 to
dc310af
Compare
coderabbitai
Bot
added
the
review-effort/large
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e.yaml:
- Around line 242-246: The Read known suites step in the e2e workflow is joining
only spaces, so the newline-separated output from known-suites.txt can still
leak into suites and break the $GITHUB_OUTPUT entry. Update the shell logic in
the known suite processing block to normalize newline-separated values before
wrapping them in backticks and comma-joining, using the existing Read known
suites step and the known id as the place to fix it.
In `@docs/ci.md`:
- Around line 54-58: The fenced command block in the CI docs is missing a
language hint, triggering MD040. Update the markdown fence around the /test-e2e
examples to use an explicit text/plain code language, keeping the existing
command content unchanged. Locate the block in the docs CI section that shows
/test-e2e discovery, /test-e2e pr-extended, and /test-e2e full.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 7db8025b-4d7c-46d8-b799-22cab2df5d83
📒 Files selected for processing (20)
.claude/rules/e2e-tests.md.github/workflows/e2e-auth.yaml.github/workflows/e2e-nightly.yaml.github/workflows/e2e-on-demand.yaml.github/workflows/e2e.yamlbuild/e2e.mkbuild/known-suites.txtbuild/suite-router.shdocs/ci.mdtests/e2e/CLAUDE.mdtests/e2e/auth_policy_test.gotests/e2e/custom_tls_test.gotests/e2e/elicitation_test.gotests/e2e/happy_path_test.gotests/e2e/multi_gateway_test.gotests/e2e/oauth_protected_resource_test.gotests/e2e/tool_discovery_test.gotests/e2e/tool_validation_test.gotests/e2e/url_elicitation_test.gotests/e2e/user_specific_list_test.go
💤 Files with no reviewable changes (2)
- .github/workflows/e2e-auth.yaml
- .github/workflows/e2e-nightly.yaml
✅ Files skipped from review due to trivial changes (12)
- tests/e2e/elicitation_test.go
- tests/e2e/tool_validation_test.go
- tests/e2e/oauth_protected_resource_test.go
- build/known-suites.txt
- tests/e2e/custom_tls_test.go
- .claude/rules/e2e-tests.md
- tests/e2e/url_elicitation_test.go
- tests/e2e/auth_policy_test.go
- tests/e2e/multi_gateway_test.go
- tests/e2e/user_specific_list_test.go
- tests/e2e/tool_discovery_test.go
- tests/e2e/happy_path_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- build/e2e.mk
jasonmadigan
force-pushed
the
1195-e2e-functional-suites
branch
from
dc310af to
af0075e
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e-on-demand.yaml:
- Around line 43-47: The shared-suite checkout in the e2e-on-demand workflow is
reading build/known-suites.txt from the default branch instead of the PR ref, so
suite validation can be out of sync with the later e2e-tests checkout. Update
the checkout step in the workflow to resolve and use the pull request head ref
before validating against known-suites.txt, and apply the same ref handling
anywhere else in the workflow that reads the suite catalog.
In @.github/workflows/e2e.yaml:
- Around line 221-225: The failure log collection step only runs the first debug
target because the shell exits on a non-zero return from ci-debug-logs, so
update the run block in the “Collect logs on failure” step to allow both make
ci-debug-logs and make ci-debug-test-servers-logs to execute even if one fails.
Use the existing step name and the two make targets as the key symbols when
adjusting the failure handling.
In `@build/suite-router.sh`:
- Line 20: The suite validation in suite-router.sh is using a word-match style
check against KNOWN_SUITES, which incorrectly accepts partial matches inside
hyphenated names. Update the membership logic in the suite lookup/validation
paths (including the checks around known-suites.txt, the suite dispatch, and the
label parsing branches) to require exact suite-name equality against the full
list of allowed suites. Use the existing suite-loading and validation symbols in
suite-router.sh so invalid inputs like auth, gateway, or specific are rejected
unless they exactly match a known suite name.
- Around line 76-93: The routing case in suite-router.sh is too broad in the
`case` statement and matches `internal/router/*` before the more specific router
branches. Reorder the `case` patterns so the specific
`internal/router/session*`, `internal/router/auth*`, and
`internal/router/elicit*` checks are evaluated before the generic
`internal/router/*|internal/ext_proc/*` catch-all, preserving the intended
`suggestions` handling in the `suggestions+=` blocks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3110980d-b979-4dfb-8472-19f4ff62b619
📒 Files selected for processing (20)
.claude/rules/e2e-tests.md.github/workflows/e2e-auth.yaml.github/workflows/e2e-nightly.yaml.github/workflows/e2e-on-demand.yaml.github/workflows/e2e.yamlbuild/e2e.mkbuild/known-suites.txtbuild/suite-router.shdocs/ci.mdtests/e2e/CLAUDE.mdtests/e2e/auth_policy_test.gotests/e2e/custom_tls_test.gotests/e2e/elicitation_test.gotests/e2e/happy_path_test.gotests/e2e/multi_gateway_test.gotests/e2e/oauth_protected_resource_test.gotests/e2e/tool_discovery_test.gotests/e2e/tool_validation_test.gotests/e2e/url_elicitation_test.gotests/e2e/user_specific_list_test.go
💤 Files with no reviewable changes (2)
- .github/workflows/e2e-nightly.yaml
- .github/workflows/e2e-auth.yaml
✅ Files skipped from review due to trivial changes (11)
- tests/e2e/oauth_protected_resource_test.go
- build/known-suites.txt
- tests/e2e/tool_validation_test.go
- tests/e2e/custom_tls_test.go
- tests/e2e/url_elicitation_test.go
- tests/e2e/auth_policy_test.go
- tests/e2e/user_specific_list_test.go
- tests/e2e/multi_gateway_test.go
- tests/e2e/tool_discovery_test.go
- .claude/rules/e2e-tests.md
- tests/e2e/elicitation_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
- tests/e2e/happy_path_test.go
- build/e2e.mk
- docs/ci.md
… PR runs Organise e2e tests into named functional suites so the PR gate runs a curated ~31-spec happy-path set, with named suites available on demand via /test-e2e comments and e2e/ labels. - fix shell injection in e2e-on-demand.yaml (env var, not inline interpolation) - add Ginkgo Label() annotations to all 79 e2e specs across 13 suites - centralise suite definitions in build/e2e.mk with new make targets - add build/suite-router.sh for CI suite selection and changed-file inference - evolve e2e-on-demand.yaml to support all named suites - add suite router, label triggers, and bot comment to e2e.yaml - prune redundant e2e-auth.yaml and e2e-nightly.yaml workflows - consolidate nightly schedule into e2e.yaml Closes Kuadrant#1195 Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
jasonmadigan
force-pushed
the
1195-e2e-functional-suites
branch
from
af0075e to
e315cb0
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e.yaml:
- Around line 70-76: Add a manual workflow input named suite under
on.workflow_dispatch so users can choose a suite when dispatching the workflow,
and make sure the Route suites step exports that value into DISPATCH_SUITE.
Update the e2e workflow around workflow_dispatch and the router step that runs
build/suite-router.sh so the suite selection is forwarded instead of always
falling back to the default.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 61eebca3-e8e9-4a98-a3d6-11eb13bdf161
📒 Files selected for processing (20)
.claude/rules/e2e-tests.md.github/workflows/e2e-auth.yaml.github/workflows/e2e-nightly.yaml.github/workflows/e2e-on-demand.yaml.github/workflows/e2e.yamlbuild/e2e.mkbuild/known-suites.txtbuild/suite-router.shdocs/ci.mdtests/e2e/CLAUDE.mdtests/e2e/auth_policy_test.gotests/e2e/custom_tls_test.gotests/e2e/elicitation_test.gotests/e2e/happy_path_test.gotests/e2e/multi_gateway_test.gotests/e2e/oauth_protected_resource_test.gotests/e2e/tool_discovery_test.gotests/e2e/tool_validation_test.gotests/e2e/url_elicitation_test.gotests/e2e/user_specific_list_test.go
💤 Files with no reviewable changes (2)
- .github/workflows/e2e-auth.yaml
- .github/workflows/e2e-nightly.yaml
✅ Files skipped from review due to trivial changes (11)
- tests/e2e/oauth_protected_resource_test.go
- build/known-suites.txt
- tests/e2e/tool_validation_test.go
- tests/e2e/user_specific_list_test.go
- tests/e2e/custom_tls_test.go
- tests/e2e/auth_policy_test.go
- tests/e2e/elicitation_test.go
- tests/e2e/multi_gateway_test.go
- tests/e2e/tool_discovery_test.go
- docs/ci.md
- tests/e2e/happy_path_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
- .claude/rules/e2e-tests.md
- tests/e2e/url_elicitation_test.go
- build/e2e.mk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Organise e2e tests into named functional suites. The required PR gate narrows to a curated ~31-spec happy-path set (
prlabel), with the current broader coverage (~61 specs) available aspr-extended. Named suites can be triggered on demand via/test-e2e <suite>comments ande2e/<suite>labels.What changed
e2e-on-demand.yaml(comment body via env var, not inline interpolation)Label()across 13 suitestest-e2e-pr,test-e2e-pr-extended,test-e2e-suite SUITE=Xinbuild/e2e.mkbuild/suite-router.shfor CI suite selection and changed-file inferencee2e-on-demand.yamlexpanded fromhappy|fullto all named suitese2e.yamle2e-auth.yamlande2e-nightly.yaml, consolidated nightly schedule intoe2e.yamldocs/ci.mdwith full suite referenceLabels
Apply these labels to a PR to trigger the corresponding e2e suite on top of the default
prgate:e2e/coree2e/routinge2e/sessionse2e/discoverye2e/promptse2e/auth-policye2e/trusted-headerse2e/elicitatione2e/url-elicitatione2e/user-specific-liste2e/tlse2e/multi-gatewaye2e/securitye2e/pr-extendede2e/fullUsage
See
docs/ci.mdfor full reference.Closes #1195
Summary by CodeRabbit
Label()metadata across E2E specs to match the new suite model (e.g., core, discovery, auth-policy, elicitation, url-elicitation, tls, multi-gateway).