Skip to content

e2e: move AdminKubeClient() from init to BeforeEach in router config_manager test#31335

Open
redhat-chai-bot wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:fix-router-init-panic
Open

e2e: move AdminKubeClient() from init to BeforeEach in router config_manager test#31335
redhat-chai-bot wants to merge 1 commit into
openshift:mainfrom
redhat-chai-bot:fix-router-init-panic

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Move kubeClient = oc.AdminKubeClient() from Ginkgo Describe-block scope (test tree construction time) to inside the BeforeEach block (test runtime) in test/extended/router/config_manager.go.

Root Cause

The AdminKubeClient() call at line 68 executes during Ginkgo's BuildTree() phase — when tests are being registered, not executed. If the kubeconfig file is unavailable at registration time (e.g. credential lifecycle issue, file cleanup race), AdminKubeClient() calls FatalErr() which panics the entire Ginkgo process.

This kills every test scheduled in that binary — not just the router tests — causing cascading failures across unrelated SIGs (sig-api-machinery, sig-apps, sig-auth, sig-cli, sig-node, etc.).

Impact

In the last 7 days on OCP 5.0:

  • 2 job runs triggered this exact panic in ci-5.0-e2e-gcp-ovn-upgrade
  • Those 2 runs produced 874 total test failures, of which 194 carried the router.init.func2 + config_manager.go:68 stacktrace
  • 137 distinct test names across unrelated SIGs were killed as collateral
  • The bug acts as a ~10× blast radius amplifier for credential lifecycle events

Fix

Move the single kubeClient = oc.AdminKubeClient() assignment into the existing BeforeEach block. The oc = exutil.NewCLIWithPodSecurityLevel(...) call remains at Describe scope — it only creates the CLI struct without making API calls.

This is the only AdminKubeClient() call at Describe scope across all router test files.

Failing Job

periodic-ci-openshift-release-main-ci-5.0-e2e-gcp-ovn-upgrade #2069723201876791296 — 435 test failures, 165 caused by the router.init.func2 panic.

Summary by CodeRabbit

  • Tests
    • Improved test setup timing so the Kubernetes client is initialized during each test’s setup phase.
    • This makes test behavior more consistent and better isolated between cases.

…manager test

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 24, 2026
@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b3b16287-3bfe-439c-8369-e9888f4e1c45

📥 Commits

Reviewing files that changed from the base of the PR and between 88018a6 and 01aa433.

📒 Files selected for processing (1)
  • test/extended/router/config_manager.go

Walkthrough

In test/extended/router/config_manager.go, the kubeClient = oc.AdminKubeClient() assignment is moved from before the g.BeforeEach registration to inside the g.BeforeEach callback, ensuring it executes per-test rather than once at suite wiring time.

Changes

Router config manager test setup

Layer / File(s) Summary
kubeClient assignment moved into BeforeEach
test/extended/router/config_manager.go
kubeClient is now assigned inside the g.BeforeEach function body instead of before that hook is registered.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: moving AdminKubeClient() setup into BeforeEach for the router e2e test.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR only moves kubeClient init; all Ginkgo titles in config_manager.go are static strings with no dynamic data.
Test Structure And Quality ✅ Passed PASS: kubeClient setup now runs in BeforeEach per spec; tests already use deferred pod cleanup and bounded waits, matching nearby router test patterns.
Microshift Test Compatibility ✅ Passed PASS: PR only moves AdminKubeClient() into BeforeEach; it adds no new Ginkgo tests and the suite is already apigroup:route.openshift.io, using only Route/SCC plus core K8s APIs.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new SNO-sensitive test was added; this change only moves kubeClient init, and the router specs shown don’t assume multiple nodes or HA topology.
Topology-Aware Scheduling Compatibility ✅ Passed Only moved kubeClient init in a router test; no manifests, controllers, or scheduling constraints were added.
Ote Binary Stdout Contract ✅ Passed No process-level stdout writes or init/main/TestMain hooks were found in router; the change only moves AdminKubeClient into BeforeEach.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new test logic was added; the file uses cluster-internal hostnames and IPv6-safe net.JoinHostPort, with no IPv4-only or public-network assumptions.
No-Weak-Crypto ✅ Passed Patch only moves kubeClient init; the changed file contains no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB or custom crypto code.
Container-Privileges ✅ Passed PR only moves kubeClient init in a Go test; no added privileged container/K8s manifest fields are present.
No-Sensitive-Data-In-Logs ✅ Passed The change only moves kubeClient initialization into BeforeEach; config_manager.go adds no new logging or sensitive-data exposure.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: redhat-chai-bot
Once this PR has been reviewed and has the lgtm label, please assign bentito for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot requested review from Miciah and knobunc June 24, 2026 16:41
@@ -65,9 +65,9 @@ var _ = g.Describe("[sig-network][Feature:Router][apigroup:route.openshift.io]",
// in order to allow binding 80/443 without being root. Without the privilege,
// the capability does not take effect and haproxy fails to start.
oc = exutil.NewCLIWithPodSecurityLevel("router-config-manager", api.LevelPrivileged)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The oc here is also suspect to me, I think this is also a previously known problem to have it outside in places that get run when ginkgo builds the tree

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See Chai's comment below, it really should've replied in the thread

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

Good question! I investigated this and NewCLIWithPodSecurityLevel at Describe scope is actually the intended pattern — the comment on NewCLI explicitly says: "Should be called outside of a Ginkgo .It() function."

Here's what it does at construction time (Describe scope):

  1. Creates the CLI struct — stores adminConfigPath = KubeConfigPath() which just reads the KUBECONFIG env var string, not the file
  2. Registers g.BeforeEach(cli.kubeFramework.BeforeEach) — deferred, runs later
  3. Calls setupStaticConfigsFromManifests() — reads from STATIC_CONFIG_MANIFEST_DIR if set (usually unset)
  4. Registers g.AfterEach(cli.TeardownProject) — deferred
  5. Registers g.BeforeEach(func() { cli.SetupProject() }) — deferred

The heavy lifting (kubeconfig file reads, API calls, project creation) all happens inside those deferred BeforeEach/AfterEach hooks at test runtime.

In contrast, AdminKubeClient() calls AdminConfig()GetClientConfig(path)os.ReadFile(kubeConfigFile) immediately, and FatalErr(err) if it fails — that's why it panics the process when called at Describe scope.

This is the standard pattern across hundreds of test files in openshift/origin — moving NewCLI into BeforeEach would be a much larger refactoring. The targeted fix here (moving only AdminKubeClient()) eliminates the cascading panic without disrupting the established pattern.

@stbenjam

Copy link
Copy Markdown
Member

/test verify-deps

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify-deps 01aa433 link true /test verify-deps

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/retest verify-deps

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test e2e-aws-csi
/test e2e-aws-jenkins
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-image-registry
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-builds
/test e2e-gcp-ovn-image-ecosystem
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test go-verify-deps
/test images
/test lint
/test okd-scos-images
/test unit
/test verify
/test verify-deps
/test verify-image-manifest-lists

The following commands are available to trigger optional jobs:

/test e2e-agnostic-ovn-cmd
/test e2e-aws-disruptive
/test e2e-aws-etcd-certrotation
/test e2e-aws-etcd-recovery
/test e2e-aws-ovn
/test e2e-aws-ovn-cgroupsv2
/test e2e-aws-ovn-dra-example
/test e2e-aws-ovn-edge-zones
/test e2e-aws-ovn-etcd-scaling
/test e2e-aws-ovn-kube-apiserver-rollout
/test e2e-aws-ovn-kubevirt
/test e2e-aws-ovn-serial-fast
/test e2e-aws-ovn-serial-ipsec
/test e2e-aws-ovn-serial-publicnet-1of2
/test e2e-aws-ovn-serial-publicnet-2of2
/test e2e-aws-ovn-single-node
/test e2e-aws-ovn-single-node-serial
/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-single-node-techpreview-serial
/test e2e-aws-ovn-single-node-upgrade
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-rollback
/test e2e-aws-ovn-upi
/test e2e-aws-proxy
/test e2e-aws-tls-observed-config
/test e2e-aws-tls-observed-config-fips
/test e2e-aws-tls-observed-config-hypershift
/test e2e-azure
/test e2e-azure-ovn-etcd-scaling
/test e2e-azure-ovn-upgrade
/test e2e-baremetalds-kubevirt
/test e2e-external-aws
/test e2e-external-aws-ccm
/test e2e-external-vsphere-ccm
/test e2e-gcp-disruptive
/test e2e-gcp-fips-serial-1of2
/test e2e-gcp-fips-serial-2of2
/test e2e-gcp-ovn-etcd-scaling
/test e2e-gcp-ovn-kube-apiserver-rollout
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-ovn-techpreview
/test e2e-gcp-ovn-techpreview-serial-1of2
/test e2e-gcp-ovn-techpreview-serial-2of2
/test e2e-gcp-ovn-usernamespace
/test e2e-hypershift-conformance
/test e2e-metal-ipi-ovn
/test e2e-metal-ipi-ovn-bgp-virt-dualstack
/test e2e-metal-ipi-ovn-bgp-virt-dualstack-techpreview
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-dualstack-bgp
/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw
/test e2e-metal-ipi-ovn-dualstack-local-gateway
/test e2e-metal-ipi-ovn-kube-apiserver-rollout
/test e2e-metal-ipi-serial-1of2
/test e2e-metal-ipi-serial-2of2
/test e2e-metal-ipi-serial-ovn-ipv6-1of2
/test e2e-metal-ipi-serial-ovn-ipv6-2of2
/test e2e-metal-ipi-virtualmedia
/test e2e-metal-ovn-single-node-live-iso
/test e2e-metal-ovn-single-node-with-worker-live-iso
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery
/test e2e-openstack-dualstack-v6primary
/test e2e-openstack-ovn
/test e2e-openstack-serial
/test e2e-vsphere-ovn-etcd-scaling
/test okd-scos-e2e-aws-ovn

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-origin-main-go-verify-deps
pull-ci-openshift-origin-main-images
pull-ci-openshift-origin-main-lint
pull-ci-openshift-origin-main-okd-scos-images
pull-ci-openshift-origin-main-unit
pull-ci-openshift-origin-main-verify
pull-ci-openshift-origin-main-verify-deps
Details

In response to this:

/retest verify-deps

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@sosiouxme

Copy link
Copy Markdown
Member

proxy.golang.org seems to be blocked?

@redhat-chai-bot

Copy link
Copy Markdown
Contributor Author

/retest verify-deps

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test e2e-aws-csi
/test e2e-aws-jenkins
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-image-registry
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-builds
/test e2e-gcp-ovn-image-ecosystem
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi
/test go-verify-deps
/test images
/test lint
/test okd-scos-images
/test unit
/test verify
/test verify-deps
/test verify-image-manifest-lists

The following commands are available to trigger optional jobs:

/test e2e-agnostic-ovn-cmd
/test e2e-aws-disruptive
/test e2e-aws-etcd-certrotation
/test e2e-aws-etcd-recovery
/test e2e-aws-ovn
/test e2e-aws-ovn-cgroupsv2
/test e2e-aws-ovn-dra-example
/test e2e-aws-ovn-edge-zones
/test e2e-aws-ovn-etcd-scaling
/test e2e-aws-ovn-kube-apiserver-rollout
/test e2e-aws-ovn-kubevirt
/test e2e-aws-ovn-serial-fast
/test e2e-aws-ovn-serial-ipsec
/test e2e-aws-ovn-serial-publicnet-1of2
/test e2e-aws-ovn-serial-publicnet-2of2
/test e2e-aws-ovn-single-node
/test e2e-aws-ovn-single-node-serial
/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-single-node-techpreview-serial
/test e2e-aws-ovn-single-node-upgrade
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-rollback
/test e2e-aws-ovn-upi
/test e2e-aws-proxy
/test e2e-aws-tls-observed-config
/test e2e-aws-tls-observed-config-fips
/test e2e-aws-tls-observed-config-hypershift
/test e2e-azure
/test e2e-azure-ovn-etcd-scaling
/test e2e-azure-ovn-upgrade
/test e2e-baremetalds-kubevirt
/test e2e-external-aws
/test e2e-external-aws-ccm
/test e2e-external-vsphere-ccm
/test e2e-gcp-disruptive
/test e2e-gcp-fips-serial-1of2
/test e2e-gcp-fips-serial-2of2
/test e2e-gcp-ovn-etcd-scaling
/test e2e-gcp-ovn-kube-apiserver-rollout
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-ovn-techpreview
/test e2e-gcp-ovn-techpreview-serial-1of2
/test e2e-gcp-ovn-techpreview-serial-2of2
/test e2e-gcp-ovn-usernamespace
/test e2e-hypershift-conformance
/test e2e-metal-ipi-ovn
/test e2e-metal-ipi-ovn-bgp-virt-dualstack
/test e2e-metal-ipi-ovn-bgp-virt-dualstack-techpreview
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-dualstack-bgp
/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw
/test e2e-metal-ipi-ovn-dualstack-local-gateway
/test e2e-metal-ipi-ovn-kube-apiserver-rollout
/test e2e-metal-ipi-serial-1of2
/test e2e-metal-ipi-serial-2of2
/test e2e-metal-ipi-serial-ovn-ipv6-1of2
/test e2e-metal-ipi-serial-ovn-ipv6-2of2
/test e2e-metal-ipi-virtualmedia
/test e2e-metal-ovn-single-node-live-iso
/test e2e-metal-ovn-single-node-with-worker-live-iso
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery
/test e2e-openstack-dualstack-v6primary
/test e2e-openstack-ovn
/test e2e-openstack-serial
/test e2e-vsphere-ovn-etcd-scaling
/test okd-scos-e2e-aws-ovn

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-origin-main-go-verify-deps
pull-ci-openshift-origin-main-images
pull-ci-openshift-origin-main-lint
pull-ci-openshift-origin-main-okd-scos-images
pull-ci-openshift-origin-main-unit
pull-ci-openshift-origin-main-verify
pull-ci-openshift-origin-main-verify-deps
Details

In response to this:

/retest verify-deps

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants