Skip to content

NO-JIRA: inject kms sidecar in preflight deployer#2321

Open
tjungblu wants to merge 1 commit into
openshift:masterfrom
tjungblu:inject_sidecar
Open

NO-JIRA: inject kms sidecar in preflight deployer#2321
tjungblu wants to merge 1 commit into
openshift:masterfrom
tjungblu:inject_sidecar

Conversation

@tjungblu

@tjungblu tjungblu commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Bug Fixes

    • Improved KMS preflight deployment to properly manage plugin configuration and ephemeral encryption secrets.
    • Enhanced cleanup process to remove temporary secrets alongside pod deletion.
  • Tests

    • Updated KMS preflight tests to validate new pod composition and secret management workflows.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 22, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@tjungblu: This pull request explicitly references no jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 944d5f21-8d04-446f-8a63-55ee82315dd9

📥 Commits

Reviewing files that changed from the base of the PR and between c5a586b and 4575b08.

📒 Files selected for processing (4)
  • pkg/operator/encryption/controllers/kms_provider_config.go
  • pkg/operator/encryption/kms/preflight/cmd.go
  • pkg/operator/encryption/kms/preflight/deployer.go
  • pkg/operator/encryption/kms/preflight/deployer_test.go
💤 Files with no reviewable changes (1)
  • pkg/operator/encryption/kms/preflight/cmd.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • pkg/operator/encryption/kms/preflight/deployer.go
  • pkg/operator/encryption/kms/preflight/deployer_test.go

Walkthrough

Adds BuildEphemeralPreflightEncryptionConfig in controllers/kms_provider_config.go to construct a one-shot apiserver encryption config by resolving referenced Secrets/ConfigMaps from openshift-config. The KMS preflight Deploy interface and PodPreflightDeployer implementation are updated to accept a pluginConfig, use this new function to materialize an ephemeral encryption Secret, inject the KMS plugin sidecar via pluginlifecycle, then create the preflight Pod. Cleanup now also deletes the ephemeral Secret.

Changes

KMS Preflight Ephemeral Encryption Config and Sidecar Injection

Layer / File(s) Summary
BuildEphemeralPreflightEncryptionConfig function
pkg/operator/encryption/controllers/kms_provider_config.go
New function that derives a KMS provider config from KMSPluginConfig, resolves referenced Secrets and ConfigMaps from openshift-config, and returns an encryptiondata.Config with inline EncryptionConfiguration and plugin reference data.
Deployer interface, constants, Deploy + Cleanup implementation, constructor
pkg/operator/encryption/kms/preflight/deployer.go, pkg/operator/encryption/kms/preflight/cmd.go
KMSPreflightDeployer and PodPreflightDeployer.Deploy now accept pluginConfig. kmsSocketEndpoint constant moves from cmd.go to deployer.go alongside new preflight container/secret name constants. Deploy calls BuildEphemeralPreflightEncryptionConfig, applies an ephemeral encryption Secret via resourceapply.ApplySecret, injects the KMS plugin sidecar via pluginlifecycle.NewKMSPluginBuilder, then creates the Pod. Cleanup additionally deletes the ephemeral Secret. NewPodPreflightDeployer wires in eventRecorder.
Deployer tests
pkg/operator/encryption/kms/preflight/deployer_test.go
Passes DefaultKMSPluginConfig to Deploy, prepends reference Secret/ConfigMap objects to the fake clientset, extends expected Pod YAML with vault-kms-plugin-1 init container and kms-plugins-data volume, and updates Cleanup scenarios to assert both pod and Secret deletion, including a new forbidden-delete error case.

Sequence Diagram

sequenceDiagram
  rect rgba(173, 216, 230, 0.5)
    Note over Caller,KubeClient: Deploy flow
    Caller->>PodPreflightDeployer: Deploy(ctx, configHash, pluginConfig)
    PodPreflightDeployer->>BuildEphemeralPreflightEncryptionConfig: pluginConfig, encryptionKeyID, socketEndpoint
    BuildEphemeralPreflightEncryptionConfig->>KubeClient: Get Secret from openshift-config
    BuildEphemeralPreflightEncryptionConfig->>KubeClient: Get ConfigMap from openshift-config
    BuildEphemeralPreflightEncryptionConfig-->>PodPreflightDeployer: encryptiondata.Config
    PodPreflightDeployer->>KubeClient: ApplySecret(ephemeral encryption Secret)
    PodPreflightDeployer->>KMSPluginBuilder: FromEncryptionConfig(...).Apply(pod.Spec)
    KMSPluginBuilder-->>PodPreflightDeployer: pod.Spec with sidecar injected
    PodPreflightDeployer->>KubeClient: Create Pod
  end
  rect rgba(255, 200, 150, 0.5)
    Note over Caller,KubeClient: Cleanup flow
    Caller->>PodPreflightDeployer: Cleanup(ctx)
    PodPreflightDeployer->>KubeClient: Delete Pod (ignore not-found)
    PodPreflightDeployer->>KubeClient: Delete ephemeral encryption Secret (ignore not-found)
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • openshift/library-go#2307: Modifies pkg/operator/encryption/kms/preflight/deployer.go around the Deploy method to add a pluginConfig parameter, which this PR expands into full ephemeral Secret creation and sidecar injection.

Suggested labels

approved, lgtm

Suggested reviewers

  • deads2k

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 2 warnings, 1 inconclusive)

Check name Status Explanation Resolution
Container-Privileges ❌ Error Main 'kms-preflight-check' container lacks securityContext with allowPrivilegeEscalation: false and capability restrictions; init container is properly hardened. Add securityContext to kms-preflight-check container with allowPrivilegeEscalation: false, drop ALL capabilities, readOnlyRootFilesystem, and seccompProfile like the init container.
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Topology-Aware Scheduling Compatibility ⚠️ Warning Pod YAML includes nodeSelector: node-role.kubernetes.io/master: "" without topology-awareness checks; on HyperShift, no control-plane nodes exist in-cluster, causing pod to remain Pending. Check ControlPlaneTopology before deploying; on External/HyperShift, use appropriate nodeSelector or remove it for preflight pod scheduling flexibility.
Test Structure And Quality ❓ Inconclusive Custom check specifies Ginkgo test code review, but the PR contains only standard Go testing (testing.T) with no Ginkgo imports or patterns. Clarify if the check should apply to standard Go testing patterns as well, or if the check is only for Ginkgo-based test files.
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main objective: injecting a KMS sidecar in the preflight deployer, which is the central feature across multiple file changes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo tests found in PR. All test files use standard Go testing (TestXxx functions), making this check not applicable.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests are added in this PR. All changes are to implementation files (cmd.go, deployer.go, kms_provider_config.go) and unit tests (deployer_test.go using standard testing package,...
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests were added or modified in this PR. Changes are operator code and unit tests using Go's standard testing package, not e2e tests subject to SNO compatibility requirements.
Ote Binary Stdout Contract ✅ Passed No OTE Binary Stdout Contract violations found. The PR's only logging calls (klog.Infof in cmd.go lines 50, 65) are inside the run() callback method within a cobra.Command handler, not at process l...
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests were added in this PR. The only test file modified (deployer_test.go) contains standard Go unit tests using testing.T, not Ginkgo patterns. Check does not apply.
No-Weak-Crypto ✅ Passed No weak cryptographic algorithms (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or non-constant-time secret comparisons detected in the modified files.
No-Sensitive-Data-In-Logs ✅ Passed PR contains only 2 logging statements which output socket endpoint path and timing info; all error messages reference only Kubernetes metadata and infrastructure paths, never exposing passwords, to...
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from ardaguclu and dgrisonnet June 22, 2026 10:44
@openshift-ci

openshift-ci Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tjungblu
Once this PR has been reviewed and has the lgtm label, please assign ardaguclu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
pkg/operator/encryption/kms/preflight/deployer.go (1)

71-73: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Wrap pod creation failures with pod identity context.

Line 71 currently returns the raw create error, which makes controller logs harder to triage under failure.

Suggested patch
-	_, err = d.coreClient.Pods(d.namespace).Create(ctx, pod, metav1.CreateOptions{})
-	return err
+	if _, err = d.coreClient.Pods(d.namespace).Create(ctx, pod, metav1.CreateOptions{}); err != nil {
+		return fmt.Errorf("failed to create preflight pod %s/%s: %w", d.namespace, preflightPodName, err)
+	}
+	return nil
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/operator/encryption/kms/preflight/deployer.go` around lines 71 - 73, The
pod creation error returned on line 71 lacks context about which pod failed to
create, making it difficult to triage failures in controller logs. Wrap the
error returned from d.coreClient.Pods(d.namespace).Create() with the pod's
identity information (such as the pod name) using error wrapping to provide
clear context about which pod creation failed.
pkg/operator/encryption/kms/pluginlifecycle/builder_test.go (1)

88-101: 🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Add a regression case for override + multi-plugin rejection.

Line 88 covers the happy path, but the new guard in Apply (socket endpoint override supports a single KMS plugin only) should also be asserted in this table to prevent regressions.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/operator/encryption/kms/pluginlifecycle/builder_test.go` around lines 88
- 101, The test table in this file covers the happy path for socket endpoint
override but lacks a regression test case to verify the validation guard that
rejects socket endpoint overrides when multiple KMS plugins are configured. Add
a new test case in the same table-driven test structure (after or alongside the
existing "socket endpoint override" case) that configures multiple KMS plugins
in the encryption config and attempts to apply a socket endpoint override via
WithSocketEndpoint, then verify in the test that the Apply method properly
rejects this configuration with an appropriate error indicating that socket
endpoint override supports a single KMS plugin only.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/operator/encryption/kms/pluginlifecycle/builder_test.go`:
- Around line 88-101: The test table in this file covers the happy path for
socket endpoint override but lacks a regression test case to verify the
validation guard that rejects socket endpoint overrides when multiple KMS
plugins are configured. Add a new test case in the same table-driven test
structure (after or alongside the existing "socket endpoint override" case) that
configures multiple KMS plugins in the encryption config and attempts to apply a
socket endpoint override via WithSocketEndpoint, then verify in the test that
the Apply method properly rejects this configuration with an appropriate error
indicating that socket endpoint override supports a single KMS plugin only.

In `@pkg/operator/encryption/kms/preflight/deployer.go`:
- Around line 71-73: The pod creation error returned on line 71 lacks context
about which pod failed to create, making it difficult to triage failures in
controller logs. Wrap the error returned from
d.coreClient.Pods(d.namespace).Create() with the pod's identity information
(such as the pod name) using error wrapping to provide clear context about which
pod creation failed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 59370520-0596-4b40-b42f-31838c378ade

📥 Commits

Reviewing files that changed from the base of the PR and between cb15bee and da70484.

📒 Files selected for processing (5)
  • pkg/operator/encryption/kms/pluginlifecycle/builder.go
  • pkg/operator/encryption/kms/pluginlifecycle/builder_test.go
  • pkg/operator/encryption/kms/preflight/cmd.go
  • pkg/operator/encryption/kms/preflight/deployer.go
  • pkg/operator/encryption/kms/preflight/deployer_test.go
💤 Files with no reviewable changes (1)
  • pkg/operator/encryption/kms/preflight/cmd.go


type kmsConfigHasher struct {
provider kmsProviderConfig
provider KMSProviderConfig

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of exposing this, can't we move the buildephemeralencryptionconfig function here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's sensible

// Deploy creates the preflight checker with the given config hash
Deploy(ctx context.Context, configHash string) error
// Deploy creates the preflight checker with the given config hash and KMS plugin configuration.
Deploy(ctx context.Context, configHash string, pluginConfig configv1.KMSPluginConfig) error

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and pass encryptionconfig name in the namespace here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the other comment I understood, I've moved the ephemeral builder to the other package. What should I replace this with?

}

err = pluginlifecycle.NewKMSPluginBuilder().
FromEncryptionConfig(preflightEncryptionConfigSecretName, encryptionConfig).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

builder only needs an existed secret name

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's fair, but the pods needs a real secret to run and I need to create it. What should I change here?

}

func ephemeralEncryptionConfigSecret(namespace string, cfg *encryptiondata.Config) (*corev1.Secret, error) {
return encryptiondata.ToSecret(namespace, preflightEncryptionConfigSecretName, cfg)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may also need a flag to omit the finalizer, otherwise we won't be able to delete it easily

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manually deleting the finalizer from the value returned from ToSecret might work?

@tjungblu tjungblu force-pushed the inject_sidecar branch 2 times, most recently from c5a586b to 84b0efc Compare June 22, 2026 15:04
Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
@openshift-ci

openshift-ci Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

@tjungblu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/unit 4575b08 link true /test unit
ci/prow/verify 4575b08 link true /test verify

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

}

func (d *PodPreflightDeployer) Deploy(ctx context.Context, configHash string) error {
func (d *PodPreflightDeployer) Deploy(ctx context.Context, configHash string, pluginConfig configv1.KMSPluginConfig) error {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logical flow in that function makes sense to me. I think, we just may need to agree upon the how we align built-in functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants