Skip to content

[Multi_K8s-Plugin] Prune orphaned resources on rollback#6748

Merged
Warashi merged 15 commits into
pipe-cd:masterfrom
mohammedfirdouss:feat/multicluster-rollback-pruning
May 29, 2026
Merged

[Multi_K8s-Plugin] Prune orphaned resources on rollback#6748
Warashi merged 15 commits into
pipe-cd:masterfrom
mohammedfirdouss:feat/multicluster-rollback-pruning

Conversation

@mohammedfirdouss

@mohammedfirdouss mohammedfirdouss commented May 7, 2026

Copy link
Copy Markdown
Contributor

What this PR does / Why we need it:

Implements rollback pruning in the kubernetes_multicluster plugin. When a deployment pipeline fails mid-way, the ROLLBACK stage re-applies the last known good manifests but any new resource introduced by the failed deployment (e.g. a ConfigMap or Service already applied to the cluster) was never deleted, leaving orphaned resources that cause permanent live-state drift.

After applying the running manifests and cleaning up canary/baseline variants, rollback now:

  1. Loads the target (failed) manifests from TargetDeploymentSource
  2. Identifies resources present in target but absent from running (findOrphanedKeys)
  3. Deletes those resources from the cluster (pruneOrphanedResources)

Errors during target manifest loading are non-fatal, logged and skipped so a Helm/Kustomize rendering failure cannot prevent the rollback from completing.

Which issue(s) this PR fixes:

Part of #6446

How was this tested:

Unit tests added to deployment/misc_test.go for findOrphanedKeys covering:

  • No orphans when target equals running
  • All resources orphaned when running is empty
  • New resource in target not present in running is orphaned
  • Resource only in running is not pruned
  • Multiple orphaned resources

Note on single-cluster plugin:

The same TODO exists in pkg/app/pipedv1/plugin/kubernetes/deployment/rollback.go. That plugin is out of scope for this PR.

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>
@mohammedfirdouss mohammedfirdouss force-pushed the feat/multicluster-rollback-pruning branch from 6edf250 to 41df9e4 Compare May 7, 2026 22:44
@codecov

codecov Bot commented May 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.62500% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.14%. Comparing base (b5038da) to head (2985a48).

Files with missing lines Patch % Lines
.../plugin/kubernetes_multicluster/deployment/misc.go 85.71% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #6748       +/-   ##
===========================================
+ Coverage   29.33%   62.14%   +32.81%     
===========================================
  Files         598       30      -568     
  Lines       63902     3751    -60151     
===========================================
- Hits        18744     2331    -16413     
+ Misses      43711     1235    -42476     
+ Partials     1447      185     -1262     
Flag Coverage Δ
. ?
.-pkg-app-pipedv1-plugin-analysis ?
.-pkg-app-pipedv1-plugin-ecs ?
.-pkg-app-pipedv1-plugin-kubernetes ?
.-pkg-app-pipedv1-plugin-kubernetes_multicluster 62.14% <90.62%> (+0.24%) ⬆️
.-pkg-app-pipedv1-plugin-scriptrun ?
.-pkg-app-pipedv1-plugin-terraform ?
.-pkg-app-pipedv1-plugin-wait ?
.-pkg-app-pipedv1-plugin-waitapproval ?
.-pkg-plugin-sdk ?
.-tool-actions-gh-release ?
.-tool-actions-plan-preview ?
.-tool-codegen-protoc-gen-auth ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/rollback.go Outdated
Warashi
Warashi previously approved these changes May 24, 2026
Comment thread pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/misc_test.go Outdated
} else if len(namespacedLiveResources)+len(clusterScopedLiveResources) > 0 {
removeKeys := provider.FindRemoveResources(manifests, namespacedLiveResources, clusterScopedLiveResources)
if len(removeKeys) == 0 {
lp.Info("There are no live resources to prune")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good to log this info even when len(namespacedLiveResources)+len(clusterScopedLiveResources) == 0.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah exactly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohammedfirdouss This does not seem to be resolved. Please check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lint check failed for now...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the log too

@mohammedfirdouss

Copy link
Copy Markdown
Contributor Author

@Warashi this need review.

… are found

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>
… are found

Signed-off-by: Mohammed Firdous <124298708+mohammedfirdouss@users.noreply.github.com>

@Warashi Warashi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Warashi Warashi merged commit 772e2d9 into pipe-cd:master May 29, 2026
60 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for contributing to PipeCD! The changes in this pull request will be part of the upcoming release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants