feat: add velero (umbrella tracking issue)

[patrick-hermann-sva]

Goal

Stand up Velero as a first-class flux app, deploy it to the platform-sthings cluster against in-cluster MinIO, and unblock the restore-hook work in #115 (which depends on #111).

Done

Base in this repo (infra/velero/)

• feat(velero): add infra/velero base with S3/MinIO + optional ESO #119 — feat(velero): add infra/velero base with S3/MinIO + optional ESO
• feat(velero): add trust-bundle component for self-signed S3 #121 — feat(velero): add trust-bundle component for self-signed S3 endpoints
• fix(velero): use plain Secret instead of sthings-cluster helper #122 — fix(velero): use plain Secret manifest instead of sthings-cluster helper (the helper wraps customresources in spec:, invalid for core Secret)
• fix(velero): override CRD-install hook kubectl image to bitnamilegacy #123 — fix(velero): override CRD-install hook kubectl image to bitnamilegacy (bitnami sunset docker.io/bitnami/kubectl:1.35 late 2025)
• fix(velero): use literal cloud-credentials Secret name #124 — fix(velero): use literal cloud-credentials Secret name (placeholder in metadata.name broke consumer kustomize patch targets)
• docs(velero): document MinIO bucket-scoped user provisioning #126 — docs(velero): document MinIO bucket-scoped user provisioning (MinIO Console + Terraform paths)
• feat(velero): bake trust-bundle mount into base, gate via SSL_CERT_DIR #127 — feat(velero): bake trust-bundle mount into base, gate via SSL_CERT_DIR (eliminates the inline-patches footgun; pattern-matches apps/clusterbook-operator)
• docs(velero): document ServiceMonitor CRD prerequisite #128 — docs(velero): document ServiceMonitor CRD prerequisite
• README + TechDocs page + mkdocs.yml nav + docs/infra/index.md row
• v1.6.0 tag cut (f4a1a29) via semantic-release — release notes at https://github.com/stuttgart-things/flux/releases/tag/v1.6.0
• v1.7.0 tag cut (c90cd82) via semantic-release — picks up feat(velero): bake trust-bundle mount into base, gate via SSL_CERT_DIR #127 (trust-bundle baked into base) and docs(velero): document ServiceMonitor CRD prerequisite #128 (ServiceMonitor docs)

Platform-sthings overlay (stuttgart-things/stuttgart-things)

• stuttgart-things/stuttgart-things#2200 — feat(platform-sthings): deploy velero to MinIO velero-test bucket
• stuttgart-things/stuttgart-things#2202 — fix(platform-sthings): retarget velero cred patch to plain Secret
• stuttgart-things/stuttgart-things#2203 — chore(platform-sthings): bump flux-infra to v1.6.0, pin velero to it (overlay now references the stable tag instead of apps-upstream/main)
• stuttgart-things/stuttgart-things#2204 — chore(platform-sthings): velero uses dedicated MinIO svc account (off MinIO root creds onto bucket-scoped user velero with policy velero-test-rw; SOPS-encrypted velero-s3-credentials Secret)
• stuttgart-things/stuttgart-things#2205 — chore(platform-sthings): velero on v1.7.0 base, drop redundant trust-bundle patches, add nightly Schedule (bumped flux-infra v1.6.0 → v1.7.0; replaced inline patches with single VELERO_SSL_CERT_DIR: /etc/ssl/custom; added daily-all Schedule)
• Verified live: BackupStorageLocation/default is Available; smoke backup smoke-test-1779023397 (23 items) completed against velero-test bucket
• TLS via trust-manager cluster-trust-bundle ConfigMap + AWS_CA_BUNDLE verified
• End-to-end restore test — backup restore-test-1779027554 of throwaway velero-restore-test namespace (21 items) → ns deleted → restore (14 user-restorable items) → ConfigMap data, Secret api-key, and Deployment all came back identical. Two benign warnings (cluster-scoped ciliumendpoints.cilium.io CRD + auto-generated kube-root-ca.crt skipped) — expected Velero behavior. Test artefacts cleaned up.
• Daily Schedule live — Schedule/daily-all (Enabled, cron 0 2 * * *, 30-day TTL, full cluster minus kube-{system,public,node-lease}). Manually triggered daily-all-20260517145502 for verification → Completed, 1713 items backed up.

Outstanding

Followups in platform-sthings overlay

• Re-enable ServiceMonitor once kube-prometheus-stack (or the prometheus-operator CRDs) is on platform-sthings.
• Pod-restart-after-Secret-update wrinkle: AWS SDK reads creds at startup. Won't recur on fresh deploys but matters for rotation — consider stakater/Reloader.

Backup hygiene / next features

• Node-agent for filesystem PV backup (VELERO_DEPLOY_NODE_AGENT=true) if PV data needs to be in scope; default is off.
• Volume snapshots — currently VELERO_SNAPSHOTS_ENABLED=false; decide CSI snapshot strategy.

Crossplane-specific (parent work)

• Velero restore hooks for provider readiness gating #115 — Velero restore hooks for Crossplane provider readiness gating (now unblocked).
• Document scope: MR/XR/Claim selectors per the parent Design: Reconcile dedicated Crossplane control planes (sharded by domain) from this repo #111 backup section (avoid full-namespace dumps that cause finalizer races on restore).

Related

• Parent: Design: Reconcile dedicated Crossplane control planes (sharded by domain) from this repo #111 (sharded Crossplane control planes)
• Child: Velero restore hooks for provider readiness gating #115 (restore hooks for provider readiness)

[patrick-hermann-sva]

Done: v1.6.0 tag cut on this repo (commit f4a1a29); GitHub draft release at https://github.com/stuttgart-things/flux/releases/tag/v1.6.0. Platform-sthings overlay bumped flux-infra GitRepository v1.5.0 → v1.6.0 and re-sourced infra/velero.yaml from apps-upstream (main) → flux-infra (v1.6.0) in stuttgart-things/stuttgart-things#2203. All other infra Kustomizations re-applied cleanly against the new tag with no health regressions; BackupStorageLocation/default remains Available.

[patrick-hermann-sva]

Done: velero on platform-sthings is now using a dedicated bucket-scoped MinIO user (velero with policy velero-test-rw, created via the MinIO Console). Wiring landed in stuttgart-things/stuttgart-things#2204 (SOPS-encrypted velero-s3-credentials Secret + Kustomization switched off the root minio-secrets reference). Provisioning steps (Console + Terraform alternative) documented in #126. BackupStorageLocation/default still Available after pod restart.

[patrick-hermann-sva]

Done: End-to-end restore test successful.

• Backup restore-test-1779027554 of throwaway namespace velero-restore-test (21 items, ConfigMap + Secret + Deployment + Service + auto-generated bits) — Completed
• Namespace deleted, then velero restore create --from-backup ... — Completed, 14 user-restorable items
• All restored content verified identical: ConfigMap data, Secret api-key, Deployment ready with 1/1 replica
• Two benign warnings noted (cluster-scoped ciliumendpoints.cilium.io CRD skipped because it already exists; auto-generated kube-root-ca.crt skipped) — expected Velero behavior, not actionable
• Cleanup: test namespace deleted, restore + backup CRs deleted

Confirms the full backup/restore round-trip works against the platform-sthings MinIO velero-test bucket via the dedicated svc account.

[patrick-hermann-sva]

End of session 2026-05-17.

Today's net result: velero is live on platform-sthings against in-cluster MinIO, with bucket-scoped svc account, trust-manager TLS, and a daily Schedule running. 7 PRs in this repo (#119, #121, #122, #123, #124, #126, #127, #128) + 6 PRs on stuttgart-things/stuttgart-things (#2200, #2202, #2203, #2204, #2205). Two tags cut (v1.6.0, v1.7.0).

Pickup points for next session, in suggested order:

1. Velero restore hooks for provider readiness gating #115 — Velero restore hooks for Crossplane provider readiness. This was the original ziel; everything we did today was groundwork. Now unblocked.
2. stakater/Reloader as a small infra add — auto-restart velero pod when cloud-credentials Secret rotates. ~30min job.
3. Node-agent / Volume snapshots decisions if PV data needs to be in backup scope (currently metadata only).