A reconcile that needs external DNS makes a broken cluster unrepairable by GitOps #3405

🤖 Generated by the Agentic Engineer

Evidence

During today's DNS incident (#3404), a single bad network policy became unrecoverable through GitOps. The failure chain was circular:

1. A policy applied by the infrastructure Kustomization broke cluster DNS.
2. infrastructure-controllers then failed to build, because its kustomize render fetches resources over HTTPS and that needs DNS:
   kustomize build failed: ... Could not resolve host: raw.githubusercontent.com
3. infrastructure dependsOn infrastructure-controllers, so it stopped applying — and infrastructure is the unit that owns the offending policy.

The change that would fix the cluster could not be applied because the cluster was broken. Recovery required a manual kubectl delete against prod, in a specific order relative to the merge, or Flux would have re-applied the same policy.

Three resources are fetched at reconcile time today:

• controllers/kubelet-serving-cert-approver/ → raw.githubusercontent.com/.../ha-install.yaml
• controllers/origin-ca-issuer/ → two raw.githubusercontent.com/.../*.yaml CRDs

Problem

Every reconcile of infrastructure-controllers depends on the cluster reaching raw.githubusercontent.com. That makes cluster-internal networking a hard prerequisite for the mechanism we rely on to repair cluster-internal networking, and it removes the property GitOps is supposed to give us: that reverting the commit restores the cluster.

It is also a supply-chain and availability exposure independent of this incident — an upstream outage, a repo rename, or a deleted tag stalls reconciliation of every controller in that layer.

Hypothesis

Vendoring those manifests into the repository makes infrastructure-controllers build offline. The layer then reconciles from local content alone, so a bad change stays revertible by merge, and recovery does not need a human with cluster credentials.

Success signal

• kubectl kustomize k8s/providers/hetzner/infrastructure/controllers renders with no network access.
• Baseline: 3 remote fetches at reconcile time. Target: 0.
• Guardrail: rendered output is byte-identical to the current render, so this is behaviour-preserving.

Smallest useful change

Vendor the three manifests under the directories that consume them, keeping the pinned upstream versions unchanged, and point Renovate's existing github-releases custom manager at the vendored copies so version bumps stay automatic. Record the upstream source and tag next to each vendored file.

Acceptance criteria

• No kustomization.yaml under k8s/ lists an https:// resource.
• The controllers layer renders identically before and after (diff of the two renders is empty).
• Renovate still proposes upgrades for all three, and a bump updates the vendored content.
• A CI check fails if a remote resource URL is reintroduced, so this cannot regress silently.

Rough size: S–M. Mechanical, but touches three controller directories and the Renovate config.