openshift 4.22.0 spec targets Ignition 3.6.0, which the OCP 4.22 MCO does not support

[tmkymst]

Summary

The stable openshift variant version 4.22.0 (stabilized in Butane 0.28.0) generates MachineConfig objects with ignition.version: 3.6.0. However, the Machine Config Operator (MCO) shipped in OpenShift Container Platform 4.22 supports Ignition config spec only up to 3.5.0. A MachineConfig produced by variant: openshift / version: 4.22.0 is therefore rejected by the MCO at render time and degrades the target MachineConfigPool.

This is a direct recurrence of #312 ( openshift experimental spec uses an Ignition spec not supported by the MCO), one Ignition-spec cycle later: 3.6.0 vs the MCO's 3.5.0, mirroring the earlier 3.3.0 vs 3.2.0 that was rolled back in #269 and #307.

Expected

A MachineConfig generated for OCP 4.22 using the documented version: 4.22.0 is consumable by the OCP 4.22 MCO.

Actual

• Butane version: v0.28.0 outputs ignition.version: 3.6.0 when version 4.22.0 is input. (docs/config-openshift-v4_22.md: "generates Ignition configs with version 3.6.0").
• The OCP 4.22 MCO vendors github.com/coreos/ignition/v2 v2.20.0 and imports only config/v3_5, so its maximum supported Ignition spec is 3.5.0.
• Stable Ignition spec 3.6.0 exists only in the Ignition library from v2.26.0 (2026-02-17); v2.20.0 does not know stable 3.6.0 (it knew only 3.6.0-experimental).
• Result: machine-config-controller render fails with parsing Ignition config failed: unknown version. Supported spec versions: ... → MachineConfigPool RenderDegraded, machine-config ClusterOperator Degraded. The config is never applied, and the degraded pool blocks further MachineConfig changes and cluster upgrades.

Evidence

• Butane 0.28.0 release notes (2026-05-19): "Stabilize OpenShift spec 4.22.0, targeting Ignition spec 3.6.0".
• coreos/butane docs/config-openshift-v4_22.md: "This document is for version 4.22.0 and generates Ignition configs with version 3.6.0." (absent at tag v0.26.0; present from v0.28.0.)
• Ignition release notes: "Ignition 2.26.0 (2026-02-17) ... Mark the 3.6.0 config spec as stable".
• OCP 4.22 MCO source (release-4.22 and main): go.mod pins github.com/coreos/ignition/v2 v2.20.0; pkg/controller/common/helpers.go imports github.com/coreos/ignition/v2/config/v3_5 (no v3_6).
• Confirmed on a live OCP 4.22 cluster: go version -m on the MCO binary reports dep github.com/coreos/ignition/v2 v2.20.0.

$ sudo crictl logs $(sudo crictl ps -a --name machine-config-controller -q | head -1) 2>&1 | grep "error"
F0617 02:13:47.664286       1 bootstrap.go:47] error running MCC[BOOTSTRAP]: parsing Ignition config failed: unknown version. Supported spec versions: 2.2,3.0,3.1,3.2,3.3,3.4,3.5

$ cat /etc/mcc/bootstrap/99-master-chrony-conf-override.yaml
# Generated by Butane; do not edit
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-master-chrony-conf-override
spec:
  config:
    ignition:
      version: 3.6.0
    storage:
      files:
        - contents:
            compression: gzip
            source: ***
          mode: 420
          overwrite: true
          path: /etc/chrony.conf

$ cat /etc/mcc/bootstrap/99-worker-chrony-conf-override.yaml
# Generated by Butane; do not edit
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-chrony-conf-override
spec:
  config:
    ignition:
      version: 3.6.0
    storage:
      files:
        - contents:
            compression: gzip
            source: ***
          mode: 420
          overwrite: true
          path: /etc/chrony.conf

$ REL=$(oc --server=https://192.168.33.249:6443 --insecure-skip-tls-verify=true get clusterversion version -o jsonpath='{.status.desired.image}')
$ echo "release: $REL"
release: quay.io/openshift-release-dev/ocp-release@sha256:283887f2860a745387608d106e70e5be2314df2497ee08c69e7bc669ca091340

$ MCO_IMG=$(oc --server=https://192.168.33.249:6443 --insecure-skip-tls-verify=true adm release info "$REL" --image-for=machine-config-operator)
$ echo "MCO: $MCO_IMG"
MCO: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:45369d15184d85936216245c0d3742815a7cad17efa6a38550da78e6cccf2dbf

$ go version -m ./machine-config-controller | grep coreos/ignition
        dep     github.com/coreos/ignition      v0.35.0
        dep     github.com/coreos/ignition/v2   v2.20.0

Precedent

#312 records the standing policy for this situation:

The MCO doesn't yet support Ignition spec 3.3.0 (xref openshift/machine-config-operator#2675).
If the MCO isn't updated by the time the next 'openshift' spec is stabilized, we need to roll back
the latter to Ignition spec 3.2.0. This was previously done in #269 and #307. Note that it's more
convenient to perform the rollback after stabilizing the spec, since that leaves the new
experimental spec still using the newer Ignition spec version.

Prior rollbacks following that policy:

• openshift/v4_9: roll back to Ignition spec 3.2.0 #269 — rolled 'openshift' 4.9 back from Ignition 3.3.0 to 3.2.0 (merged 2021-08-04), because
  the MCO support PR did not land in time.
• openshift/v4_10: roll back to Ignition spec 3.2.0 #307 — rolled 'openshift' 4.10 back from Ignition 3.3.0 to 3.2.0 (merged 2022-01-21).

The 4.22.0 → 3.6.0 mapping has not been rolled back (no revert after 0.28.0), so the failure
mode #312 warned about is now reachable through the documented, stable workflow.

Requested action

Following the #312 policy and the #269 / #307 precedent: roll back the stable 'openshift 4.22.0'
variant from Ignition spec 3.6.0 to 3.5.0, while leaving the experimental spec on the newer
Ignition version. This realigns the documented 'version: 4.22.0' workflow with the Ignition spec the
OCP 4.22 MCO actually supports.

Reproduction

$ cat > chrony.bu <<'EOF'
variant: openshift
version: 4.22.0
metadata:
  name: 99-worker-chrony-conf-override
  labels:
    machineconfiguration.openshift.io/role: worker
storage:
  files:
    - path: /etc/chrony.conf
      mode: 0644
      overwrite: true
      contents:
        inline: |
          pool 2.rhel.pool.ntp.org iburst
EOF
$ butane chrony.bu        # -> output contains: ignition: {version: 3.6.0}
# Apply the result on an OCP 4.22 cluster:
#   oc apply -f machineconfig.yaml
# -> worker MachineConfigPool becomes RenderDegraded; machine-config ClusterOperator Degraded.

Environment

• Butane: 0.28.0
• OpenShift Container Platform: stable-4.22
• MCO vendored Ignition: 'github.com/coreos/ignition/v2 v2.20.0' (confirmed on cluster)

References

• openshift experimental spec uses an Ignition spec not supported by the MCO #312 (the standing policy / prior occurrence) · openshift/v4_9: roll back to Ignition spec 3.2.0 #269 · openshift/v4_10: roll back to Ignition spec 3.2.0 #307 (prior rollbacks)
• Ignition release notes — 3.6.0 stabilized in Ignition 2.26.0
• MCO-1579: Bump ignition to 3.5 openshift/machine-config-operator#4903 (MCO-1579 — last MCO Ignition bump, to 3.5)
• Ignition 3.3.0 & Ignition Kargs openshift/machine-config-operator#2675 (earlier MCO 3.3.0 support tracking, referenced by openshift experimental spec uses an Ignition spec not supported by the MCO #312)

[github-actions]

Please provide more details about this issue. What steps have you tried so far?