how to apply different labels and annotations to pods for multi-batch rollout strategy?

[ahrtolia]

Question

I would like to implement a multi-batch rollout strategy using Kruise Rollouts, and I want to apply different labels and annotations* to the Pods of each batch, so that we can distinguish the groups.

Is there a native way to configure labels and annotations per batch?

In other rollout implementations (e.g. some proposals or Argo extensions), there is a field like patchPodTemplateMetadata in each step. I tried using it but got validation errors: unknown field "patchPodTemplateMetadata".

Could you please let me know if there is a supported way to achieve this in current versions (e.g. v0.6.0)? Or is this feature planned in future releases?

[furykerry]

there is a built-in pod labeling capability in Kruise Rollout to identify pods related to each batch using the following two labels:

• rollouts.kruise.io/rollout-id
• rollouts.kruise.io/rollout-batch-id

you can see the document:
https://openkruise.io/rollouts/user-manuals/basic-usage/#how-to-examine-the-newly-deployed-pods

[ahrtolia]

there is a built-in pod labeling capability in Kruise Rollout to identify pods related to each batch using the following two labels:

• rollouts.kruise.io/rollout-id
• rollouts.kruise.io/rollout-batch-id

you can see the document: https://openkruise.io/rollouts/user-manuals/basic-usage/#how-to-examine-the-newly-deployed-pods

Thank you for your reply, but I couldn't find the corresponding label or annotation, Could you please tell me how to add batch information in the annotation of POD

[root@qa-web3-nginx-kube test]# cat cloneset-gray-demo.yaml 
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  name: cloneset-gray-demo
  namespace: kruise-test
spec:
  replicas: 4
  selector:
    matchLabels:
      app: cloneset-gray-demo
  updateStrategy:
    type: InPlaceIfPossible
    partition: 0
  template:
    metadata:
      labels:
        app: cloneset-gray-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.23

[root@qa-web3-nginx-kube test]# cat rollout-demo.yaml 
apiVersion: rollouts.kruise.io/v1beta1
kind: Rollout
metadata:
  name: rollouts-demo
  namespace: kruise-test
  annotations:
    rollouts.kruise.io/rolling-style: partition
spec:
  strategy:
    canary:
      enableExtraWorkloadForCanary: false
      steps:
      - replicas: 1
      - replicas: 50%
      - replicas: 100%
  workloadRef:
    apiVersion: apps.kruise.io/v1alpha1
    kind: CloneSet
    name: cloneset-gray-demo

[root@qa-web3-nginx-kube test]# kubectl get po -n kruise-test
NAME                       READY   STATUS    RESTARTS       AGE
cloneset-gray-demo-rf48g   1/1     Running   5 (8d ago)     8d
cloneset-gray-demo-rvg9w   1/1     Running   5 (8d ago)     8d
cloneset-gray-demo-tkd5v   1/1     Running   5 (8d ago)     8d
cloneset-gray-demo-zxrnw   1/1     Running   7 (148m ago)   8d

[root@qa-web3-nginx-kube test]# kubectl get po cloneset-gray-demo-zxrnw -n kruise-test -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    apps.kruise.io/inplace-update-state: '{"revision":"cloneset-gray-demo-858ddf8ff8","updateTimestamp":"2025-07-11T03:49:59Z","lastContainerStatuses":{"nginx":{"imageID":"docker.io/library/nginx@sha256:a484819eb60211f5299034ac80f6a681b06f89e65866ce91f356ed7c72af059c"}},"updateImages":true,"containerBatchesRecord":[{"timestamp":"2025-07-11T03:49:59Z","containers":["nginx"]}]}'
    apps.kruise.io/runtime-containers-meta: '{"containers":[{"name":"nginx","containerID":"containerd://85f7a7a6861557567acb50e109d08daca0d031b77f166f93ba23bbd9e5bfa162","restartCount":7,"hashes":{"plainHash":4290831339,"plainHashWithoutResources":0,"extractedEnvFromMetadataHash":86995377}}]}'
    lifecycle.apps.kruise.io/timestamp: "2025-07-11T03:50:02Z"
  creationTimestamp: "2025-07-02T08:35:03Z"
  generateName: cloneset-gray-demo-
  labels:
    app: cloneset-gray-demo
    apps.kruise.io/cloneset-instance-id: zxrnw
    controller-revision-hash: cloneset-gray-demo-858ddf8ff8
    lifecycle.apps.kruise.io/state: Normal
    pod-template-hash: 858ddf8ff8
  name: cloneset-gray-demo-zxrnw
  namespace: kruise-test
  ownerReferences:
  - apiVersion: apps.kruise.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: CloneSet
    name: cloneset-gray-demo
    uid: 4610c95e-fd62-4096-b450-199df92e9894
  resourceVersion: "197964692"
  uid: bb88c78e-deea-412d-96f6-6949696cd693
spec:
  containers:
  - image: nginx:1.23
    imagePullPolicy: Always
    name: nginx
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-6nkwx
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-4-212-21.ap-east-1.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  readinessGates:
  - conditionType: InPlaceUpdateReady
  - conditionType: KruisePodReady
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-6nkwx
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-07-02T08:35:03Z"
    status: "True"
    type: KruisePodReady
  - lastProbeTime: null
    lastTransitionTime: "2025-07-11T03:50:02Z"
    status: "True"
    type: InPlaceUpdateReady
  - lastProbeTime: null
    lastTransitionTime: "2025-07-02T08:35:13Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-07-02T08:35:03Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-07-11T03:50:04Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-07-02T08:35:13Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-07-02T08:35:03Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://85f7a7a6861557567acb50e109d08daca0d031b77f166f93ba23bbd9e5bfa162
    image: docker.io/library/nginx:1.23
    imageID: docker.io/library/nginx@sha256:f5747a42e3adcb3168049d63278d7251d91185bb5111d2563d58729a5c9179b0
    lastState:
      terminated:
        containerID: containerd://b0ae7f0a4fcdb3d5fce794980b4e6b61001c6163f504b1db897e3f676e309dd6
        exitCode: 0
        finishedAt: "2025-07-11T03:49:59Z"
        reason: Completed
        startedAt: "2025-07-11T03:05:48Z"
    name: nginx
    ready: true
    restartCount: 7
    started: true
    state:
      running:
        startedAt: "2025-07-11T03:50:01Z"
  hostIP: 10.4.212.21
  hostIPs:
  - ip: 10.4.212.21
  phase: Running
  podIP: 10.4.207.40
  podIPs:
  - ip: 10.4.207.40
  qosClass: BestEffort
  startTime: "2025-07-02T08:35:03Z"`

[root@qa-web3-nginx-kube test]# kubectl kruise describe rollout rollouts-demo -n kruise-test
Name:              rollouts-demo
Namespace:         kruise-test
Status:            ✔ Healthy
Message:           Rollout progressing has been cancelled
Strategy:          Canary
 Step:             1/3
 Steps:
  -  Replicas:     1
     State:        StepPaused
  -  Replicas:     50%
  -  Replicas:     100%
Images:            nginx:1.23
Current Revision:  858ddf8ff8
Update Revision:   858ddf8ff8
Replicas:
 Desired:          4
 Updated:          4
 Current:          4
 Ready:            4
 Available:        4

[furykerry]

you need add the label rollouts.kruise.io/rollout-id in your workload metadata, rollout-id autogeneration is not release yet.

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  name: cloneset-gray-demo
  namespace: kruise-test
  labels:
     rollouts.kruise.io/rollout-id: order-1 
spec:
  replicas: 4
  selector:
    matchLabels:
      app: cloneset-gray-demo
  updateStrategy:
    type: InPlaceIfPossible
    partition: 0
  template:
    metadata:
      labels:
        app: cloneset-gray-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.23

[ahrtolia]

you need add the label rollouts.kruise.io/rollout-id in your workload metadata, rollout-id autogeneration is not release yet.

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  name: cloneset-gray-demo
  namespace: kruise-test
  labels:
     rollouts.kruise.io/rollout-id: order-1 
spec:
  replicas: 4
  selector:
    matchLabels:
      app: cloneset-gray-demo
  updateStrategy:
    type: InPlaceIfPossible
    partition: 0
  template:
    metadata:
      labels:
        app: cloneset-gray-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.23

Thank you for your answer. I understand. But what I ultimately want to achieve is to fill in the batch of the POD in the environment variable. I hope to write the batch into the annotation, and then the environment variable references the annotation to get the current batch. Is this possible?

[furykerry]

the batch-id is patched by the rollout-controller during the rollout process, not during the pod creation. So it is not possible to use downward api with environment, which is statically set during pod creation, but you can access the annotation using downward api with volume, which can be dynamically updated after pod creation.

what is the point of set batch-id as the environment variable, why does your application need access the batch-id ?

[ahrtolia]

the batch-id is patched by the rollout-controller during the rollout process, not during the pod creation. So it is not possible to use downward api with environment, which is statically set during pod creation, but you can access the annotation using downward api with volume, which can be dynamically updated after pod creation.

what is the point of set batch-id as the environment variable, why does your application need access the batch-id ?

我如果想实现这个需求，只能在POD升级时，将批次Patch到annotation中对吧
我们的RPC访问和日志查看，需要根据POD中的环境变量来区分，RPC有一个自己实现的灰度策略，是通过环境变量来实现的，所以希望在环境变量中放入当前POD所属的批次

[furykerry]

the batch-id label is patched automatically by kruise rollouts. However if your app depends on the batch-id to influence RPC it is hard to use the inplace-update of cloneset, since the app may read the batch-id from previous rollout-id. If the gray strategy of RPC only affects the first few batches, a better solution is to use the canary strategy instead of the multi-batch strategy. In canary strategy, kruise rollout will create a canary deployment first and you can use patchPodTemplateMetadata to change the podtemplate of the canary deployment. Of course kruise rollout can also be improved to better support canary deployment

1. support canary strategy for cloneset (currently canary deployment only support deployment)
2. support the combination of canary strategy and multi-batch strategy, so that you can use the canary step for patchPodTemplateMetadata, and use multi-batch steps for regular rollout.

[furykerry]

maybe we can have a discussion in next community call on 7.24

[ahrtolia]

Kruise Rollout 会自动为 batch-id 标签打补丁。但是，如果你的应用依赖于 batch-id 来影响 RPC，则很难使用 cloneset 的 inplace-update 功能，因为应用可能会从之前的 rollout-id 中读取 batch-id。如果 RPC 的灰度策略只影响前几个 batch，那么更好的解决方案是使用金丝雀策略而不是多 batch 策略。在金丝雀策略中，Kruise Rollout 会先创建一个金丝雀部署，你可以使用 patchPodTemplateMetadata 来修改金丝雀部署的 podtemplate。当然，Kruise Rollout 也可以改进，以更好地支持金丝雀部署。

1. 支持 cloneset 的金丝雀策略（目前金丝雀部署仅支持部署）
2. 支持金丝雀策略和多批次策略的组合，这样您就可以使用金丝雀步骤进行 patchPodTemplateMetadata，并使用多批次步骤进行定期发布。

I know，thanks for your reply