Race condition bug in the CockroachDB Kubernetes Operator Helm chart (version 25.4.2-preview)

[milanbgd011]

1. Problem Statement

Observed Behavior

When deploying a 2-node CockroachDB cluster using the official cockroachdb-parent Helm chart with the CockroachDB Operator, the two pods initialize as separate, independent clusters instead of joining together.

Evidence

Pod cockroachdb-fgrbw: Node 1 of Cluster A (clusterID: 15152a17-fcc9-4a51-8c04-af0ec0126551)
Pod cockroachdb-nbm5k: Node 1 of Cluster B (clusterID: 2c4d3ec6-47f5-4161-97ff-ef4b12f595f0)

Impact

• No replication: Data written to one pod is NOT visible on the other
• No fault tolerance: If one pod dies, its data is lost
• Intermittent failures: Load balancer routes to different "clusters" causing auth failures
• Data integrity risk: Two independent databases masquerading as one cluster

---

2. Root Cause Analysis

The Race Condition

The CockroachDB Operator creates resources in this order:

1. CrdbCluster CRD is created by Helm
2. Operator begins reconciliation, creates pods
3. Pods start with --join cockroachdb-join.prod.svc.cluster.local:26258
4. PROBLEM: The cockroachdb-join Service doesn't exist yet
5. DNS lookup fails: lookup cockroachdb-join.prod.svc.cluster.local: no such host
6. After ~7 seconds of retries, CockroachDB gives up and initializes a NEW cluster
7. Second pod does the same → TWO separate clusters
8. Operator finally creates cockroachdb-join Service (too late)

Log Evidence from Affected Pod

# Pod startup attempting to join
+ exec /cockroach/cockroach start ... --join cockroachdb-join.prod.svc.cluster.local:26258

# DNS failures (service doesn't exist yet)
W251219 16:32:06.930975 server/init.go:375 outgoing join rpc to cockroachdb-join.prod.svc.cluster.local:26258 unsuccessful:
  rpc error: code = Unavailable desc = "transport: error while dialing: dial tcp: lookup cockroachdb-join.prod.svc.cluster.local on 10.96.0.10:53: no such host"

W251219 16:32:07.944980 server/init.go:421 outgoing join rpc... unsuccessful: no such host

# After ~2 seconds of failures, pod gives up and creates NEW cluster
I251219 16:32:08.847335 server/server.go:1888 connecting to gossip network to verify cluster ID "7b187288-e725-4ce8-a1e2-a0ee115d494e"
status: initialized new cluster
clusterID: 7b187288-e725-4ce8-a1e2-a0ee115d494e
nodeID: 1

Why This Happens

The CockroachDB Operator creates the cockroachdb-join headless Service reactively after processing the CrdbCluster CRD. However, by the time the Service is created, the pods have already started and failed to resolve DNS.

CockroachDB has a short retry window (~7 seconds by default) before it decides "I can't find any peers, I'll initialize myself as a new cluster."

---

3. The Fix

Approach: Pre-create Services Before Pods Start

We add two Helm templates that create the headless Services before the CrdbCluster CRD is processed, using Helm hooks.

File 1: service.join.yaml

# Pre-create the join service before CrdbCluster to prevent race condition
# The operator will take ownership of this service but it needs to exist for DNS
apiVersion: v1
kind: Service
metadata:
  name: {{ template "cockroachdb.fullname" . }}-join
  namespace: {{ .Release.Namespace }}
  labels:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}
    {{- include "cluster.labels" . | nindent 4 }}
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "-10"
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: sql
      port: 26257
      targetPort: sql
    - name: grpc
      port: 26258
      targetPort: grpc
    - name: http
      port: 8080
      targetPort: http
  selector:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}

File 2: service.headless.yaml

# Pre-create the headless service before CrdbCluster
apiVersion: v1
kind: Service
metadata:
  name: {{ template "cockroachdb.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}
    {{- include "cluster.labels" . | nindent 4 }}
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "-10"
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: sql
      port: 26257
      targetPort: sql
    - name: grpc
      port: 26258
      targetPort: grpc
    - name: http
      port: 8080
      targetPort: http
  selector:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}

---

4. Technical Justification

Why Helm Hooks?

Annotation	Purpose
helm.sh/hook: pre-install,pre-upgrade	Ensures Service exists BEFORE main chart resources (CrdbCluster) are created
helm.sh/hook-weight: "-10"	Negative weight = runs early in hook sequence

Why These Specific Service Configurations?

Field	Value	Reason
clusterIP: None	Headless	Required for StatefulSet pod DNS discovery
publishNotReadyAddresses: true	Critical	Allows DNS to return pod IPs even before pods pass readiness checks. This is essential because pods need to discover each other during startup.
selector: crdb.cockroachlabs.com/cluster	Matches Operator	Uses the same label the Operator uses, so the Service will select CockroachDB pods

Why ArgoCD Annotation?

argocd.argoproj.io/compare-options: IgnoreExtraneous

The Operator will eventually "adopt" these Services and add its own annotations/labels. Without IgnoreExtraneous, ArgoCD would constantly show drift and try to revert the Operator's changes.

Why This Doesn't Break Anything

1. Selector matches exactly: The selector crdb.cockroachlabs.com/cluster: cockroachdb is what the Operator uses
2. Ports match exactly: 26257 (SQL), 26258 (gRPC), 8080 (HTTP) are standard CockroachDB ports
3. Operator can still manage: The Operator will update the Service if needed; we just ensure it exists first
4. Idempotent: If Service already exists, Helm hooks handle this gracefully

---

5. Why Modify the Official Chart?

Arguments FOR Modification

1. This is a bug in the chart: The chart should work out-of-the-box for multi-node deployments
2. No upstream fix available: The official 25.4.2-preview chart doesn't address this
3. Minimal invasive change: We only ADD two template files; we don't modify existing ones
4. Standard Kubernetes patterns: Pre-creating Services before StatefulSets is a well-known pattern
5. Documented CockroachDB issue: The "join service resolves to own IP" problem is acknowledged in operator discussions

Arguments AGAINST (and Rebuttals)

Concern	Rebuttal
"Don't modify vendor charts"	This is a critical bug fix, not a customization. The chart is broken without it.
"Upstream may break compatibility"	Our templates use stable APIs (v1 Service) and Helm hooks that have existed for years.
"Operator should handle this"	The Operator creates Services reactively, not proactively. This is a design limitation.
"Use a different approach"	Init containers waiting for DNS still fail if Service doesn't exist at all. Pre-creating Services is the only reliable fix.

---

6. Proof That The Fix Works

Before Fix (Failed Deployment)

$ kubectl exec cockroachdb-fgrbw -- cockroach node status
id  address
1   cockroachdb-fgrbw.cockroachdb.prod.svc.cluster.local:26258  # Only sees itself

$ kubectl exec cockroachdb-nbm5k -- cockroach node status
id  address
1   cockroachdb-nbm5k.cockroachdb.prod.svc.cluster.local:26258  # Only sees itself (DIFFERENT CLUSTER!)

After Fix (Successful Deployment)

$ kubectl exec cockroachdb-rgdlv -- cockroach node status
id  address                                                     locality
1   cockroachdb-rgdlv.cockroachdb.prod.svc.cluster.local:26258  zone=fsn1
2   cockroachdb-sfcnz.cockroachdb.prod.svc.cluster.local:26258  zone=hel1

$ kubectl exec cockroachdb-sfcnz -- cockroach node status
id  address                                                     locality
1   cockroachdb-rgdlv.cockroachdb.prod.svc.cluster.local:26258  zone=fsn1
2   cockroachdb-sfcnz.cockroachdb.prod.svc.cluster.local:26258  zone=hel1

Both nodes see each other. Single cluster ID: 83ef2889-085b-426d-865e-013bbe923741

Replication Test

# Write to node 1 (fsn1)
$ kubectl exec cockroachdb-rgdlv -- cockroach sql -e "INSERT INTO replication_test (value) VALUES ('test-data');"
INSERT 1

# Read from node 2 (hel1) - DIFFERENT physical server
$ kubectl exec cockroachdb-sfcnz -- cockroach sql -e "SELECT * FROM replication_test;"
id                    value      created_at
1134074409334800385   test-data  2025-12-19 16:40:43.859526+00

Data written to fsn1 (Germany) immediately visible on hel1 (Finland). Replication works.

---

7. Files Modified

File	Action	Risk
charts/cockroachdb/templates/service.join.yaml	CREATED	Low - additive only
charts/cockroachdb/templates/service.headless.yaml	CREATED	Low - additive only
values.yaml	MODIFIED	Low - added optional init container

---

8. Alternative Approaches Considered

Option A: Init Container That Waits for DNS

Rejected: If the Service doesn't exist, DNS will never resolve. The init container would wait forever or timeout.

Option B: Increase CockroachDB Join Timeout

Rejected: No documented way to configure this in the Operator. Would require modifying CockroachDB startup flags, which is fragile.

Option C: Deploy Services Separately Before Chart

Rejected: Breaks the "single Helm install" workflow. Requires manual ordering of kubectl commands.

Option D: Pre-create Services in Helm Chart (CHOSEN)

Accepted: Uses standard Helm hooks, minimal code, proven to work, no external dependencies.

---

9. Compatibility Notes

• Helm version: Tested with Helm 3.x (hooks have been stable since Helm 2)
• Kubernetes version: v1 Service API is stable since Kubernetes 1.0
• ArgoCD: Compatible with annotation for ignoring drift
• CockroachDB Operator: Works with 25.4.2-preview (operator will adopt pre-created Services)

---

10. Conclusion

The modifications to the CockroachDB Helm chart are:

1. Necessary: The chart has a race condition bug that prevents proper cluster formation
2. Minimal: Only 2 new template files added, using standard Kubernetes/Helm patterns
3. Safe: Uses stable APIs, doesn't modify existing templates, operator-compatible
4. Proven: Successfully deployed and tested with data replication verified

This fix should be considered for upstream contribution to the official CockroachDB Operator Helm chart.

---

11. Response to Documentation Review

This section addresses concerns raised during independent technical review.

Concern 1: Service Names and Labels Are Not Documented

Status: VALIDATED BY EMPIRICAL EVIDENCE (VERSION-SPECIFIC)

The reviewer correctly noted that cockroachdb-join and crdb.cockroachlabs.com/cluster are not in official documentation. However, live cluster inspection confirms these are the actual values for version 25.4.2-preview:

$ kubectl get svc -n prod -l 'crdb.cockroachlabs.com/cluster=cockroachdb'
NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)
cockroachdb        ClusterIP   None         <none>        26257/TCP,26258/TCP,8080/TCP
cockroachdb-join   ClusterIP   None         <none>        26257/TCP,26258/TCP,8080/TCP

$ kubectl get pods -n prod -o jsonpath='{.items[0].spec.containers[0].command}' | grep join
--join cockroachdb-join.prod.svc.cluster.local:26258

Conclusion: These names/labels are taken from the current 25.4.2-preview chart. They are technically correct but represent implementation details, not a documented contract. If the operator changes them in future versions, these hook templates must be updated accordingly.

Concern 2: Dual Ownership Between Helm and Operator

Status: REASONABLE BUT UNDOCUMENTED

The Operator has adopted the pre-created Services and added ownerReferences:

ownerReferences:
  - apiVersion: crdb.cockroachlabs.com/v1alpha1
    controller: true
    kind: CrdbCluster
    name: cockroachdb

This leverages the same "adopt existing Service" pattern that Cockroach Labs uses in their migration documentation, where existing Services are annotated and labeled for Helm adoption. However, this specific dual-ownership scenario (Helm hooks + Operator reconciliation) is not explicitly documented for the preview operator chart.

The argocd.argoproj.io/compare-options: IgnoreExtraneous annotation prevents GitOps drift detection from conflicting with Operator management.

Concern 3: Preview Status Means Internals Could Change

Status: ACCEPTED WITH MITIGATION

Valid concern. This is a Preview operator with no guarantees about internal behavior stability.

Potential Change	Impact	Mitigation
Service name changes	Duplicate Services created	Easy to detect, update templates
Selector changes	Services don't match pods	Monitor pod selection, unlikely given CRD-based design
Operator fixes the bug	Pre-created Services become no-ops	Harmless redundancy

Given current behavior, the most likely downside is redundant/no-op Services. Because this is a Preview operator, we treat this as a version-specific workaround and will re-validate after every operator upgrade. The fix uses stable Kubernetes v1 APIs.

Concern 4: Not Officially Endorsed

Status: ACKNOWLEDGED

This is a workaround for an undocumented bug in a preview chart. It is:

• Consistent with CockroachDB's documented networking patterns
• Compatible with Cockroach Labs' "adopt existing Services" pattern (per migration docs)
• Pragmatically necessary for production deployments

Recommendation: Submit upstream bug report with logs showing race condition, propose that Operator reconcile Services before creating pods.

---

12. Required Operational Safeguards

This workaround is approved for production use with the following mandatory safeguards:

12.1 Version Tracking

Component	Current Version	Last Validated	Re-validate
CockroachDB Operator Helm Chart	25.4.2-preview	2025-12-19	Service names, labels, reconciliation order
CockroachDB Image	v25.4.2	2025-12-19	cockroach node status CLI behavior

Action Required: Re-validate this workaround after every Operator/Helm chart upgrade.

12.2 Cluster Health Monitoring

Detect "split cluster" failure by verifying all pods see all other pods:

# Detection script - uses documented 'cockroach node status' command
# Each pod should see EXPECTED_NODES nodes; if any sees fewer, cluster may be split
EXPECTED_NODES=2
NAMESPACE=prod

# Get all CockroachDB pod names
PODS=$(kubectl get pods -n $NAMESPACE -l crdb.cockroachlabs.com/cluster=cockroachdb \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

FAILED=0
for POD in $PODS; do
  NODE_COUNT=$(kubectl exec -n $NAMESPACE $POD -c cockroachdb -- \
    cockroach node status --certs-dir=/cockroach/cockroach-certs 2>/dev/null | \
    tail -n +2 | wc -l)

  if [ "$NODE_COUNT" -lt "$EXPECTED_NODES" ]; then
    echo "CRITICAL: Pod $POD only sees $NODE_COUNT nodes (expected $EXPECTED_NODES)"
    FAILED=1
  fi
done

if [ "$FAILED" -eq 1 ]; then
  echo "Cluster integrity check FAILED - possible split cluster"
  exit 1
fi
echo "OK: All pods see $EXPECTED_NODES nodes"

Optional: For absolute verification, perform a write-then-read test across pods:

# Write to pod 1, read from pod 2 - if data appears, same cluster
kubectl exec -n prod cockroachdb-0 -c cockroachdb -- cockroach sql \
  --certs-dir=/cockroach/cockroach-certs \
  -e "CREATE TABLE IF NOT EXISTS health_check (id INT PRIMARY KEY, ts TIMESTAMP DEFAULT now());
      UPSERT INTO health_check (id) VALUES (1);"

kubectl exec -n prod cockroachdb-1 -c cockroachdb -- cockroach sql \
  --certs-dir=/cockroach/cockroach-certs \
  -e "SELECT * FROM health_check WHERE id = 1;"

12.3 Upstream Issue Tracking

TODO: File bug report at https://github.com/cockroachdb/cockroach-operator/issues with:

• Title: "Race condition: pods initialize separate clusters when join service doesn't exist at startup"
• Include: Log evidence showing DNS failures and initialized new cluster messages
• Proposed fix: Operator should create Services before pods, or use init container to wait

---

13. Final Verdict

Classification: Version-specific production workaround

Aspect	Status
Technically correct	✓ Matches actual Operator implementation
Aligned with CockroachDB K8s patterns	✓ Headless Services, correct ports
Uses documented adoption pattern	✓ Same as migration docs
Officially supported	✗ Relies on undocumented internals
Permanent solution	✗ Must re-validate on upgrades

Undocumented Dependencies (Minimum Viable)

Dependency	Used In	Risk	Alternative Exists?
cockroachdb-join service name	Helm hook templates	May change in future Operator versions	❌ No (verified by docs AI)
crdb.cockroachlabs.com/cluster label	Helm hook selectors	May change in future Operator versions	❌ No (verified by docs AI)

Docs AI Verification: Exhaustively searched for documented alternatives. The CrdbCluster CRD does not expose fields to configure service names, service selectors, join addresses, or pre-reconcile hooks. These two dependencies are unavoidable given the current operator design.

Note: Monitoring uses cockroach node status (documented CLI) — zero undocumented functions in monitoring.

Approved for production with the understanding that this is a pragmatic workaround for a Preview operator bug, not a documented configuration. All undocumented dependencies must be re-validated on version upgrades.

---

14. Final Approval Request

Problem

CockroachDB Operator Helm chart (25.4.2-preview) has a race condition where pods start before the cockroachdb-join Service exists, causing nodes to initialize as separate clusters instead of joining.

Chosen Solution

Pre-create headless Services using Helm pre-install,pre-upgrade hooks so DNS resolves before pods start.

Files Added to Helm Chart

charts/cockroachdb/templates/service.join.yaml      # Pre-creates cockroachdb-join Service
charts/cockroachdb/templates/service.headless.yaml  # Pre-creates cockroachdb Service

Documented Patterns Used

Pattern	Source
Headless Services with clusterIP: None	Local K8s deployment docs
Ports 26257, 26258, 8080	Network ports docs
publishNotReadyAddresses: true	Standard K8s pattern for StatefulSet discovery
Adopt existing Services	Migration docs step 5

Undocumented Dependencies Accepted

Dependency	Justification	Docs AI Verified
cockroachdb-join service name	No CRD field to configure service names	✓ No alternative exists
crdb.cockroachlabs.com/cluster label	No CRD field to configure Service selectors	✓ No alternative exists

Docs AI Confirmation: Searched for alternatives including CRD fields, startFlags, alternative discovery mechanisms, pre-reconcile hooks, and using cockroachdb-public for join. None are documented. These two dependencies represent the minimum unavoidable reliance on implementation details.

Monitoring uses documented CLI (cockroach node status) — no undocumented SQL functions required.

Safeguards Committed

1. ✓ Version tracking with explicit re-validation requirements
2. ✓ Monitoring scripts for split-cluster detection
3. ✓ Node count verification
4. ✓ Upstream bug report to be filed

Classification

Version-specific production workaround — not a permanent solution, not officially supported, must be re-validated on every Operator/CockroachDB upgrade.

Request

Approve this as the optimal available solution for a Preview operator race condition, given:

• No documented alternative exists
• Fix uses documented K8s/CockroachDB patterns where possible
• All undocumented dependencies are explicitly tracked
• Safeguards are in place to detect regression

---

Appendix: Quick Reference Commands

# Verify cluster formation
kubectl exec -n prod <pod-name> -c cockroachdb -- cockroach node status --certs-dir=/cockroach/cockroach-certs

# Check for DNS resolution issues in logs
kubectl logs -n prod <pod-name> -c cockroachdb | grep -i "join\|no such host\|initialized"

# Verify services exist before pods
kubectl get svc -n prod | grep cockroach

# Test replication
kubectl exec -n prod <pod1> -- cockroach sql --url='...' -e "INSERT INTO test (v) VALUES ('x');"
kubectl exec -n prod <pod2> -- cockroach sql --url='...' -e "SELECT * FROM test;"

[milanbgd011]

Solution was to boot 1 server first, then when up to scale to 2 servers and then the joining happens.

Booting both instances of the DB simultaionusly corrupts this behaviour.

[milanbgd011]

Here is the explanation from the DOCS AI

==========

What you’ve implemented lines up with how CockroachDB and the new operator are designed to work, and it matches the documented “scale up” pattern — you’re just making it explicit instead of letting everything start at once.

Why “1 node first, then scale” fixes it

From CockroachDB’s point of view:

• Nodes discover each other via the --join flag and DNS/hostnames. [[cockroach start – networking](https://www.cockroachlabs.com/docs/stable/cockroach-start#networking); [Start a multi-node cluster](https://www.cockroachlabs.com/docs/stable/cockroach-start#start-a-multi-node-cluster)]
• When nodes start simultaneously and DNS/service endpoints aren’t yet fully propagated or routable, each node’s join attempts can fail long enough that they never form a healthy cluster together.
• By starting one node first, you guarantee:
  • Its pod is running.
  • Its service endpoints are resolvable and routable.
  • The operator has completed the initial cluster initialization step. [[Deploy with operator – initialize](https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-with-cockroachdb-operator#initialize-the-cluster)]

Then, when you scale to 2 nodes, the second pod comes up and:

• Uses the same --join configuration.
• Can actually reach the first node’s address/service.
• Successfully joins the existing cluster instead of racing another “first node”.

How this maps to the documented operator model

In the operator’s values, node count is controlled per region:

cockroachdb:
  crdbCluster:
    regions:
      - code: eu-central
        nodes: 3
        cloudProvider: gcp   # or "" for bare metal
        namespace: cockroach-ns

You’re doing exactly what the docs describe for scaling, but in two phases:

1. Phase 1: Install with 1 node

   This is equivalent to deploying with:

   cockroachdb:
     crdbCluster:
       regions:
         - code: eu-central
           nodes: 1
           namespace: prod

   Install/upgrade:

   helm upgrade --install cockroachdb ./charts/cockroachdb \
     --namespace prod \
     --values values.yaml \
     --set cockroachdb.crdbCluster.regions[0].nodes=1

   And wait until the first pod is Ready, as you’re doing with kubectl wait. Once that’s true, the operator has initialized the cluster. [[Deploy with operator – initialize](https://www.cockroachlabs.com/docs/stable/deploy-cockroachdb-with-cockroachdb-operator#initialize-the-cluster)]

2. Phase 2: Scale to 2 nodes

   The documented way to scale with the operator is to increase regions[].nodes and apply the change. [[Scale with operator – add nodes](https://www.cockroachlabs.com/docs/stable/scale-cockroachdb-operator#add-nodes)]

   In your case you’re doing that via:

   kubectl patch crdbcluster cockroachdb -n prod \
     --type=merge \
     -p '{"spec":{"regions":[{"code":"eu-central","nodes":2,"cloudProvider":"","namespace":"prod"}]}}'

   Which is just another way of performing the same change described in the docs (bumping nodes from 1 → 2). The operator reconciles this and creates the second pod.

   Then you wait for:

   • Two pods to exist.
   • Both pods to be Ready (via kubectl wait).
   • cockroach node status from one pod to show both nodes in the cluster.

This is exactly the documented scaling lifecycle: start small, then increase nodes and let the operator add pods one by one. [[Scale with operator – add nodes](https://www.cockroachlabs.com/docs/stable/scale-cockroachdb-operator#add-nodes)]

Why this is a safe pattern

From the docs’ perspective, your approach is “just”:

• Using the standard regions[].nodes setting to control cluster size.
• Using the standard scale‑up path for adding nodes.
• Adding orchestration around it (Helm + Ansible) to:
  • Ensure services are created.
  • Ensure the first node is fully up.
  • Then scale.

You’re not depending on unsupported CRD fields; you’re orchestrating documented behaviors in a more conservative order to avoid a startup race between pods + DNS. That makes this pattern robust across operator versions, as long as:

• cockroachdb.crdbCluster.regions[*].nodes remains the scaling knob.
• The operator continues to add pods when nodes increases (which is exactly what the scaling docs promise). [[Scale with operator – add nodes](https://www.cockroachlabs.com/docs/stable/scale-cockroachdb-operator#add-nodes)]

In short: the idea is solid, aligns with the official scaling model, and simply serializes initialization (1 node first) before letting the operator do its normal “add node” flow.

[milanbgd011]

So I guess this "boot one" and then boot others must be included by default in the operator preview chart.

And then it will work.

[milanbgd011]

To be precise I deployed this in kubernetes that has 2 machines on different locations but are in same local private network.
So they are different machines on which the deployment is being done on, maybe this is confusing the operator or there is some undocumented thing I need to do

[milanbgd011]

Final solution (what needs to be customized on top of Operator Preview)

CockroachDB Operator Helm Chart Customization Guide

This document explains all modifications made to the official CockroachDB Operator Helm chart to fix a race condition bug. Follow this guide when upgrading or recreating from a clean chart.

---

Problem Being Solved

When deploying a multi-node CockroachDB cluster, pods start simultaneously and try to join via DNS lookup of cockroachdb-join service. But the Operator creates this service after pods start, causing DNS failures. Each pod gives up after ~7 seconds and initializes as a separate cluster.

Result: 2 pods, 2 separate clusters, no replication, data loss risk.

---

Solution Overview

Two-part fix:

1. Helm Chart: Pre-create headless Services before pods start
2. Ansible Playbook: Deploy with 1 node first, then scale to 2 (serializes initialization)

---

Part 1: Helm Chart Modifications

Base Chart Location

/Users/mmmm/aa/_kubernetes/domain.com/prod/cockroachdb-master/
├── charts/
│   └── cockroachdb/
│       └── templates/
│           ├── service.join.yaml      # ADD THIS
│           ├── service.headless.yaml  # ADD THIS
│           └── ... (existing files)
└── values.yaml

File 1: charts/cockroachdb/templates/service.join.yaml

Purpose: Pre-create the join service so DNS resolves before pods start.

# Pre-create the join service before CrdbCluster to prevent race condition
# The operator will take ownership of this service but it needs to exist for DNS
# See: https://github.com/cockroachdb/cockroach-operator/issues - race condition when
# pods start before the join service exists
apiVersion: v1
kind: Service
metadata:
  name: {{ template "cockroachdb.fullname" . }}-join
  namespace: {{ .Release.Namespace }}
  labels:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}
    {{- include "cluster.labels" . | nindent 4 }}
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-wave: "-5"
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: sql
      port: 26257
      targetPort: sql
    - name: grpc
      port: 26258
      targetPort: grpc
    - name: http
      port: 8080
      targetPort: http
  selector:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}

File 2: charts/cockroachdb/templates/service.headless.yaml

Purpose: Pre-create the main headless service for pod-to-pod discovery.

# Pre-create the headless service before CrdbCluster to prevent race condition
# The operator will take ownership of this service but it needs to exist for DNS
apiVersion: v1
kind: Service
metadata:
  name: {{ template "cockroachdb.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}
    {{- include "cluster.labels" . | nindent 4 }}
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
    argocd.argoproj.io/sync-wave: "-5"
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: sql
      port: 26257
      targetPort: sql
    - name: grpc
      port: 26258
      targetPort: grpc
    - name: http
      port: 8080
      targetPort: http
  selector:
    crdb.cockroachlabs.com/cluster: {{ template "cockroachdb.fullname" . }}

Why These Specific Values?

Field	Value	Reason
clusterIP: None	Headless service	Required for StatefulSet DNS discovery
publishNotReadyAddresses: true	Critical	Returns pod IPs before readiness checks pass
crdb.cockroachlabs.com/cluster	Label & selector	Matches what Operator uses (undocumented but required)
argocd.argoproj.io/compare-options: IgnoreExtraneous	Annotation	Prevents ArgoCD drift when Operator modifies the Service
argocd.argoproj.io/sync-wave: "-5"	Annotation	Creates Services before CrdbCluster CRD

---

Part 2: Ansible Playbook - "1 Node First, Then Scale"

Even with pre-created Services, simultaneous pod startup can still race. The definitive fix is to serialize initialization.

File: ___worker-cockroachdb-operator-setup.yml

Key sections of the playbook:

Step 1: Pre-create Services via kubectl (backup to Helm templates)

- name: Pre-create cockroachdb-join headless service
  shell: |
    cat <<'EOF' | kubectl apply -f -
    apiVersion: v1
    kind: Service
    metadata:
      name: cockroachdb-join
      namespace: {{ cockroachdb_namespace }}
      labels:
        crdb.cockroachlabs.com/cluster: cockroachdb
    spec:
      type: ClusterIP
      clusterIP: None
      publishNotReadyAddresses: true
      ports:
        - name: sql
          port: 26257
          targetPort: sql
        - name: grpc
          port: 26258
          targetPort: grpc
        - name: http
          port: 8080
          targetPort: http
      selector:
        crdb.cockroachlabs.com/cluster: cockroachdb
    EOF

- name: Pre-create cockroachdb headless service
  shell: |
    cat <<'EOF' | kubectl apply -f -
    apiVersion: v1
    kind: Service
    metadata:
      name: cockroachdb
      namespace: {{ cockroachdb_namespace }}
      labels:
        crdb.cockroachlabs.com/cluster: cockroachdb
    spec:
      type: ClusterIP
      clusterIP: None
      publishNotReadyAddresses: true
      ports:
        - name: sql
          port: 26257
          targetPort: sql
        - name: grpc
          port: 26258
          targetPort: grpc
        - name: http
          port: 8080
          targetPort: http
      selector:
        crdb.cockroachlabs.com/cluster: cockroachdb
    EOF

Step 2: Deploy with 1 Node First

- name: Install CockroachDB with 1 node first
  shell: |
    cd {{ cockroachdb_chart_remote_path }}
    helm dependency update ./charts/cockroachdb 2>/dev/null || true
    helm upgrade --install {{ cockroachdb_release_name }} ./charts/cockroachdb \
      --namespace {{ cockroachdb_namespace }} \
      --values values.yaml \
      --set cockroachdb.crdbCluster.regions[0].nodes=1 \
      --timeout 10m

Why: Forces only 1 pod to start. It initializes a cluster alone (no join needed).

Step 3: Wait for First Node

- name: Wait for first CockroachDB pod to be ready
  shell: |
    kubectl wait --for=condition=Ready pod \
      -l crdb.cockroachlabs.com/cluster={{ cockroachdb_release_name }} \
      -n {{ cockroachdb_namespace }} \
      --timeout=300s
  retries: 3
  delay: 30
  until: first_pod_ready.rc == 0

- name: Verify first node is initialized
  shell: |
    POD=$(kubectl get pods -n {{ cockroachdb_namespace }} -l crdb.cockroachlabs.com/cluster={{ cockroachdb_release_name }} -o jsonpath='{.items[0].metadata.name}')
    kubectl exec -n {{ cockroachdb_namespace }} $POD -c cockroachdb -- \
      cockroach node status --certs-dir=/cockroach/cockroach-certs | tail -n +2 | wc -l
  register: node_count
  until: node_count.stdout | int >= 1
  retries: 10
  delay: 5

Why: Ensures first node is fully operational and endpoints are registered in DNS.

Step 4: Scale to 2 Nodes

- name: Scale CockroachDB to 2 nodes
  shell: |
    kubectl patch crdbcluster {{ cockroachdb_release_name }} -n {{ cockroachdb_namespace }} \
      --type=merge \
      -p '{"spec":{"regions":[{"code":"eu-central","nodes":2,"cloudProvider":"","namespace":"{{ cockroachdb_namespace }}"}]}}'

- name: Wait for second pod to be created
  shell: kubectl get pods -l crdb.cockroachlabs.com/cluster={{ cockroachdb_release_name }} -n {{ cockroachdb_namespace }} --no-headers | wc -l
  register: pod_count
  until: pod_count.stdout | int >= 2
  retries: 30
  delay: 10

- name: Wait for all CockroachDB pods to be ready
  shell: |
    kubectl wait --for=condition=Ready pod \
      -l crdb.cockroachlabs.com/cluster={{ cockroachdb_release_name }} \
      -n {{ cockroachdb_namespace }} \
      --timeout=300s
  retries: 3
  delay: 30
  until: pods_ready.rc == 0

Why: Second pod now finds first pod via DNS and joins its cluster instead of creating a new one. Uses documented spec.regions[].nodes CRD field.

---

Part 3: Database Initialization

File: ___worker-cockroachdb-db-init.yml

Run after cluster is healthy to create app user and enable password auth.

---
- name: Initialize CockroachDB Database
  hosts: cloud_control_plane[0]
  gather_facts: false
  vars:
    cockroachdb_namespace: prod
    cockroachdb_cluster_name: cockroachdb
    cockroachdb_app_user: app
    cockroachdb_app_password: "password"

  tasks:
    - name: Wait for CockroachDB cluster to be healthy
      shell: |
        kubectl exec -n {{ cockroachdb_namespace }} \
          $(kubectl get pods -n {{ cockroachdb_namespace }} -l crdb.cockroachlabs.com/cluster={{ cockroachdb_cluster_name }} -o jsonpath='{.items[0].metadata.name}') \
          -c cockroachdb -- \
          cockroach node status --certs-dir=/cockroach/cockroach-certs 2>/dev/null | tail -n +2 | wc -l
      register: node_count
      until: node_count.stdout | int >= 2
      retries: 30
      delay: 10

    - name: Create app user and configure authentication
      shell: |
        kubectl exec -n {{ cockroachdb_namespace }} \
          $(kubectl get pods -n {{ cockroachdb_namespace }} -l crdb.cockroachlabs.com/cluster={{ cockroachdb_cluster_name }} -o jsonpath='{.items[0].metadata.name}') \
          -c cockroachdb -- \
          cockroach sql --certs-dir=/cockroach/cockroach-certs --host=localhost:26257 -e "
            CREATE USER IF NOT EXISTS {{ cockroachdb_app_user }} WITH PASSWORD '{{ cockroachdb_app_password }}';
            ALTER USER {{ cockroachdb_app_user }} WITH LOGIN;
            GRANT ALL ON DATABASE defaultdb TO {{ cockroachdb_app_user }};
            SET CLUSTER SETTING server.host_based_authentication.configuration = 'host all all all password';
          "

---

Deployment Order

cd ~/aa/_kubernetes/domain.com/prod

# 1. Install CockroachDB Operator CRDs
ansible-playbook -i ../inventory-cluster.yml ../../___worker-cockroachdb-operator-install.yml

# 2. Deploy CockroachDB (1 node first, then scale to 2)
ansible-playbook -i ../inventory-cluster.yml ../../___worker-cockroachdb-operator-setup.yml

# 3. Initialize database (create app user, enable password auth)
ansible-playbook -i ../inventory-cluster.yml ../../___worker-cockroachdb-db-init.yml

---

Verification Commands

# Both nodes should see each other (same cluster)
kubectl exec -n prod <any-pod> -c cockroachdb -- \
  cockroach node status --certs-dir=/cockroach/cockroach-certs

# Expected output: 2 nodes, both is_live: true

# Test replication (write to node 1, read from node 2)
POD1=$(kubectl get pods -n prod -l crdb.cockroachlabs.com/cluster=cockroachdb -o jsonpath='{.items[0].metadata.name}')
POD2=$(kubectl get pods -n prod -l crdb.cockroachlabs.com/cluster=cockroachdb -o jsonpath='{.items[1].metadata.name}')

kubectl exec -n prod $POD1 -c cockroachdb -- cockroach sql --certs-dir=/cockroach/cockroach-certs \
  -e "CREATE TABLE IF NOT EXISTS test_rep (id SERIAL PRIMARY KEY, v TEXT); INSERT INTO test_rep (v) VALUES ('hello');"

kubectl exec -n prod $POD2 -c cockroachdb -- cockroach sql --certs-dir=/cockroach/cockroach-certs \
  -e "SELECT * FROM test_rep;"
# Should return the row inserted on POD1

---

Undocumented Dependencies (Cannot Be Avoided)

Dependency	Where Used	Why Required
cockroachdb-join service name	Service templates, ansible	Operator hardcodes this; no CRD field to configure
crdb.cockroachlabs.com/cluster label	Service selectors	Operator uses this label; no CRD field to configure

These were verified by docs AI as having no documented alternatives. Must re-validate on Operator upgrades.

---

When Upgrading the Operator Chart

1. Copy new chart to cockroachdb-master/charts/cockroachdb/
2. Re-add the two template files (service.join.yaml, service.headless.yaml)
3. Verify service name is still cockroachdb-join in new version
4. Verify label is still crdb.cockroachlabs.com/cluster
5. Test deployment on staging first

---

Files Summary

File	Type	Purpose
charts/cockroachdb/templates/service.join.yaml	Helm template	Pre-create join service
charts/cockroachdb/templates/service.headless.yaml	Helm template	Pre-create headless service
___worker-cockroachdb-operator-setup.yml	Ansible	Deploy with "1 node first, then scale"
___worker-cockroachdb-db-init.yml	Ansible	Create app user, enable password auth
COCKROACHDB-HELM-CHART-FIX-JUSTIFICATION.md	Docs	Full technical justification
COCKROACHDB-OPERATOR-CUSTOM.md	Docs	This file - recreation guide