roachtest: replicagc-changed-peers/restart=false failed

[cockroach-teamcity]

roachtest.replicagc-changed-peers/restart=false failed with artifacts on master @ 0ff013855fdee74d72243c42da24c01a62767848:

Failed with:

(replicagc.go:337).waitForZeroReplicasOnN3: pq: could not determine data type of placeholder $1
test artifacts and logs in: /artifacts/replicagc-changed-peers/restart=false/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21524325-1789883542-48-n6cpu4-0001	40.76.98.245	10.1.0.136
teamcity-21524325-1789883542-48-n6cpu4-0002	20.115.96.6	10.1.0.138
teamcity-21524325-1789883542-48-n6cpu4-0003	20.127.170.69	10.1.0.139
teamcity-21524325-1789883542-48-n6cpu4-0004	172.178.82.30	10.1.0.135
teamcity-21524325-1789883542-48-n6cpu4-0005	74.235.161.232	10.1.0.14
teamcity-21524325-1789883542-48-n6cpu4-0006	20.62.193.234	10.1.0.131

Parameters:

• arch=amd64
• cloud=azure
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicDRPC=true
• metamorphicLeases=default
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

• release-26.3: roachtest: replicagc-changed-peers/restart=false failed #171810 release-26.3: roachtest: replicagc-changed-peers/restart=false failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3]

/cc @cockroachlabs/kv-triage

Improve this report!

Jira issue: CRDB-68571

[cockroach-teamcity]

roachtest.replicagc-changed-peers/restart=false failed with artifacts on master @ b7bc1edd57b69c14b874d525f5fb8d330142d93c:

Failed with:

(replicagc.go:337).waitForZeroReplicasOnN3: pq: could not determine data type of placeholder $1
test artifacts and logs in: /artifacts/replicagc-changed-peers/restart=false/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21525026-1789969975-59-n6cpu4-0001	172.208.91.200	10.1.0.174
teamcity-21525026-1789969975-59-n6cpu4-0002	20.106.172.58	10.1.0.175
teamcity-21525026-1789969975-59-n6cpu4-0003	20.102.99.76	10.1.0.176
teamcity-21525026-1789969975-59-n6cpu4-0004	172.206.228.208	10.1.0.173
teamcity-21525026-1789969975-59-n6cpu4-0005	74.235.166.231	10.1.0.172
teamcity-21525026-1789969975-59-n6cpu4-0006	20.127.158.171	10.1.0.177

Parameters:

• arch=amd64
• cloud=azure
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=expiration
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

• release-26.4: roachtest: replicagc-changed-peers/restart=false failed #175688 release-26.4: roachtest: replicagc-changed-peers/restart=false failed [C-test-failure O-roachtest O-robot T-kv branch-release-26.4 release-blocker]
• release-26.3: roachtest: replicagc-changed-peers/restart=false failed #171810 release-26.3: roachtest: replicagc-changed-peers/restart=false failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3]

Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.replicagc-changed-peers/restart=false failed with artifacts on master @ b7bc1edd57b69c14b874d525f5fb8d330142d93c:

Failed with:

(replicagc.go:337).waitForZeroReplicasOnN3: pq: could not determine data type of placeholder $1
test artifacts and logs in: /artifacts/replicagc-changed-peers/restart=false/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21525028-1789970127-80-n6cpu4-0001		10.91.98.189
teamcity-21525028-1789970127-80-n6cpu4-0002		10.91.98.160
teamcity-21525028-1789970127-80-n6cpu4-0003		10.91.99.54
teamcity-21525028-1789970127-80-n6cpu4-0004		10.91.100.251
teamcity-21525028-1789970127-80-n6cpu4-0005		10.91.101.5
teamcity-21525028-1789970127-80-n6cpu4-0006		10.91.98.161

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• metamorphicLeases=epoch
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

• release-26.4: roachtest: replicagc-changed-peers/restart=false failed #175688 release-26.4: roachtest: replicagc-changed-peers/restart=false failed [C-test-failure O-roachtest O-robot T-kv branch-release-26.4 release-blocker]
• release-26.3: roachtest: replicagc-changed-peers/restart=false failed #171810 release-26.3: roachtest: replicagc-changed-peers/restart=false failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3]

Improve this report!

[miraradeva]

/investigate

[github-actions]

Investigation: replicagc-changed-peers/restart=false

Investigated failure: issuecomment-5758181804
Failure SHA: b7bc1edd57b69c14b874d525f5fb8d330142d93c
Confidence: high (likely cause — test regression)

What This Test Does

replicagc-changed-peers sets up a 6-node cluster, decommissions the
original three nodes (one in absentia), and verifies that a downed node
quickly replica-GCs its now-stale replicas after it rejoins. As part of
the wait, waitForZeroReplicasOnN3 polls crdb_internal.ranges_no_leases
for any ranges that still list n3's store.

Where the Failure Occurs

(replicagc.go:337).waitForZeroReplicasOnN3: pq: could not determine data type of placeholder $1

The error originates from the query at
replicagc.go:309-310:

const q = `select range_id, replicas from crdb_internal.ranges_no_leases where replicas @> ARRAY[$1];`
rows, err := db.QueryContext(ctx, q, storeID)

storeID is a roachpb.StoreID (int32), passed as placeholder $1
inside the ARRAY[$1] constructor. The db.QueryContext call fails at
statement preparation with could not determine data type of placeholder $1,
which is then surfaced through the retry loop and fatals at
replicagc.go:337.

Analysis

Likely cause (high confidence): a test regression introduced by commit
bb9708a1577 ("roachtest: translate physical nodes to Cockroach
identities", merged ~Sep 16 2026).

That commit rewrote the query from a hardcoded literal to a parameterized
placeholder:

- const q = `... where replicas @> ARRAY[3];`
- rows, err := db.QueryContext(ctx, q)
+ const q = `... where replicas @> ARRAY[$1];`
+ rows, err := db.QueryContext(ctx, q, storeID)

A placeholder inside an ARRAY[...] constructor has no type context that
CockroachDB's type-checker can use to infer $1's type. With the old
literal ARRAY[3], the constant defaulted to INT; with a bare
placeholder, the element type is indeterminate, so preparation fails with
could not determine data type of placeholder $1. The type of replicas
on the left of @> is not propagated down through the array constructor
to the placeholder.

This is deterministic, not flaky — every run since the commit landed
hits the same error at the same line. That matches the observed behavior:
the same signature appears on the original report (0ff013855fd) and on
both runbot builds at b7bc1edd57b (Azure and GCE, with different
metamorphic lease/DRPC settings), and independently on release-26.4
(#175688). This explains why the issue is flagged release-blocker.

Suggested fix: give the placeholder an explicit type so preparation
can resolve it, e.g.:

... where replicas @> ARRAY[$1::INT8];
-- or
... where replicas @> ARRAY[$1]::INT8[];

(The replicas column of crdb_internal.ranges_no_leases is INT[].)
Casting the storeID at the Go call site does not help — the driver still
sends a bare $1 whose type the server must resolve from the statement text.

Related Issues and PRs

• release-26.4: roachtest: replicagc-changed-peers/restart=false failed #175688 (release-26.4) — same failure mode. Identical error
  signature (could not determine data type of placeholder $1 at
  replicagc.go:337). The offending commit bb9708a1577 is present on
  release-26.4, so this is the same root cause on that branch.
• release-26.3: roachtest: replicagc-changed-peers/restart=false failed #171810 (release-26.3) — different failure mode, grouped only by
  test name. That failure is
  (replicagc.go:96).runReplicaGCChangedPeers: COMMAND_PROBLEM: exit status 1
  on 769f38f9d34 — a decommission command failure in older code that
  predates the placeholder change. Not the same root cause; should not be
  conflated with this one.

Timeline

• Commit bb9708a1577 ("roachtest: translate physical nodes to Cockroach
  identities", author Ludovic Leroux, ~Sep 16 2026) rewrote the query to
  use ARRAY[$1].
• First reported failure: 0ff013855fd (issue body).
• Most recent failures: two builds at b7bc1edd57b (Sep 21 2026), the
  investigated SHA.
• git log -L on lines 309-310 confirms the placeholder was introduced by
  bb9708a1577; the immediately prior revisions used the literal
  ARRAY[3].

Detailed evidence and investigation notes

Code References

• Query & call site:
  replicagc.go:308-331
• Fatal on error:
  replicagc.go:336-338
• cockroachStoreID returns roachpb.StoreID:
  node_identities.go:17-31

git blame (lines 309-310)

bb9708a1577 roachtest: translate physical nodes to Cockroach identities
-  const q = `... where replicas @> ARRAY[3];`
-  rows, err := db.QueryContext(ctx, q)
+  const q = `... where replicas @> ARRAY[$1];`
+  rows, err := db.QueryContext(ctx, q, storeID)

Things Ruled Out

• Infra flake / metamorphic-setting sensitivity: ruled out. The two
  b7bc1edd57b builds used different metamorphic settings
  (metamorphicLeases=expiration vs epoch, DRPC on/off) yet failed
  identically at the same line. The error is a SQL type-check failure at
  statement preparation, independent of cluster state or timing.
• Genuine replica-GC hang: ruled out. The failure is not the
  t.Fatalf("replica count ... didn't drop to zero") path; it is the
  query itself erroring before any replica count is evaluated.
• release-26.3 release-26.3: roachtest: replicagc-changed-peers/restart=false failed #171810 being the same bug: ruled out — different error
  (COMMAND_PROBLEM at line 96) on code predating the placeholder change.

Note on artifacts

Artifacts were not downloaded: the failure is deterministic and fully
explained by the query text and the error message, and the same signature
reproduces across three builds on two branches, all of which carry the
offending commit. Downloading artifacts.zip would only re-show the same
could not determine data type of placeholder $1 error.

Tooling Feedback

No blocked tools materially impeded this investigation. gh search prs --state all rejected the value all (only open/closed accepted) — a
minor CLI-usage note, not a permission issue.

---

Investigation workflow run:
https://github.com/cockroachdb/cockroach/actions/runs/35607791023

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

---

Generated by claude-opus-4-8 at high effort. Use /investigate fable-5 max to go deeper.

Workflow run · Was this investigation helpful? Leave a 👍 or 👎 on this comment.