release-26.4: roachtest: perturbation/full/splits/asleep=true failed

[cockroach-teamcity]

roachtest.perturbation/full/splits/asleep=true failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:

Failed with:

(splits.go:243).startPerturbation: query Cockroach node ID: dial tcp 10.91.101.80:26257: connect: connection refused
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=true/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21523842-1789803587-02-n13cpu16sm-0001		10.91.101.150
teamcity-21523842-1789803587-02-n13cpu16sm-0002		10.91.101.161
teamcity-21523842-1789803587-02-n13cpu16sm-0003		10.91.101.149
teamcity-21523842-1789803587-02-n13cpu16sm-0004		10.91.101.138
teamcity-21523842-1789803587-02-n13cpu16sm-0005		10.91.101.151
teamcity-21523842-1789803587-02-n13cpu16sm-0006		10.91.101.152
teamcity-21523842-1789803587-02-n13cpu16sm-0007		10.91.101.148
teamcity-21523842-1789803587-02-n13cpu16sm-0008		10.91.101.168
teamcity-21523842-1789803587-02-n13cpu16sm-0009		10.91.101.169
teamcity-21523842-1789803587-02-n13cpu16sm-0010		10.91.101.139
teamcity-21523842-1789803587-02-n13cpu16sm-0011		10.91.101.74
teamcity-21523842-1789803587-02-n13cpu16sm-0012		10.91.101.79
teamcity-21523842-1789803587-02-n13cpu16sm-0013		10.91.101.80

Parameters:

• acMode=defaultOption
• arch=amd64
• blockSize=4096
• cloud=gce
• concurrency=500
• coverageBuild=false
• cpu=16
• diskBandwidthLimit=0
• diskCount=2
• disks=2
• encrypted=false
• fillDuration=10m0s
• followerReadPercent=50
• leaseType=leader
• mem=standard
• numNodes=12
• numWorkloadNodes=1
• perturbationDuration=10m0s
• ratioOfMax=0.5
• readPercent=50
• runtimeAssertionsBuild=false
• seed=0
• splits=500
• validationDuration=5m0s
• vcpu=16
• workloadConnInit=

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

• release-26.3.2-rc: roachtest: perturbation/full/splits/asleep=true failed #175589 release-26.3.2-rc: roachtest: perturbation/full/splits/asleep=true failed [A-kv C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-26.3.2-rc]
• basalt-26.3: roachtest: perturbation/full/splits/asleep=true failed #174942 basalt-26.3: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-basalt-26.3]
• release-26.3.1-rc: roachtest: perturbation/full/splits/asleep=true failed #173742 release-26.3.1-rc: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3.1-rc]
• release-26.3: roachtest: perturbation/full/splits/asleep=true failed #171839 release-26.3: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3]
• roachtest: perturbation/full/splits/asleep=true failed #171278 roachtest: perturbation/full/splits/asleep=true failed [C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-master]

/cc @cockroachlabs/kv-triage

Improve this report!

Jira issue: CRDB-68549

[miraradeva]

After the connection type issue is fixed, I expect this to fail the same way as #171278, so I'm going to assign it to myself and keep an eye on it.