basalt-26.3: roachtest: import/nodeShutdown/worker/distmerge=false/nodes=4 failed

[cockroach-teamcity]

roachtest.import/nodeShutdown/worker/distmerge=false/nodes=4 failed with artifacts on basalt-26.3 @ 4fda4910b2015319b2f775e42d2f2fb15130d19a:

Failed with:

(test_runner.go:2762).func1: task ``task-2`` returned error: import tpch/customer: could not restart node :3: failed to wait for SQL to be ready: dial tcp 10.91.100.67:26257: connect: connection refused
test artifacts and logs in: /artifacts/import/nodeShutdown/worker/distmerge=false/nodes=4/run_1

Fatal entries found in Cockroach logs:

F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18  while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18  while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 18 !	pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119  while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304  while ingesting auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested: create auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested: grpc: file "i34.t6.ingested" already exists [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !	pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !	pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !	pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763  [T1,Vsystem,n3,s3,r102/2:/Table/106/1/1{17242…-31250…},raft] 304 !	pkg/util/log/channels.go:104 +0x650
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119  while ingesting ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: create ‹auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested›: grpc: ‹file "i34.t6.ingested" already exists› [code 6/AlreadyExists]
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !goroutine 1150 [running]:
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0x36f77ad9f018?, 0xd6afcb?})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/allstacks/allstacks.go:38 +0x125
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/allstacks/allstacks.go:19
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x36f77d770cc0, 0x24}, {0x82309ac, 0x1}, {0x82309aa, 0x1}, {0x8137de9, 0x6}, {0x82309ac, ...}}, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/log/clog.go:294 +0xc6
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x86df4c8, 0x36f7821bc960}, 0x2, 0x4, 0xf, 0x0?, {0x8124688, 0x2}, {0x36f77ad9f898, 0x1, ...})
F260913 08:48:51.909229 1150 15@kv/kvserver/replica_proposal.go:763 ⋮ [T1,Vsystem,n3,s3,r102/2:/Table/106/1/‹1›{‹17242›…-‹31250›…},raft] 119 !	pkg/util/log/channels.go:104 +0x650

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21514658-1789279845-98-n4cpu4-0001		10.91.100.68
teamcity-21514658-1789279845-98-n4cpu4-0002		10.91.99.214
teamcity-21514658-1789279845-98-n4cpu4-0003		10.91.100.67
teamcity-21514658-1789279845-98-n4cpu4-0004		10.91.99.211

Parameters:

• MetamorphicBasaltMultiAZ=false
• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=true
• fs=ext4
• localSSD=true
• metamorphicLeases=default
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=false
• storageNodesVersion=v0.1.19-nightly.20260912

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Logs were not uploaded to Datadog

/cc @cockroachlabs/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-68684

[iskettaneh]

/investigate

[github-actions]

Investigation: import/nodeShutdown/worker/distmerge=false/nodes=4

Investigated failure: issue #175405 body (TeamCity build 21514658)
Failure SHA: 4fda4910b2015319b2f775e42d2f2fb15130d19a
Confidence: high (the node crash is directly observed; the reported error is a downstream symptom)

What This Test Does

import/nodeShutdown/worker runs a TPC-H customer IMPORT and, while the
import job is in flight, hard-kills one node (the "shutdown target", node 3
here) with SIGKILL, then restarts it and verifies the cluster recovers and the
import completes. It exercises node-restart / crash-recovery paths under load.

Where the Failure Occurs

The reported error is a downstream symptom, not the root cause:

task `task-2` returned error: import tpch/customer: could not restart node :3:
failed to wait for SQL to be ready: dial tcp 10.91.100.67:26257: connect: connection refused

Node 3 (10.91.100.67) refuses SQL connections because it crashed a few
milliseconds into startup with a fatal error while replaying its raft log:

F260913 08:48:51.914601 1150 15@kv/kvserver/replica_proposal.go:763 [T1,Vsystem,n3,s3,r102/2:/Table/106/1/...,raft] 18
  while ingesting auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested:
  create auxiliary/sideloading/r0XXXX/r102/i34.t6.ingested:
  grpc: file "i34.t6.ingested" already exists [code 6/AlreadyExists]

The fatal is raised in addSSTablePreApply
(replica_proposal.go:763),
which treats any error from the sideloaded-SST ingest as fatal. The
call chain from the stack trace confirms this happens during raft command
application on the restarting node:

kvserver.addSSTablePreApply
kvserver.(*appBatch).runPostAddTriggers        app_batch.go:200
kvserver.(*replicaAppBatch).Stage              replica_app_batch.go:144
kvserver/apply.(*Task).applyOneBatch           apply/cmd.go / task.go

Analysis

Primary hypothesis (high confidence): sideloaded-SST ingest is not
idempotent across an unclean restart, so raft-log replay re-ingests an SST
that the storage engine already has, and the ingest path fatals on
AlreadyExists.

Timeline (all node-3, from test.log and logs/3.cockroach.log):

• 08:46:43 — test SIGKILLs node 3 (jobs_util.go:144: stopping node (using SIGKILL) :3).
• 08:48:48 — test restarts node 3 (cockroach.go:1044: starting node 3).
• 08:48:51.823 — node 3 process comes up, begins raft replay.
• 08:48:51.909 — fatal re-applying r102 raft entry index 34 / term 6
  (i34.t6), ingesting a sideloaded SST whose target file already exists.
• 08:48:53 — test observes SQL never becomes ready → "connection refused".

The filename i34.t6 encodes raft index 34, term 6. The mechanism is the
classic replay-idempotency gap: node 3 applied this entry (creating the ingest
file object in the storage engine) before the SIGKILL, but the durable raft
applied-index did not advance past it, so on restart the same entry is applied
again. addSSTablePreApply hard-links the sideloaded file and calls the
engine's ingest, and the engine rejects the create because a file with that
name already exists — surfaced over gRPC as [code 6/AlreadyExists]. Because
the ingest helper Fatalfs on any error, the node dies instead of treating a
duplicate ingest during replay as a no-op / recoverable condition.

The grpc: ... [code 6/AlreadyExists] shape indicates the create is rejected
by the storage engine used on this build configuration (note the separate
storageNodesVersion parameter on the issue) rather
than by a plain local filesystem, where a duplicate hard-link/ingest during
replay would not surface this way. This points at an engine-level ingest
idempotency gap on this branch's storage stack, not at the SQL IMPORT logic.

Severity / routing. This is a real crash-recovery correctness bug (a node
cannot restart cleanly after a mid-write kill), consistent with the
release-blocker label — not a test flake. The test is only the messenger.
Although the issue is auto-assigned to T-sql-foundations, the failure is in the
KV raft-apply / storage-engine ingest path; it likely belongs with the
KV/storage owners of addSSTablePreApply and the sideloaded-SST ingest engine
integration.

Suggested next steps for the owning team:

• Make sideloaded-SST ingest idempotent on replay: when the ingest target
  already exists in the engine (AlreadyExists), treat it as an
  already-applied no-op rather than fatal, or check/skip before re-ingesting.
• Confirm whether the raft applied-index and the engine ingest are made
  durable atomically for this storage engine; if the engine acks the create
  durably but the applied-index lags, replay will always re-hit this file.
• Consider whether addSSTablePreApply's unconditional Fatalf should
  distinguish AlreadyExists (recoverable during replay) from genuinely fatal
  ingest failures.

Related Issues and PRs

• Prior occurrences of this exact test are the usual import/nodeShutdown flakes
  on branch-master / release-26.2 (e.g. #166071,
  #160959,
  #165258) — those
  are older and, based on their teams/branches, do not share this
  AlreadyExists-on-ingest fatal signature. This appears to be a new,
  distinct failure mode specific to this branch's storage configuration.
• A search for the AlreadyExists / "while ingesting ... already exists" fatal
  signature returned no matching open or closed issues, suggesting this is not
  yet separately tracked.

Timeline

• Fatal call site addSSTablePreApply / Fatalf("while ingesting ...") is
  long-standing (the ingest-error-is-fatal behavior predates this branch). What
  changed is the engine now returning AlreadyExists for a duplicate
  create during replay; recent storage-engine changes on this branch touch the
  AddSSTable ingest path (e.g. c4f58255c0d kvserver,storage: excise range flush deletions on AddSSTable ingest and related commits in the log for
  pkg/kv/kvserver/replica_proposal.go), which is where I'd start bisecting.

Detailed evidence and investigation notes

Log Excerpts

Node 3 restart → crash (logs/3.cockroach.log):

08:48:51.823739 server/node.go:808  [n3] started with attributes []
08:48:51.849203 replica_proposal_buf.go:627 [n3,s3,r5/2] campaigning because Raft leader not live
08:48:51.909176 replica_proposal.go:763 [n3,s3,r102/2] the server is terminating due to a fatal error
08:48:51.914601 F replica_proposal.go:763 [n3,s3,r102/2] while ingesting <...i34.t6.ingested>:
    create <...i34.t6.ingested>: grpc: file "i34.t6.ingested" already exists [code 6/AlreadyExists]

Fatal stack (raft apply path, from unredacted logs/3.unredacted/cockroach-kv-exec.log and 3.cockroach.log):

kvserver.addSSTablePreApply                pkg/kv/kvserver/replica_proposal.go:763
kvserver.(*appBatch).runPostAddTriggers    pkg/kv/kvserver/app_batch.go:200
kvserver.(*replicaAppBatch).Stage          pkg/kv/kvserver/replica_app_batch.go:144
kvserver/apply.mapCmdIter                  pkg/kv/kvserver/apply/cmd.go:187
kvserver/apply.(*Task).applyOneBatch

Test driver (test.log):

08:45:33 jobs_util.go:81: test has chosen shutdown target node 3, and watcher node 4
08:46:43 jobs_util.go:144: stopping node (using SIGKILL) :3
08:48:39 jobs_util.go:163: test status: restarting :3 (node restart test is done)
08:48:48 cockroach.go:1044: starting node 3
08:48:53 test_impl.go:486: test failure #1 ... could not restart node :3 ... connection refused

Code References

• Fatal ingest handling: replica_proposal.go:763
  — addSSTablePreApply hard-links the sideloaded file then ingests it; any
  ingest error is Fatalf'd.
• Test restart logic: pkg/cmd/roachtest/tests/jobs_util.go (executeNodeShutdown, lines ~163-170)
  and pkg/cmd/roachtest/tests/import.go (lines ~511, 772-782).

Things Ruled Out

• Not a networking / provisioning flake. The "connection refused" is fully
  explained: node 3's process exited on the fatal, so nothing was listening on
  26257. Other nodes stayed healthy.
• Not the same as prior import/nodeShutdown flakes. Earlier issues for this
  test don't carry the ingest-AlreadyExists fatal signature.
• Not SQL/IMPORT-layer logic. The crash is in KV raft apply during log
  replay, independent of the import job's SQL execution.

Other Failure Occurrences

Only one failure occurrence is present in this issue thread (the issue body).
No additional failure comments to compare against.

Tooling Feedback

• gh search issues --state all is rejected (only open/closed accepted);
  had to run searches without a state filter. Minor — worth noting in the
  prompt so future runs don't waste a call.
• No blockers otherwise; TeamCity guest artifact download and log extraction
  worked on the first try.

---

Investigation workflow run: https://github.com/cockroachdb/cockroach/actions/runs/34765431722

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

---

Generated by claude-opus-4-8 at high effort. Use /investigate fable-5 max to go deeper.

Workflow run · Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[iskettaneh]

This is fixed by https://github.com/cockroachdb/basalt/pull/1151