importer: job restart resets checkpointed resume positions, so a second interruption re-imports completed files

[iskettaneh]

NOTE: This is identified and filed by claude, so take it with a grain of salt

Describe the problem

When an IMPORT job is restarted, whether by PAUSE/RESUME, a coordinator restart,
or re-adoption after the coordinator's SQL liveness session expires, the first
progress checkpoint of the new attempt overwrites the job's persisted per-file
resume positions with zeros. The running attempt is unaffected, since its
processors were planned from the record as it stood before the overwrite, but
the persisted checkpoint now claims that files completed in earlier attempts
were never started. A second interruption therefore plans from that record and
imports those files again from offset 0, and when interruptions recur faster
than a full pass the job never converges.

This is the same defect class as #173109, which fixed the declarative schema
changer's merge checkpoint: the in-memory accumulator is seeded empty instead of
from the durable checkpoint, and the durable checkpoint is replaced rather than
merged.

The mechanism, as of master @ 8812064 (identical on release-26.3), has three
parts:

1. Planning seeds each processor's ResumePos from the job record
   (import_processor_planning.go#L706)
   and keeps files whose position is math.MaxInt64 in the spec. The reader
   skips those files
   (read_import_base.go#L107-L110)
   and marks them complete again only once every file of the processor is done
   (read_import_base.go#L140).
2. The coordinator's importCheckpointTracker is constructed with zeroed
   rowProgress and fractionProgress
   (import_progress_tracker.go#L58-L66)
   and is never seeded from the persisted ResumePos/ReadProgress.
   RecordProcessorUpdate overwrites each reported file unconditionally
   (#L95),
   and Persist copies the tracker wholesale over prog.ResumePos and
   prog.ReadProgress and recomputes FractionCompleted
   (#L199-L201).
   The progress ticker calls Persist 10 s after the attempt starts
   (import_processor_planning.go#L354-L364).
3. Each processor's importProgressTracker likewise starts every file in
   spec.Uri at zero without consulting spec.ResumePos
   (import_processor.go#L599,
   #L610),
   and formatProgress reports every file in the spec
   (#L633),
   so files it never reads, which are exactly the completed ones, are reported
   as position 0 and fraction 0 on every push. This is what makes the reset
   durable for the rest of the attempt rather than a 10 s blip.

To Reproduce

Derived from the code and the roachtest run below; not yet reduced to a unit
test.

1. IMPORT a table from several files, with more than one file per processor.
2. After at least one file has completed, interrupt the job (PAUSE JOB then
   RESUME JOB, or restart the coordinator) and let the new attempt run for more
   than 10 s.
3. Observe that the SHOW JOB fraction drops to roughly 0 and that
   ImportProgress.ResumePos for the completed files reads 0 instead of
   math.MaxInt64.
4. Interrupt again and observe the new attempt re-reading the completed files
   from offset 0.

TestCSVImportCanBeResumed and TestImportHonorsResumePosition each interrupt
exactly once, so neither catches the regression; extending one of them to a
second interruption with an assertion on ResumePos should fail today.

Expected behavior

The persisted checkpoint is monotonic across attempts: a completed file stays
marked complete, an in-progress file never regresses below its last flushed
position, and a restart re-reads at most the unflushed tail of in-progress
files.

Additional data / screenshots

Observed in roachtest import/pause/distmerge=false/nodes=4 (#174633, run of
2026-09-08) importing tpch/lineitem on 4 nodes with 4 vCPUs each, where cluster
overload made the coordinator's SQL liveness session expire repeatedly and the
jobs registry re-adopted the job seven times in 64 minutes.
system.job_progress_history for the job:

time (UTC)	fraction	event
09:04:04	0.323	last checkpoint of attempt 1
09:04:04		coordinator re-adopts the job under a new session
09:04:16	0.0008	first ticker Persist of attempt 2
09:06:39	0.164	attempt 2's processors report in-progress files; completed files stay at 0
09:07:01		another node adopts the job (attempt 3)
09:07:11	0.0008	first Persist of attempt 3, planned from the reset record
10:13:51	0.0025	after five more re-adoptions

Each of the four PAUSE/RESUME cycles earlier in the same run also dropped the
fraction to 0 ten seconds after the resume (07:35:00, 07:36:01, 07:36:47), which
is the single-interruption, cosmetic form of the same bug.

Environment:

• CockroachDB v26.3 development build; the code on master is unchanged. The
  defect predates c5fc818 (2026-03-11, "importer: extract
  importCheckpointTracker from distImport"), whose parent zero-initialized the
  same slice and copied it over ResumePos the same way.

Additional context

Correctness is not affected, as IMPORT writes at a fixed timestamp and
re-ingesting a file produces identical KVs; the cost is wall-clock time, and
non-convergence when interruptions are frequent. A likely secondary effect,
not verified, is inflated row counts in the job summary, since planning only
resets the accumulated BulkOpSummary when every file reads position 0.

A fix would seed importCheckpointTracker from the persisted ResumePos and
ReadProgress at construction; seed the processor's importProgressTracker
from spec.ResumePos, reporting position math.MaxInt64 and fraction 1.0 for
completed files or omitting unread files from formatProgress; and make
RecordProcessorUpdate and Persist never lower a position, as a backstop
against reordered reports.

Jira issue: CRDB-68134

Epic CRDB-68473

[blathers-crl]

Hi @iskettaneh, please add a branch-* label to identify the earliest affected branch for this C-bug

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Please add a branch-* label to identify the earliest affected branch for this C-bug.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}