Skip to content

review(substrate): close 2 MEDIUM concurrency gaps + trap-clean tempfiles (follow-up to #200)#202

Merged
jakebromberg merged 1 commit into
mainfrom
worktree-review-200-followup
May 30, 2026
Merged

review(substrate): close 2 MEDIUM concurrency gaps + trap-clean tempfiles (follow-up to #200)#202
jakebromberg merged 1 commit into
mainfrom
worktree-review-200-followup

Conversation

@jakebromberg

Copy link
Copy Markdown
Owner

Post-merge review of #200 caught two race-condition gaps in the substrate's S3 concurrency story plus a tempfile-cleanup hole. This change closes all three with regression tests.

MEDIUM — pipeline/refresh-index.mjs retry reused stale serialized body

Brief §3.4: "each writer lists by-repo/<R>/ to compute the latest, then PUTs index.json with conditional If-Match on its ETag, retrying on 412 conflict." The previous implementation listed once at script start, serialized once, then on 412 only re-read the ETag — never re-listed. Result: byte-level CAS but not logical CAS. Writer A's eventual successful retry would carry A's view of the world, silently overwriting whatever writer B added between A's list and A's PUT.

Fix: extract the list / enumerate / serialize pipeline into a recompute() function. The first attempt uses initial = recompute(); conflict retries call recompute() again so the loser's eventual successful PUT carries the winner's view, not the loser's stale serialization.

MEDIUM — pipeline/refresh-index.mjs first-publish race had no protection

When headETag returned null (no existing index.json), putBytes omitted any precondition. Two concurrent first-publishers both succeeded; one silently lost.

Fix: pass --if-none-match '*' whenever ifMatch is null, so even the create-vs-create race triggers a 412 and routes through the same retry path.

LOW — pipeline/publish-catalog.sh tempfile cleanup not exception-safe

Three /tmp/publish-catalog-*.$$.* tempfiles were only removed on the happy path. Under set -eu, any failure mid-pipeline (validation, upload, jq) bypassed the explicit rm -f calls.

Fix: add a trap '...rm -f ... EXIT' at the top of the script. Explicit rm -f calls remain as happy-path belt-and-suspenders.

Test plan

  • Extractor: 97 / 97
  • Canonical: 26 / 26
  • Integration: 124 / 124
  • gojq parity: 54 / 54
  • Substrate: 58 / 58 (was 56; added byte-stability check on recompute() and a tempfile-leak regression check on publish-catalog.sh)
  • shellcheck clean on all four substrate shell scripts

Refs #152, #153, #200.

…iles (follow-up to #200)

Post-merge review of #200 surfaced two race-condition gaps in the substrate's S3 concurrency story and a tempfile-cleanup hole in publish-catalog.sh. This change closes all three.

**MEDIUM** — `pipeline/refresh-index.mjs` ETag-CAS retry was reusing a stale serialized body. Brief §3.4 prescribes "each writer lists by-repo/<R>/ to compute the latest, then PUTs index.json with conditional If-Match on its ETag, retrying on 412 conflict" — re-listing per attempt. The previous implementation listed once at script start, serialized once, then on 412 only re-read the ETag and retried with the same body. That gave PUT-level CAS (writer A's bytes won't clobber B's if B PUT between A's list and A's PUT) but NOT logical CAS — A's eventual successful retry would still carry A's view of the world, silently overwriting whatever B added. Fix: extract the listing + enumeration + serialization into a `recompute()` function and call it at the top of each retry attempt. The first attempt uses `initial = recompute()` from the regular execution path; conflict retries call `recompute()` again to incorporate the winning writer's additions before the next PUT.

**MEDIUM** — `pipeline/refresh-index.mjs` first-publish race had no protection. When `headETag` returns null (no existing index.json), `putBytes` previously omitted any precondition — two concurrent first-publishers would both succeed, one silently losing. Fix: pass `--if-none-match '*'` whenever `ifMatch` is null, so even the create-vs-create race triggers a 412 and routes through the same retry path.

**LOW** — `pipeline/publish-catalog.sh` tempfile cleanup was not exception-safe. The script creates `/tmp/publish-catalog-*.$$.*` tempfiles for validation status, per-catalog metadata, and the latest.json staging buffer. Under `set -eu`, any failure (validation, upload, jq) caused an early exit that bypassed the explicit `rm -f` calls. Fix: add a `trap '...rm -f /tmp/publish-catalog-*.$$.*...' EXIT` at the top of the script so cleanup runs unconditionally. The explicit `rm -f` calls remain as belt-and-suspenders on the happy path.

Verified: 97 extractor tests + 26 canonical + 124 integration + 54 gojq-parity + 58 substrate = 359 tests pass (was 357; +2 for byte-stability check on `recompute()` and a tempfile-leak regression test on `publish-catalog.sh`). shellcheck clean on all four substrate shell scripts.

Refs #152, #153, #200.
@jakebromberg
jakebromberg merged commit 389889b into main May 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant