feat(substrate): cross-repo R2 catalog landing zone + fetch tooling#199
Merged
Conversation
Lays down the substrate that #118's cross-repo queries (#156/#157/#158) and the per-repo CI publication step (#154) consume. Four scripts plus a hermetic mock so the whole surface is exercisable without an actual R2/S3 bucket. `pipeline/fetch-catalogs.sh` is the auditor read path — pulls `index.json`, then every ok-repo's per-catalog SHA-keyed objects, into `$AUDIT_LOCAL_CACHE`. Warm-cache via sha256 compare; supports `--repos` filtering and `file://` URLs for tests. `pipeline/publish-catalog.sh` is the CI write path — validates each catalog against the v1.1 wrapper, uploads under `by-repo/<R>/<iso-utc>_<short-sha>/<kind>.json`, writes a `latest.json` pointer, and triggers a self-healing index rebuild. `pipeline/refresh-index.mjs` derives `index.json` from a bucket listing. Idempotent — running it twice produces byte-identical output; running it after manual edits or partial publishes restores correctness without losing data. The "rewrite from listing, not append-on-write" pattern the brief calls out (§7.2). Per-repo `latest.catalogs[]` array reflects the multi-catalog reality (type-catalog + function-catalog + file-hashes + package-graph), not the single-catalog assumption in the original brief. `pipeline/verify-index.sh` is the drift detector. Reports orphan bucket prefixes and dangling index references. Run by `refresh-index` callers and a nightly cron. All four ship with a `--bucket-fs DIR` local-filesystem backend for tests and dev — same code path, no S3 SDK required. Production uses `--bucket-name`/`--bucket-endpoint` (S3-API, sigv4 via the `aws` CLI). For Cloudflare R2 the only change is the endpoint URL. Schema reconciles the brief against current main: the catalog substrate now publishes a *directory* of catalogs per repo (the README caveat in `docs/plans/README.md` flagged this — original brief assumed one catalog file). `latest.catalogs[]` is the per-catalog manifest; per-row `extractor` provenance comes from the v1.1 wrapper that lands in PR #181/#190/#196. Tests: 45 hermetic cases under `pipeline/_tests/test_substrate.sh`, wired into CI's `pipeline + typescript extractor` job. Covers cold/warm fetch, repo filter, malformed-index handling, sha256 mismatch + cleanup, refresh idempotence, `--known-repos` missing-repo surfacing, stale-snapshot detection, `--dry-run` no-op, drift detection on orphan and dangling cases, publish round-trip via fetch, bare-array refusal, `--skip-refresh`. `docs/substrate.md` covers one-time R2 provisioning, IAM/token scoping with OIDC, env-var contract, local auditor workflow, recovery procedures, and the cost model (well under $1/month at 30-repo scale on R2). Explicitly out of scope and tracked elsewhere: real R2 bucket provisioning (operational, needs the user's Cloudflare account); live integration tests against a real bucket (per brief §6.2 — follow-up); per-repo CI workflow that calls `publish-catalog.sh` on push (#154); `coverage.jq` query against the substrate's coverage block (#155); F1/F2/F3 cross-repo queries (#156/#157/#158, all blocked on this + #152). Closes #153.
5 tasks
jakebromberg
added a commit
that referenced
this pull request
May 30, 2026
…M + LOW polish) Independent reviews of PR #196 (imports kind) and PR #199 (R2 substrate) surfaced two real bugs in the substrate's S3 path plus several MEDIUM-severity correctness issues. This change closes them all and adds regression tests for each. **HIGH** — `pipeline/refresh-index.mjs:147,154`: `require('node:fs').unlinkSync(tmp)` inside an `.mjs` ESM module throws `ReferenceError`, silently swallowed by the surrounding `try{}catch{}`. Every S3 GET / PUT leaked a `/tmp/refresh-index-*` file in production. Hermetic tests didn't exercise the S3 path so the bug never surfaced. Fix: hoist `unlinkSync` into the top-level `import { ... } from 'node:fs'`. **HIGH** — `pipeline/refresh-index.mjs` S3 `putBytes`: the file's own header (line 35) declares `If-Match on the current ETag; retry on 412`, and brief §2.5 / §3.4 / §7.2 prescribe ETag-CAS. The implementation was a bare `s3api put-object` with no preconditions and no retry — two concurrent publishes would silently lose one writer's view. Fix: add `headETag()` to the S3 backend; `putBytes({ifMatch})` passes `--if-match` and translates 412 to a tagged `__precondition` error; the write loop reads the ETag, attempts the PUT, and on precondition failure retries with capped exponential backoff (up to 5 attempts). **MEDIUM** — `pipeline/refresh-index.mjs`: when a newer SHA-keyed prefix exists in the bucket but `latest.json` still points at the older one (interrupted publish, partial restore, manual upload), the emitted index record mixed `prefix` and `short_sha` from the bucket-derived newest with `commit_sha` and `published_at` read from the stale pointer. The mismatched `published_at` also poisoned the `status` calculation. Fix: trust `latest.json`'s metadata only when its `prefix` field actually matches the chosen `latestPrefix`. Otherwise log a WARN, derive `short_sha` + `published_at` from the prefix directory name, and emit `commit_sha: null`. New `refresh-index.mjs: latest.json points at older prefix than bucket newest (drift)` test exercises this. **MEDIUM** — `pipeline/publish-catalog.sh`: validation and upload were interleaved in a single `while ... | read` loop. If catalog A validated and uploaded, then catalog B failed validation, the script exited 1 leaving A as an orphan under the SHA-keyed prefix with no `latest.json` ever written — and the next `refresh-index` run would treat the orphan as the canonical "latest." Fix: split into a validate-all-first pass (record failure to a tempfile so the subshell `exit` doesn't get swallowed) then an upload-all-after pass; the bucket stays untouched if any catalog fails validation. New `publish-catalog.sh: mixed-validity batch uploads NOTHING (no orphan SHA prefix)` test exercises this. **MEDIUM** — `pipeline/fetch-catalogs.sh`: the `distinct_repos` counter was incremented for every repo named in the work list (i.e., every ok-status repo in the index) regardless of whether any catalog was actually present in the cache afterwards. An index with 3 ok repos where all GETs failed would exit 0 with an empty cache and `Summary: 3 repos / 0 fetched / 0 cached / 3 failed`. Downstream `jq -s 'map(.entries) | add'` then silently produced empty output. Fix: rename the counter to `present_repos` and only `echo` to its tempfile after a successful cache hit or fetch. New `fetch-catalogs.sh: exits 1 when every fetch fails` test exercises this. **LOW polish** rolled into the same change: - `extractors/typescript/type-catalog.mjs:524`: `const { a: { b } } = require('pkg')` previously emitted zero rows (every element was `continue`d as a nested binding, then the explicit `return` short-circuited the bare-call fallback) so the consumer edge to the package was completely lost. Now falls through to the namespace-form `pushRequire({ name: '*', imported_as: null })` so the package edge survives. Fixture `12-nested-require.ts` + matching assertion added. - `extractors/typescript/type-catalog.mjs:467`: `import {} from "pkg"` (empty named-imports clause — semantically a side-effect import) now emits a side-effect row instead of zero rows. Fixture `11-empty-named.ts` + matching assertion added. - `extractors/typescript/type-catalog.mjs:404-409`: corrected the misleading "interleave with declaration rows by line" comment to describe actual ordering (top-level imports first in source-line order; declarations + dynamic-import + require rows after, in AST-walk order). Both halves remain deterministic, which is what the contract depends on. - `pipeline/refresh-index.mjs`: `--bucket-fs DIR` now preflight-checks the directory exists with a friendly error message instead of crashing with a raw `ENOENT mkdir` stack inside `putBytes`. - `pipeline/refresh-index.mjs:196`: dropped the hardcoded `wxyc/` org prefix in the canonical-name fallback. When `latest.json` is missing or unparseable, `canonicalRepo` now falls back to `pathSegment` alone — keeps the substrate org-agnostic. Verified: 97 extractor tests + 26 canonical + 124 integration + 54 gojq-parity + 56 substrate = 357 tests pass. shellcheck on the four substrate shell scripts is clean. Refs #152, #153.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pipeline/:fetch-catalogs.sh(read),publish-catalog.sh(write),refresh-index.mjs(self-healing index rebuild),verify-index.sh(drift detector). All four support a--bucket-fs DIRlocal-filesystem backend for hermetic tests, plus an S3-API backend (R2 / AWS S3) for production.pipeline/_tests/test_substrate.sh, wired into the existing CI job. No network. No R2/S3 dependency.docs/substrate.mdoperator guide: provisioning, IAM/OIDC, env-var contract, recovery procedures, cost model.Bucket layout
Schema reconciliation
The brief assumed one catalog file per repo. Today's substrate has 4 catalog kinds (type-catalog + function-catalog + file-hashes + package-graph), so each push publishes a directory of them, and
index.json's per-repolatest.catalogs[]array reflects the manifest. Documented in plan-153 and the operator guide.Test plan
bash pipeline/_tests/test_substrate.sh— 45 / 45 passbash pipeline/queries/_tests/test_canonical.sh— 26 / 26 (no regression)bash pipeline/queries/_tests/test_queries_integration.sh— 124 / 124 (no regression)bash pipeline/queries/_tests/test_gojq_parity.sh— 54 / 54 (no regression)npm testinextractors/typescript/— 95 / 95 (no regression)shellcheckon all new scripts — cleanpublish-catalog.shthenfetch-catalogs.shagainst the samefile://bucket recovers the published catalogs byte-for-byteExplicitly out of scope (separate follow-ups)
docs/substrate.md.publish-catalog.shon every push — that's Per-repo CI publication: extract catalog on push, upload to R2 substrate #154.pipeline/queries/coverage.jqagainst the substrate's coverage block — that's Operational safety: preflight + coverage + run-cross-repo-query wrapper #155.Closes #153.