refactor(type-catalog): extract compareBy + writeSiblingArtifact helpers#190
Merged
Merged
Conversation
Pulls three hand-rolled multi-key sort comparators and two parallel sibling-artifact emission blocks out of type-catalog.mjs into reusable _lib helpers. Same byte-identical output, fewer lines, single source of truth for the sibling-wrapper shape.
While consolidating the writers, references.json gains the {language, name, version} extractor block that the wrapper convention requires (catalog.json and files.json already had it; references.json was the lone holdout). Additive change — consumers reading `.edges` keep working unchanged. Documented in docs/pipeline-contract.md.
Closes #185
Closes #186
jakebromberg
added a commit
that referenced
this pull request
May 30, 2026
Lays down the substrate that #118's cross-repo queries (#156/#157/#158) and the per-repo CI publication step (#154) consume. Four scripts plus a hermetic mock so the whole surface is exercisable without an actual R2/S3 bucket. `pipeline/fetch-catalogs.sh` is the auditor read path — pulls `index.json`, then every ok-repo's per-catalog SHA-keyed objects, into `$AUDIT_LOCAL_CACHE`. Warm-cache via sha256 compare; supports `--repos` filtering and `file://` URLs for tests. `pipeline/publish-catalog.sh` is the CI write path — validates each catalog against the v1.1 wrapper, uploads under `by-repo/<R>/<iso-utc>_<short-sha>/<kind>.json`, writes a `latest.json` pointer, and triggers a self-healing index rebuild. `pipeline/refresh-index.mjs` derives `index.json` from a bucket listing. Idempotent — running it twice produces byte-identical output; running it after manual edits or partial publishes restores correctness without losing data. The "rewrite from listing, not append-on-write" pattern the brief calls out (§7.2). Per-repo `latest.catalogs[]` array reflects the multi-catalog reality (type-catalog + function-catalog + file-hashes + package-graph), not the single-catalog assumption in the original brief. `pipeline/verify-index.sh` is the drift detector. Reports orphan bucket prefixes and dangling index references. Run by `refresh-index` callers and a nightly cron. All four ship with a `--bucket-fs DIR` local-filesystem backend for tests and dev — same code path, no S3 SDK required. Production uses `--bucket-name`/`--bucket-endpoint` (S3-API, sigv4 via the `aws` CLI). For Cloudflare R2 the only change is the endpoint URL. Schema reconciles the brief against current main: the catalog substrate now publishes a *directory* of catalogs per repo (the README caveat in `docs/plans/README.md` flagged this — original brief assumed one catalog file). `latest.catalogs[]` is the per-catalog manifest; per-row `extractor` provenance comes from the v1.1 wrapper that lands in PR #181/#190/#196. Tests: 45 hermetic cases under `pipeline/_tests/test_substrate.sh`, wired into CI's `pipeline + typescript extractor` job. Covers cold/warm fetch, repo filter, malformed-index handling, sha256 mismatch + cleanup, refresh idempotence, `--known-repos` missing-repo surfacing, stale-snapshot detection, `--dry-run` no-op, drift detection on orphan and dangling cases, publish round-trip via fetch, bare-array refusal, `--skip-refresh`. `docs/substrate.md` covers one-time R2 provisioning, IAM/token scoping with OIDC, env-var contract, local auditor workflow, recovery procedures, and the cost model (well under $1/month at 30-repo scale on R2). Explicitly out of scope and tracked elsewhere: real R2 bucket provisioning (operational, needs the user's Cloudflare account); live integration tests against a real bucket (per brief §6.2 — follow-up); per-repo CI workflow that calls `publish-catalog.sh` on push (#154); `coverage.jq` query against the substrate's coverage block (#155); F1/F2/F3 cross-repo queries (#156/#157/#158, all blocked on this + #152). Closes #153.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extractors/typescript/type-catalog.mjsinto two small_libhelpers:compareBy(...keyFns)andwriteSiblingArtifact({...}). Same byte-identical output for catalog/files/edges payloads, fewer lines, single source of truth for the sibling-wrapper shape.extractor: {language, name, version}block toreferences.json's wrapper — the lone sibling artifact that wasn't carrying it. Additive change; existing consumers reading.edgeskeep working. Documented as such indocs/pipeline-contract.md.Test plan
npm testinextractors/typescript/— 70 tests pass (10 new acrosssort.test.mjs+artifacts.test.mjs, plus areferences.jsonwrapper-shape assertion added toextract.test.mjs).pipeline/queries/_tests/test_queries_integration.sh— 124 pass / 0 fail.pipeline/queries/_tests/test_canonical.sh— 26 pass / 0 fail.pipeline/queries/_tests/test_gojq_parity.sh— 54 pass / 0 fail.fixtures-files/— catalog.json, references.json, files.json all emit with the unified{schema_version, extractor, <payloadKey>}shape.Closes #185
Closes #186