Skip to content

feat(type-catalog): emit kind:import consumer-edge rows behind --include-imports#196

Merged
jakebromberg merged 1 commit into
mainfrom
worktree-issue-152-imports-kind
May 30, 2026
Merged

feat(type-catalog): emit kind:import consumer-edge rows behind --include-imports#196
jakebromberg merged 1 commit into
mainfrom
worktree-issue-152-imports-kind

Conversation

@jakebromberg

Copy link
Copy Markdown
Owner

Summary

Schema

Field Meaning
kind "import"
name origin-side spelling (queryable against declarations); null for side-effect
import_form named | default | namespace | side-effect | re-export | dynamic | require
imported_as local binding name (== name when no as); null for side-effect / bare-call require
origin_specifier raw text from source (or "<computed>" for templated dynamic imports)
origin_package bare-specifier-stripped published name (@wxyc/shared/foo -> @wxyc/shared); null for relative/computed
origin_resolution bare-specifier | relative | computed
type_only true for import type {...} (declaration-level) or { type X } (per-binding)

See docs/pipeline-contract.md §"Import rows" for the full table including row counts per import form.

What's affected

  • New CLI flag --include-imports; manifest.toml registers it as an optional arg.
  • New _lib/origin-package.mjs pure resolver (12 tests).
  • 10 synthetic fixtures + 13 per-form assertions in test/imports.test.mjs.
  • Positive kind whitelist on name-collisions.jq and touched-window-debt-summary.jq's name-collisions section.
  • docs/pipeline-contract.md, extractors/typescript/README.md, manifest.toml updated.

Test plan

  • All extractor unit tests pass (95: type-catalog 70 + function-catalog 14 + sort 6 + artifacts 4 + origin-package 12 + imports 13, minus existing test count drift).
  • Canonical test suite (26 / 26) and integration suite (124 / 124) and gojq parity (54 / 54) — total 299 tests green.
  • On real dj-site (~160 declarations, ~3000 import rows when flag is on), every type-catalog query produces byte-identical output with vs. without the flag.
  • Wall-clock budget under 110% target.
  • Two consecutive runs against the same root produce byte-identical output (determinism).
  • Recall sanity check on dj-site: 1623 import/export statements via git grep -> 2959 per-symbol extractor rows (named-multi-symbol expansion accounts for the difference).

Closes #152.

…ude-imports

Adds one `kind: "import"` row per imported symbol when `--include-imports` is passed, providing the consumer-edge data that cross-repo queries like `consumers-of.jq` (#156) and `renamed-consumers.jq` (#158) need. Default-off so legacy catalogs and queries stay byte-stable; the flag flips to default-on once #156 exercises the data shape.

Schema (per row): `name` (origin-side spelling, null for side-effect), `import_form` (named | default | namespace | side-effect | re-export | dynamic | require), `imported_as` (local binding), `origin_specifier` (raw text), `origin_package` (bare-specifier-stripped published name, null for relative), `origin_resolution` (bare-specifier | relative | computed), `type_only`. Coverage spans `import`/`export`-from / dynamic `import()` (templated args record `<computed>`) / module-level `require()` (with destructured-name recovery).

Resolution is text-only (Tier 1, bare-specifier strip) — no `tsconfig.json` paths or `package.json` walking. Catches >=95% of cross-repo consumer edges in well-behaved npm codebases at zero filesystem cost; Tier 2 deferred until a query demands it.

Defensive guards added in the same change so existing query output stays byte-stable: positive kind whitelists on `name-collisions.jq` and `touched-window-debt-summary.jq`'s name-collisions section (mirroring `cross-package-shadows.jq`'s pattern). All other queries already filter on `.kind` / `.shape_sig` / `.fields` against whitelists that import rows don't satisfy.

Verified: 95 extractor tests + 26 canonical + 124 integration + 54 gojq-parity pass; full query suite produces byte-identical output on a real ~3000-row dj-site catalog with vs. without the flag; wall-clock budget under the 110% target.

Closes #152.
@jakebromberg
jakebromberg merged commit 22f00f0 into main May 30, 2026
3 checks passed
jakebromberg added a commit that referenced this pull request May 30, 2026
Lays down the substrate that #118's cross-repo queries (#156/#157/#158) and the per-repo CI publication step (#154) consume. Four scripts plus a hermetic mock so the whole surface is exercisable without an actual R2/S3 bucket.

`pipeline/fetch-catalogs.sh` is the auditor read path — pulls `index.json`, then every ok-repo's per-catalog SHA-keyed objects, into `$AUDIT_LOCAL_CACHE`. Warm-cache via sha256 compare; supports `--repos` filtering and `file://` URLs for tests.

`pipeline/publish-catalog.sh` is the CI write path — validates each catalog against the v1.1 wrapper, uploads under `by-repo/<R>/<iso-utc>_<short-sha>/<kind>.json`, writes a `latest.json` pointer, and triggers a self-healing index rebuild.

`pipeline/refresh-index.mjs` derives `index.json` from a bucket listing. Idempotent — running it twice produces byte-identical output; running it after manual edits or partial publishes restores correctness without losing data. The "rewrite from listing, not append-on-write" pattern the brief calls out (§7.2). Per-repo `latest.catalogs[]` array reflects the multi-catalog reality (type-catalog + function-catalog + file-hashes + package-graph), not the single-catalog assumption in the original brief.

`pipeline/verify-index.sh` is the drift detector. Reports orphan bucket prefixes and dangling index references. Run by `refresh-index` callers and a nightly cron.

All four ship with a `--bucket-fs DIR` local-filesystem backend for tests and dev — same code path, no S3 SDK required. Production uses `--bucket-name`/`--bucket-endpoint` (S3-API, sigv4 via the `aws` CLI). For Cloudflare R2 the only change is the endpoint URL.

Schema reconciles the brief against current main: the catalog substrate now publishes a *directory* of catalogs per repo (the README caveat in `docs/plans/README.md` flagged this — original brief assumed one catalog file). `latest.catalogs[]` is the per-catalog manifest; per-row `extractor` provenance comes from the v1.1 wrapper that lands in PR #181/#190/#196.

Tests: 45 hermetic cases under `pipeline/_tests/test_substrate.sh`, wired into CI's `pipeline + typescript extractor` job. Covers cold/warm fetch, repo filter, malformed-index handling, sha256 mismatch + cleanup, refresh idempotence, `--known-repos` missing-repo surfacing, stale-snapshot detection, `--dry-run` no-op, drift detection on orphan and dangling cases, publish round-trip via fetch, bare-array refusal, `--skip-refresh`.

`docs/substrate.md` covers one-time R2 provisioning, IAM/token scoping with OIDC, env-var contract, local auditor workflow, recovery procedures, and the cost model (well under $1/month at 30-repo scale on R2).

Explicitly out of scope and tracked elsewhere: real R2 bucket provisioning (operational, needs the user's Cloudflare account); live integration tests against a real bucket (per brief §6.2 — follow-up); per-repo CI workflow that calls `publish-catalog.sh` on push (#154); `coverage.jq` query against the substrate's coverage block (#155); F1/F2/F3 cross-repo queries (#156/#157/#158, all blocked on this + #152).

Closes #153.
jakebromberg added a commit that referenced this pull request May 30, 2026
…M + LOW polish)

Independent reviews of PR #196 (imports kind) and PR #199 (R2 substrate) surfaced two real bugs in the substrate's S3 path plus several MEDIUM-severity correctness issues. This change closes them all and adds regression tests for each.

**HIGH** — `pipeline/refresh-index.mjs:147,154`: `require('node:fs').unlinkSync(tmp)` inside an `.mjs` ESM module throws `ReferenceError`, silently swallowed by the surrounding `try{}catch{}`. Every S3 GET / PUT leaked a `/tmp/refresh-index-*` file in production. Hermetic tests didn't exercise the S3 path so the bug never surfaced. Fix: hoist `unlinkSync` into the top-level `import { ... } from 'node:fs'`.

**HIGH** — `pipeline/refresh-index.mjs` S3 `putBytes`: the file's own header (line 35) declares `If-Match on the current ETag; retry on 412`, and brief §2.5 / §3.4 / §7.2 prescribe ETag-CAS. The implementation was a bare `s3api put-object` with no preconditions and no retry — two concurrent publishes would silently lose one writer's view. Fix: add `headETag()` to the S3 backend; `putBytes({ifMatch})` passes `--if-match` and translates 412 to a tagged `__precondition` error; the write loop reads the ETag, attempts the PUT, and on precondition failure retries with capped exponential backoff (up to 5 attempts).

**MEDIUM** — `pipeline/refresh-index.mjs`: when a newer SHA-keyed prefix exists in the bucket but `latest.json` still points at the older one (interrupted publish, partial restore, manual upload), the emitted index record mixed `prefix` and `short_sha` from the bucket-derived newest with `commit_sha` and `published_at` read from the stale pointer. The mismatched `published_at` also poisoned the `status` calculation. Fix: trust `latest.json`'s metadata only when its `prefix` field actually matches the chosen `latestPrefix`. Otherwise log a WARN, derive `short_sha` + `published_at` from the prefix directory name, and emit `commit_sha: null`. New `refresh-index.mjs: latest.json points at older prefix than bucket newest (drift)` test exercises this.

**MEDIUM** — `pipeline/publish-catalog.sh`: validation and upload were interleaved in a single `while ... | read` loop. If catalog A validated and uploaded, then catalog B failed validation, the script exited 1 leaving A as an orphan under the SHA-keyed prefix with no `latest.json` ever written — and the next `refresh-index` run would treat the orphan as the canonical "latest." Fix: split into a validate-all-first pass (record failure to a tempfile so the subshell `exit` doesn't get swallowed) then an upload-all-after pass; the bucket stays untouched if any catalog fails validation. New `publish-catalog.sh: mixed-validity batch uploads NOTHING (no orphan SHA prefix)` test exercises this.

**MEDIUM** — `pipeline/fetch-catalogs.sh`: the `distinct_repos` counter was incremented for every repo named in the work list (i.e., every ok-status repo in the index) regardless of whether any catalog was actually present in the cache afterwards. An index with 3 ok repos where all GETs failed would exit 0 with an empty cache and `Summary: 3 repos / 0 fetched / 0 cached / 3 failed`. Downstream `jq -s 'map(.entries) | add'` then silently produced empty output. Fix: rename the counter to `present_repos` and only `echo` to its tempfile after a successful cache hit or fetch. New `fetch-catalogs.sh: exits 1 when every fetch fails` test exercises this.

**LOW polish** rolled into the same change:

- `extractors/typescript/type-catalog.mjs:524`: `const { a: { b } } = require('pkg')` previously emitted zero rows (every element was `continue`d as a nested binding, then the explicit `return` short-circuited the bare-call fallback) so the consumer edge to the package was completely lost. Now falls through to the namespace-form `pushRequire({ name: '*', imported_as: null })` so the package edge survives. Fixture `12-nested-require.ts` + matching assertion added.
- `extractors/typescript/type-catalog.mjs:467`: `import {} from "pkg"` (empty named-imports clause — semantically a side-effect import) now emits a side-effect row instead of zero rows. Fixture `11-empty-named.ts` + matching assertion added.
- `extractors/typescript/type-catalog.mjs:404-409`: corrected the misleading "interleave with declaration rows by line" comment to describe actual ordering (top-level imports first in source-line order; declarations + dynamic-import + require rows after, in AST-walk order). Both halves remain deterministic, which is what the contract depends on.
- `pipeline/refresh-index.mjs`: `--bucket-fs DIR` now preflight-checks the directory exists with a friendly error message instead of crashing with a raw `ENOENT mkdir` stack inside `putBytes`.
- `pipeline/refresh-index.mjs:196`: dropped the hardcoded `wxyc/` org prefix in the canonical-name fallback. When `latest.json` is missing or unparseable, `canonicalRepo` now falls back to `pathSegment` alone — keeps the substrate org-agnostic.

Verified: 97 extractor tests + 26 canonical + 124 integration + 54 gojq-parity + 56 substrate = 357 tests pass. shellcheck on the four substrate shell scripts is clean.

Refs #152, #153.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Imports kind: emit consumer-edge rows in the TypeScript catalog

1 participant