feat(parser-pipeline): create @endo/parser-pipeline

[boneskull]

Introduces @endo/parser-pipeline — a new package that eliminates redundant
Babel AST parsing for @endo/compartment-mapper consumers. Tools like LavaMoat
have historically parsed each module 2–3 times (once for import/export analysis,
once for evasive transforms, once for policy-globals analysis). This package
composes all those passes into a single parse → analyze → transform → generate
cycle per module.

Core API

createParsers(configs, options?) is the primary entry point. It accepts a
language-keyed config map and returns { sync, async } parser maps compatible
with @endo/compartment-mapper's parserForLanguage option. The
@endo/module-source analysis step is handled implicitly by the pipeline based on the
source language (mjs or cjs); consumers layer in user-defined analyzer and
transform factories.

moduleSourceConfigs(extras?) builds a pre-wired config map for both ESM and
CJS by calling analyzeModule() / analyzeCjs() from @endo/module-source
internally. Extra factories and per-language finalizeRecord hooks are merged
on top.

createEvasiveTransformPass(options?) wraps @endo/evasive-transform's
visitor as a pipeline-compatible TransformPass owned by this package (so
structural compatibility is verified at compile time, not by duck typing).

Workers

runPipelineInWorker(port, configs) powers the async path. Consumer worker
scripts call this to handle parse messages dispatched by WorkerParserPool.
Workers are unref()'d at spawn time; an idle timeout terminates them after
inactivity. Pending dispatches hold the event loop via their main-thread
promises, so the process exits cleanly once the last dispatch settles — no
explicit terminate() call required.

WorkerParserPool manages worker spawning, queueing when at capacity, idle
timeouts, error handling, and result routing. Tests cover pool exhaustion,
worker crash, unexpected exit, and malformed response shapes.

Internals

runPipeline(params) is the shared core (used by both sync and async paths)
that owns the Babel parse → implicit module-source analyze → user analyzers →
user transforms → implicit module-source transform → generate → buildRecord
sequence.

The package exports AnalyzerFactory, TransformFactory, AnalyzerPass,
TransformPass, PipelineConfigs, CreateParsersOptions, CreateParsersResult,
and the worker-pool types.

[boneskull]

Warning

This PR is part of a stack and targets branch boneskull/export-find-canonical-names2, not master.
DO NOT MERGE until feat(compartment-mapper): expose findUnknownCanonicalNames #3221 is merged into master.

📚 Pull Request Stack

• branch: master
  • feat(evasive-transform): expose makeEvasiveTransformVisitor #3217 - branch: boneskull/evasive-pipeline2
    • fix(compartment-mapper): policy, parser type fixes #3247 - branch: boneskull/fix-module-source-types2
      • feat(module-source): fix types, expose analyzer.js subpath #3218 - branch: boneskull/module-source-pipeline2
        • feat(module-source): expose AST-based CJS parser #3220 - branch: boneskull/cjs-ast-parser2
          • feat(compartment-mapper): expose findUnknownCanonicalNames #3221 - branch: boneskull/export-find-canonical-names2
            • feat(parser-pipeline): create @endo/parser-pipeline #3158 - branch: boneskull/ast-service (this PR)

---

Managed by gh-stack

[changeset-bot]

🦋 Changeset detected

Latest commit: 11a3feb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@endo/parser-pipeline	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[boneskull]

TODO: figure out why the docs build is failing

[Copilot]

Pull request overview

This PR introduces a new @endo/parser-pipeline workspace package to avoid redundant Babel AST parsing by composing multiple analyzer/transform visitor passes into a single parse→traverse→generate cycle, and adds an async worker-pool-backed parser option for parallel policy-generation workflows. It also updates @endo/compartment-mapper’s parser typing and mapping logic to support asynchronous parsers alongside synchronous ones.

Changes:

• Add @endo/parser-pipeline with createComposedParser (sync) and createWorkerParser (async worker pool), plus worker-side runPipelineInWorker.
• Refactor @endo/compartment-mapper parser types and makeMapParsers to support async parsers and async module transforms.
• Add tests for composed parsing and worker-pool behavior.

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
yarn.lock	Adds new workspace package and Babel-related deps/types.
typedoc.json	Formatting tweaks and excludes the new package from typedoc output.
packages/parser-pipeline/package.json	Defines the new package, dependencies, and exports.
packages/parser-pipeline/index.js	Public entrypoint exporting types + parser constructors.
packages/parser-pipeline/src/composed-parser.js	Sync composed parser implementation.
packages/parser-pipeline/src/worker-parser.js	Async parser wrapper backed by worker pool.
packages/parser-pipeline/src/worker-pool.js	Worker thread pool and dispatch/queue/terminate logic.
packages/parser-pipeline/src/worker-runner.js	Worker-side message listener that runs the pipeline per task.
packages/parser-pipeline/src/types/external.ts	Public types for pipeline configuration and worker protocol.
packages/parser-pipeline/src/external.types.*	Type-export shim for JS consumers.
packages/parser-pipeline/test/*	AVA tests for composed parser, worker runner, and worker pool.
packages/parser-pipeline/README.md	New package documentation and usage examples.
packages/parser-pipeline/SECURITY.md	Package security policy doc.
packages/parser-pipeline/LICENSE	Apache-2.0 license file.
packages/parser-pipeline/tsconfig*.json	Typechecking/build config for the new package.
packages/compartment-mapper/src/types/external.ts	Splits sync vs async parser types and updates ParserForLanguage typing.
packages/compartment-mapper/src/types/internal.ts	Adjusts internal operator typings to accept ParseFn or AsyncParseFn.
packages/compartment-mapper/src/map-parser.js	Refactors parser selection/trampolines for sync+async support.
packages/compartment-mapper/src/import-hook.js	Adds a sync-parse type guard for importNow/dynamic-require constraints.
packages/compartment-mapper/src/link.js	Simplifies heuristicImports access with updated parser types.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.