create Babel-based AST parser for CommonJS sources

[boneskull]

feat(compartment-mapper): add Babel-based CJS parser with import() support

Adds parse-cjs-babel.js, a new opt-in CJS parser that uses CjsModuleSource from @endo/module-source instead of the character-level lexer. Consumers swap it in via parserForLanguage:

import { parserForLanguageWithCjsBabel } from '@endo/compartment-mapper/import-parsers.js';

await importLocation(readPowers, entryUrl, {
  parserForLanguage: parserForLanguageWithCjsBabel,
});

Also adds import() support for CJS modules:

• parse-cjs-shared-export-wrapper.js now provides an importFn that invokes compartment.import(), bridging dynamic imports in CJS onto the SES module loader
• The CJS functor receives $h͏_import as a parameter when import() calls are detected, matching the ESM evasion pattern

The existing parse-cjs.js (lexer-based) remains the default; nothing breaks for current consumers.

Includes integration tests and test fixtures covering both the new parser and dynamic import() in CJS.

May supersede #3154.

feat(module-source): add CJS Babel AST analysis, transformation, and CjsModuleSource

Replaces the character-level lexer approach to CJS module analysis with Babel AST visitors, enabling @endo/parser-pipeline to share a single parse across CJS analysis, transformation, evasive-transform, and other consumers.

New exports:

• CjsModuleSource: constructor that parses, analyzes, transforms CJS source and produces a frozen record with imports, exports, reexports, cjsFunctor, and __needsImport__
• createCjsModuleSourcePasses(): paired analyzer/transform visitor passes for use with @endo/parser-pipeline, including buildRecord

The CJS Babel plugin detects all patterns the lexer handles: require(), exports.*, module.exports, Object.defineProperty, Babel/TS reexport helpers, esbuild hints, spread reexports, and import() calls. The transform rewrites import() to the SES hidden identifier ($h͏_import) for compartment evasion.

Includes 77 new tests: unit tests for both APIs plus a full compat suite ported from @endo/cjs-module-analyzer's test cases.

---

This is probably a better solution than #3154 for adding import() support to CommonJS. We may want to do both... unsure.

There could be a performance penalty to using the new parser over the lexer, but I'm hoping that leveraging @endo/parser-pipeline (#3158) in LavaMoat will mitigate that.

[boneskull]

Warning

This PR is part of a stack and targets branch boneskull/module-source-pipeline, not master.
DO NOT MERGE until feat(module-source): expose createModuleSourcePasses() #3159 is merged into master.

📚 Pull Request Stack

• branch: master
  • fix(compartment-mapper,bundle-source): async parser support #3186 - branch: boneskull/async-parser
    • fix(module-source): fix SourceMapHook & other types #3187 - branch: boneskull/fix-module-source-types
      • feat(parser-pipeline): create @endo/parser-pipeline #3158 - branch: boneskull/ast-service
        • feat(evasive-transform): expose createEvasiveTransformPass for use with @endo/parser-pipeline #3188 - branch: boneskull/evasive-pipeline
          • feat(module-source): expose createModuleSourcePasses() #3159 - branch: boneskull/module-source-pipeline
            • create Babel-based AST parser for CommonJS sources #3162 - branch: boneskull/cjs-ast-parser (this PR)
              • feat(compartment-mapper): expose findUnknownCanonicalNames() #3205 - branch: boneskull/export-find-canonical-names

---

Managed by gh-stack

[changeset-bot]

🦋 Changeset detected

Latest commit: d74629e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@endo/compartment-mapper	Minor
@endo/module-source	Minor
@endo/parser-pipeline	Patch
@endo/bundle-source	Patch
@endo/check-bundle	Patch
@endo/daemon	Patch
@endo/import-bundle	Patch
@endo/test262-runner	Patch
@endo/cli	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Copilot]

Pull request overview

Adds an opt-in Babel-AST-based CommonJS parsing/analysis path (via @endo/module-source) and wires dynamic import() bridging for CJS execution in compartment-mapper, alongside extensive test coverage and fixtures.

Changes:

• Introduce CjsModuleSource + CJS visitor passes / functor builder in @endo/module-source.
• Add parse-cjs-babel and parserForLanguageWithCjsBabel opt-in parser in @endo/compartment-mapper.
• Add integration/compat tests and fixtures for CJS/ESM dynamic import() and CJS analyzer compatibility.

Reviewed changes

Copilot reviewed 20 out of 25 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
packages/module-source/test/cjs-visitor-passes.test.js	New tests for CJS Babel visitor passes + transform behavior (incl import() rewrite).
packages/module-source/test/cjs-module-source.test.js	New unit tests for CjsModuleSource record shape, freezing, shebang handling, and import() support.
packages/module-source/test/cjs-compat.test.js	Large compat suite ported from lexer analyzer tests, targeting CjsModuleSource.
packages/module-source/src/visitor-passes.js	Add createCjsModuleSourcePasses() alongside existing ESM passes.
packages/module-source/src/types/visitor-passes.ts	Add CJS analysis/passes types and buildRecord() return type.
packages/module-source/src/types/module-source.ts	Add CJS record/state types; reorganize/extend source map hook types.
packages/module-source/src/source-options.js	Add createCjsSourceOptions() state bag factory for CJS plugin.
packages/module-source/src/cjs-transform-analyze.js	New parse/traverse/generate pipeline to produce frozen CjsModuleSourceRecord.
packages/module-source/src/cjs-module-source.js	New CjsModuleSource constructor wrapper around the analyzer.
packages/module-source/src/cjs-functor.js	Build CJS functor source and assemble frozen CJS module record.
packages/module-source/src/cjs-babel-plugin.js	New Babel plugin for CJS analysis (require/exports/reexports/import()) + transform (import() rewrite + hidden-id guard).
packages/module-source/README.md	Document CJS variant and tweak XS heading.
packages/module-source/index.js	Export CjsModuleSource + createCjsModuleSourcePasses.
packages/compartment-mapper/test/fixtures-dynamic-import-esm/node_modules/app/package.json	Add ESM dynamic import fixture package metadata.
packages/compartment-mapper/test/fixtures-dynamic-import-esm/node_modules/app/index.js	Fixture: ESM module that uses dynamic import().
packages/compartment-mapper/test/fixtures-dynamic-import-esm/node_modules/app/foo.js	Fixture: ESM dependency providing default export.
packages/compartment-mapper/test/fixtures-cjs-compat/node_modules/dynamic-import/package.json	Add CJS dynamic import fixture package metadata.
packages/compartment-mapper/test/fixtures-cjs-compat/node_modules/dynamic-import/index.js	Fixture: CJS module exporting async fn that calls import().
packages/compartment-mapper/test/dynamic-import-esm.test.js	Integration test for ESM dynamic import() via scaffold.
packages/compartment-mapper/test/cjs-compat.test.js	Run compat suite with both default lexer parser and new Babel CJS parser; add dynamic-import CJS test.
packages/compartment-mapper/src/types/external.ts	Minor import ordering adjustment (SourceMapObject import).
packages/compartment-mapper/src/parse-cjs-shared-export-wrapper.js	Add importFn that bridges to compartment.import() for CJS functors.
packages/compartment-mapper/src/parse-cjs-babel.js	New opt-in CJS parser using CjsModuleSource and importFn injection when needed.
packages/compartment-mapper/src/import-parsers.js	Add parserForLanguageWithCjsBabel helper alongside default parsers.
packages/compartment-mapper/import-parsers.js	Re-export parserForLanguageWithCjsBabel from package entrypoint.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.