Skip to content

wip: add compact + compact-js with ledger 9 support#1711

Draft
ozgb wants to merge 21 commits into
mainfrom
ozgb-compact-ledger-9-pre
Draft

wip: add compact + compact-js with ledger 9 support#1711
ozgb wants to merge 21 commits into
mainfrom
ozgb-compact-ledger-9-pre

Conversation

@ozgb

@ozgb ozgb commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

This PR adds pre-release compact + compact-js with ledger 9 support.

Because these packages are unpublished, this PR adds a new workflow to build and commit the unreleased packages as .tar.gz blobs. This means node repo devs can still run npm i without requiring any midnight org tokens and without installing nix and yarn (required for building compact-js and compact-runtime)

🗹 TODO before merging

  • Ready

📌 Submission Checklist

  • All commits are signed off (git commit -s) for the DCO
  • Changes are backward-compatible (or flagged if breaking)
  • Pull request description explains why the change is needed
  • Self-reviewed the diff
  • I have included a change file, or skipped for this reason:
  • If the changes introduce a new feature, I have bumped the node minor version
  • Update documentation (if relevant)
  • Updated AGENTS.md if build commands, architecture, or workflows changed
  • No new todos introduced

🧪 Testing Evidence

Please describe any additional testing aside from CI:

  • Additional tests are provided (if possible)

🔱 Fork Strategy

  • Node Runtime Update
  • Node Client Update
  • Other:
  • N/A

Links

#1624

ozgb added 2 commits June 16, 2026 15:00
@ozgb
refactor(toolkit): dispatch toolkit-js compact variants on full patch…
4fcf424 
… version

toolkit-js now selects its compact-js variant on the full
<major>.<minor>.<patch> compactc version instead of truncating to
<major>.<minor>, since a compactc patch can ship a contract format expecting a
different compact-js patch. Renames the variant workspaces (compact-0.29 ->
compact-0.29.0, etc.), strips any tree-hash suffix from COMPACTC_VERSION when
matching, and errors on unsupported patches.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
feat(toolkit): add compact-0.31.108 toolkit-js variant (compact-js 2.…
810d640 
…5.3)

Add a toolkit-js variant for ledger 9 / compactc 0.31.108, pinning
@midnight-ntwrk/compact-js* 2.5.3. Since 2.5.3 and its matching
compact-runtime are unpublished, the variant consumes five npm-pack
tarballs via file: references (compact-js{,-command,-node}, compact-runtime,
ledger-v9); everything else resolves from public npm, so the consume path
needs no registry token.

The blobs are built in CI from the new midnight-sdk submodule (pinned to
49f45a76) by the +compact-js-bundle Earthly target
(scripts/build-compact-js-bundle.sh) and committed to the PR branch by the
rebuild-compact-js-bundle-bot workflow (/bot rebuild-compact-js-bundle).
The vendor/ dir ships only a placeholder README until the bot populates it.

Also regenerates COMPACTC_VERSION from the bumped compact/ submodule
(0.31.108-0eb1677c6f54) and registers the variant in the resolver, root
package.json, the compat test script, README, and AGENTS.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb ozgb force-pushed the ozgb-compact-ledger-9-pre branch from b389bef to 810d640 Compare June 17, 2026 08:59
@ozgb

ozgb commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

/bot rebuild-compact-js-bundle

ozgb added 7 commits June 17, 2026 15:07
@ozgb
chore(toolkit): refine compact-js-bundle build for real SDK layout
0209ca1 
Update the +compact-js-bundle Earthly target and build script to match the
actual midnight-sdk layout and build mechanics discovered while running it:

- build script: use the real package dirs (compact-js/{compact-js,
  compact-js-command,compact-js-node} + compact-submodule/runtime), build the
  runtime via nix with an explicit `path:` flake ref (the COPY'd submodule has
  no .git, so a bare `nix develop` fails to resolve the gitdir), then turbo-build
  the three packages; pin each package's compact-runtime dep to the concrete
  built version before packing.
- Earthfile: drop `nixpkgs#yarn` (corepack provides the SDK's pinned yarn
  4.10.3), fix the `--secret` form, and add buildkit CACHE mounts (nix download
  cache, npm, yarn) to speed up re-runs.

Validated end-to-end up to building the SDK's own compact-js: the pinned WIP
commit (49f45a76) does not yet typecheck because its compact-runtime (new
events-in-circuit API) is ahead of its platform-js/ledger-v8 deps (old API).
Pending an SDK-side bump; re-pin midnight-sdk to the consistent commit to finish.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
fix(ci): pass compact-js-bundle secret via env, not command line
0e8c434 
Use Earthly's `--secret NAME` (no value) form so the GitHub Packages token is
read from the environment instead of being interpolated onto the command line,
keeping it out of the build log.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
chore(toolkit): bump compact + midnight-sdk submodules (ledger-9 dev …
116d64f 
…sync)

Track the compact-runtime dev's progress:
- midnight-sdk: 49f45a76 -> ae2780749 (compact-js-v2.5.2-10), which re-pins its
  nested compact-submodule to f3a751c7 (compactc-v0.31.0-91).
- compact: 6624fcf3 -> f3a751c7 to match the SDK's nested compact (both 0.31.108),
  so the compactc that compiles contracts matches the compact-runtime the variant
  ships. Regenerates COMPACTC_VERSION (0.31.108-3ab382536931).

Note: the bundle build still does not complete — compact-js at ae2780749 still
depends on platform-js@^2.2.4 (-> ledger-v8@8.0.2, old events/SigningKey API),
which conflicts with the new-events compact-runtime. Pending the compact-js dev
bumping platform-js/ledger; then re-run +compact-js-bundle.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
chore(toolkit): bump midnight-sdk to 21f6e4d8 (ledger-v9 rc.2)
f9012c6 
compact-js-v2.5.2-12-g21f6e4d. This moves compact-js's deps to published
versions — @midnight-ntwrk/compact-runtime@0.16.103-dev (GH Packages) and
@midnightntwrk/ledger-v9@^1.0.0-rc.2 (public npm) — which resolves the earlier
QueryContext/events + SigningKey type errors: the SDK's compact-js now builds
cleanly (yarn install + turbo build, no nix / no compact-submodule runtime build).

Still blocked for npm consumption by the toolkit variant: compact-js-command@2.5.3
mixes @effect/* releases (platform/platform-node 0.96-era, but cli@^0.74.0 /
printer@^0.48.0 / effect@^3.20.0 0.95-era). @effect/cli@0.74's peer wants
@effect/platform <0.96, so npm strict-peer resolution rejects it; forcing cli@0.75
skews types against the 3.20-built packages. Pending the compact-js dev aligning
@effect/* to one release.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
feat(toolkit): vendor compact-js 2.5.3 blobs for the compact-0.31.108…
b7af423 
… variant

Add the four GitHub-Packages-only tarballs the variant consumes via file:
references, built from midnight-sdk @ 21f6e4d8 (compact-js 2.5.3):

- compact-js.tgz, compact-js-command.tgz, compact-js-node.tgz — built from the
  SDK source (yarn install + turbo build; plain TS, no nix / no compact-submodule)
- compact-runtime.tgz — npm pack of the published @midnight-ntwrk/compact-runtime
  @0.16.103-dev.<hash> from GitHub Packages (no longer built from source)

Drop the vendored ledger-v9: at this SDK rev it's @midnightntwrk/ledger-v9
@^1.0.0-rc.2, published to public npm, so it (and onchain-runtime-v4, platform-js,
wallet-sdk-address-format, ledger-v8) resolves token-free. Vendor set is now 4.

Known limitation: npm strict-peer resolution can't yet install compact-js-command
@2.5.3 — it mixes @effect/* releases (platform 0.96-era vs cli@^0.74/printer@^0.48/
effect@^3.20 0.95-era), and @effect/cli@0.74's peer wants @effect/platform <0.96.
Pending the compact-js dev aligning @effect/*; the lockfile regen + variant
typecheck finish then.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
feat(toolkit): compact-0.31.108 variant builds against compact-js 2.5.3
cf97665 
The variant now typechecks and installs token-free end to end:

- midnight-sdk -> 0aaf8fc6: the compact-js dev aligned @effect/* to one 0.96-era
  release (cli ^0.75.0, printer ^0.49.0, printer-ansi ^0.49.0, effect ^3.21.2),
  fixing the strict-peer ERESOLVE that blocked npm consumption.
- variant package.json: @effect/cli ^0.75.0 + effect ^3.21.2 to match.
- re-pack the 3 compact-js blobs via the SDK's build-utils `package` step (npm pack
  from each package's processed dist/), not a raw npm pack of the source dir — the
  source-dir exports point at ./src/*.ts, whose .d.ts didn't match @effect/cli's
  Command type (the [TypeId] mismatch). compact-runtime.tgz is unchanged (published).
- regenerate package-lock.json: records all four file: blobs with sha512 integrity;
  compact-runtime hoists to top-level; everything else (ledger-v9 rc.2, onchain-
  runtime-v4, platform-js, ...) resolves from public npm with no token.

Verified: `npm install` (no token) + `npm run build` -> 4/4 variants green;
resolver dispatch 0.31.108-<hash> -> @midnight-ntwrk/node-toolkit-compact-0.31.108.
Remaining: runtime vitest needs compactc 0.31.108 (just compactc) to recompile the
test contracts; run `./scripts/test-all-compactc.sh 0.31.108` once it's on PATH.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
refactor(toolkit): rework compact-js-bundle to the no-nix flow
c60f141 
Now that compact-runtime is a published dev build (not an in-tree file: runtime),
the bundle build is pure JS — drop nix, the compact-submodule runtime build, and
the recursive submodule checkout.

- scripts/build-compact-js-bundle.sh: `yarn install` + the SDK's build-utils
  `package` step, then pack the three compact-js blobs from each package's dist/
  (the published layout — a raw npm pack of the source dir emits .d.ts that don't
  match @effect/cli's Command type). compact-runtime is npm pack'd from GitHub
  Packages at the version compact-js depends on (read dynamically). 4 blobs, no
  ledger-v9 (now public npm).
- Earthfile +compact-js-bundle: FROM node:22-bookworm (+ node-gyp toolchain)
  instead of nixos/nix; CACHE npm + yarn caches; SAVE the 4 blobs.
- rebuild-compact-js-bundle-bot: submodules: true (non-recursive) — only the SDK's
  compact-js source is needed; its nested compact-submodule is unused.
- vendor/README: document the build-utils package step; drop the stale nix notes.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@datadog-official

This comment has been minimized.

Sign in to view
ozgb added 12 commits June 19, 2026 11:29
@ozgb
chore: bump compact to released dev version
028c9a5 
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
feat(toolkit): fetch compactc dev builds by commit SHA
f1a4b6a 
+compactc-fetch now picks the release tag/asset from the COMPACTC_VERSION
suffix: a bare 40-char hex commit SHA fetches a public dev build
(compactc-dev-<sha> / compactc_dev-<sha>_<arch>...), while a plain version or
semver pre-release (e.g. 0.30.0-rc.1) keeps the conventional compactc-v<version>
naming. Pin COMPACTC_VERSION to the dev build
0.31.108-73ebfbbff78118e77a83fdc99dca352db0020869 (compiler 0.31.108, already a
supported toolkit-js variant). The compact submodule and its build-from-source
path are unchanged.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
fix(toolkit): correct +compactc-fetch artifact COPY syntax
e2d7d79 
The build-arg artifact COPY in node-ci-image-single-platform put the artifact
path outside the parens — (+compactc-fetch --VERSION=...)/compact-home — which
Earthly cannot parse as a target reference, so it walks the literal string as a
path and fails with "no such file or directory". Move the artifact path inside
the parens: (+compactc-fetch/compact-home --VERSION=...).

This ELSE/fetch branch was never exercised before because CI's COMPACTC_VERSION
always matched the submodule tree-hash (IF branch -> +compactc-bundle); pinning
a dev-build commit SHA routes through it for the first time.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
Merge branch 'main' into ozgb-compact-ledger-9-pre
ddcd3bf
@ozgb
Merge branch 'main' into ozgb-compact-ledger-9-pre
f4a28c8
@ozgb
chore(toolkit-js): re-pin compact-js SDK and refresh vendored ledger-…
a0fcdd7 
…9 bundle

Re-pin midnight-sdk 0aaf8fc -> 44bec76, which lands the relative-import ".js"
extension fix (merged PR #217) and aligns effect to ^3.21.2 across the compact-js
workspaces (fixing the effect dual-package hazard that broke +compact-js-bundle).
Refresh the vendored compact-0.31.108 tarballs (compact-js, compact-js-command,
compact-js-node) so they carry the ESM fix, and regenerate package-lock.json.

Assisted-by: Claude:claude-opus-4-8[1m]
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
fix(toolkit-js): pin contract-config type-check to the active compact…
c650a0e 
…-js variant

compact-js-command's ConfigCompiler type-checks contract.config.ts with ts-node,
which does not honor the runtime registerHooks resolver. It therefore resolved
@midnight-ntwrk/compact-js to whichever variant npm hoisted to the workspace root
(an older line, e.g. 2.4.3) and typed the contract against a mismatched
@midnight-ntwrk/compact-runtime, collapsing the generated Witnesses type to `never`
and failing config compilation in a clean build.

Inject TS_NODE_COMPILER_OPTIONS `paths` from installCompactcResolver, pinning
compact-js / compact-runtime resolution to the COMPACTC_VERSION-matched variant
(resolved via <pkg>/package.json). Computed before the hook is registered so the
lookups can't re-enter it.

Assisted-by: Claude:claude-opus-4-8[1m]
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
chore(toolkit): regenerate counter test fixtures for ledger 9 rc.2
4341284 
Bumping the Rust ledger from crate-ledger-9.0.1.0-alpha.1 to
crate-ledger-9.1.0.0-rc.2 (via #1692, merged through main) advanced
the serialization tags. Regenerate the checked-in counter fixtures so
they match the toolkit-js / compact-js 2.5.3 output and the rc.2 Rust
ledger:

  contract_state.mn   contract-state[v6] -> contract-state[v8]
  deploy.bin          intent[v6]         -> intent[v8]
  deploy_tx.mn        transaction[v9]    -> transaction[v11]
  contract_address.mn (re-derived from the new deploy_tx)

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
test(toolkit): re-enable ledger-9 toolkit-js tests and e2e CI jobs
ffd9f39 
Now that the Rust ledger is on crate-ledger-9.1.0.0-rc.2 (intent[v8]),
matching the toolkit-js / compact-js 2.5.3 output, revert the
LEDGER9-TOOLKIT-JS gates that were holding back the toolkit-js
generate-intent path:

- generate_intent.rs: drop the #[ignore]s on test_generate_deploy,
  test_generate_circuit_call, test_generate_maintain_contract and
  test_generate_maintain_remove_circuit (test_generate_maintain_circuit
  keeps only its pre-existing intermittent-failure ignore).
- toolkit_e2e.rs: drop the #[ignore] on bboard_private_witness_not_leaked.
- Earthfile: drop the GENERATE_JS_TEST_TXS gate so the counter/mint
  JS test-tx fixtures are generated under the normal GENERATE_TEST_TXS arg.
- CI: uncomment the toolkit-maintenance / mint / tokens-minter / contracts
  e2e jobs.

All four `commands::generate_intent::test` tests pass against the
regenerated rc.2 fixtures. No LEDGER9-TOOLKIT-JS markers remain.

Closes: #1624
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
Merge branch 'main' into ozgb-compact-ledger-9-pre
f49b550 
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
chore(toolkit): bump compactc 0.31.108 → 0.31.110 dev build
d76169d 
Repin COMPACTC_VERSION to the 0.31.110 dev build
(3a289c2e7811d2868e7810bd5a5f1f0b7055995f, fetched from the LFDT-Minokawa
compactc-dev release) and rename the toolkit-js variant
compact-0.31.108 → compact-0.31.110 so the resolver's major.minor.patch
dispatch selects it.

No bundle rebuild needed: the vendored compact-runtime.tgz is already
0.16.103-dev.3a289c2e, built from the same source commit as the new
compiler (compact-js 2.5.3 unchanged). The compact/ submodule is left in
place; the dev-build SHA suffix differs from its tree hash, so CI and
just compactc fetch the prebuilt binary rather than building from source.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb
docs(toolkit): consolidate ledger-9 compact-js change files
d141356 
Merge the four toolkit-js / compactc ledger-9 change files
(compact-0.31.108 variant, full-patch dispatch, dev-build fetch,
0.31.110 bump) plus the re-enabled-tests note into a single
compact-js-ledger-9-support.md describing the PR's final state,
and add the PR #1711 link.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
@ozgb ozgb mentioned this pull request Jun 26, 2026
Open
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ozgb