Skip to content

feat(client,cli): add multi-packaging support for wkg publish#215

Open
mkatychev wants to merge 64 commits into
bytecodealliance:mainfrom
mkatychev:feat/multipackage-publishing
Open

feat(client,cli): add multi-packaging support for wkg publish#215
mkatychev wants to merge 64 commits into
bytecodealliance:mainfrom
mkatychev:feat/multipackage-publishing

Conversation

@mkatychev

@mkatychev mkatychev commented Jun 10, 2026

Copy link
Copy Markdown
Member

Added the ability to publish multiple WIT packages at once.
Introduced the PublishPlan struct to resolve local references into a sequence of batches for publication:

/// State for tracking dependencies during upload.
pub struct PublishPlan {
/// Graph of publishable packages where the edges are `(dependency -DependencyOf->) dependent)`
dependents: DependencyGraph<PackageSpec>,
// TODO look at using cargo's `InternedString` type for `PackageRef`:
// https://docs.rs/cargo/latest/cargo/util/interning/struct.InternedString.html
indices: LocalPackageIndex,
}

Reference implementation (cargo multi-package publishing):

Comment thread crates/wasm-pkg-client/src/lib.rs

@vados-cosmonic vados-cosmonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey I took a pass through, a lot of NITs but it looks mostly good to me!

One thing I'd love to have is a few more tests... This is a pretty substantial feature, so figured there would be more edges/fixtures. That said, we can certainly add tests down the line if we run into anything.

Comment thread crates/wasm-pkg-client/src/lib.rs
Comment thread crates/wasm-pkg-client/src/local.rs Outdated
Comment thread crates/wasm-pkg-client/src/local.rs Outdated
Comment thread crates/wasm-pkg-common/src/package.rs Outdated
Comment thread crates/wasm-pkg-common/src/registry.rs Outdated
Comment thread crates/wkg/src/main.rs
// 2. Publish our packages in "waves" to the actual registries ensuring all
// possible dependency free pacakges are published in the same group
while !plan.is_empty() {
let ready_for_publish = plan.take_ready();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure this will always terminate? Should we add a check just in case? We're not allowed to have cycles AFAIK so maybe I'm being a bit too defensive here.

@mkatychev mkatychev Jun 25, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're totally right, cargo's take_ready bakes in a timeout.
EDIT: forgot to post link:
https://github.com/rust-lang/cargo/blob/a595d0da21f228b7fdae64d3d5c0e527ea66bb59/src/cargo/ops/registry/publish.rs#L289-L294

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh Ok great! but it's not like... async? I think I actually have more questions now -- I was wondering if it's possible for ready_for_publish to not have anything in it and the plan to not be empty, essentially.

Not looking for a proof or anything of course, but basically... could the graph be misconfigured in a way that nodes are kinda hanging out but don't end up looking ready? The answer could easily be "absolutely not" and then this isn't an issue, but a small invariant check (i.e. "if plan is empty, take_ready will always produce at least one thing to do on each iteration") would be nice

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's what I've reasoned myself into:

  • ready_for_publish is guaranteed to be nonempty IF !plan.is_empty() and
    • a DependencyGraph (petgraph::Acyclic) will produce AcyclicEdgeError errors during try_update_edge for any "danging graphs"1
  • a node is determined to be ready when it has zero Incoming edges2:
    self.dependents
    .neighbors_directed(*id, Direction::Incoming)
    .count()
    == 0

The one area we don't have guarantees is if PackagePublisher::publish will ever terminate.

Footnotes

  1. https://github.com/bytecodealliance/wasm-pkg-tools/blob/8555d1bbee9ac20290b5516437f35346add95be0/crates/wasm-pkg-core/src/wit.rs#L216-L227

  2. https://docs.rs/petgraph/latest/src/petgraph/graph_impl/mod.rs.html#913

@vados-cosmonic vados-cosmonic Jun 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO @mkatychev this would make for a great comment close in the code!

Comment thread crates/wkg/src/main.rs
Comment thread crates/wkg/src/main.rs Outdated
Comment thread crates/wkg/src/main.rs Outdated
Comment thread crates/wkg/tests/e2e.rs Outdated
@mkatychev

This comment was marked as resolved.

mkatychev and others added 3 commits June 24, 2026 22:14
Co-authored-by: Victor Adossi <123968127+vados-cosmonic@users.noreply.github.com>
Co-authored-by: Victor Adossi <123968127+vados-cosmonic@users.noreply.github.com>
@vados-cosmonic

This comment was marked as resolved.

@mkatychev

This comment was marked as resolved.

Comment thread crates/wkg/src/main.rs
@mkatychev mkatychev requested a review from vados-cosmonic June 25, 2026 21:33
Comment thread crates/wkg/src/main.rs Outdated
Comment thread crates/wasm-pkg-core/src/wit.rs Outdated
Comment thread crates/wasm-pkg-core/src/resolver.rs
@mkatychev

This comment was marked as outdated.

@mkatychev

Copy link
Copy Markdown
Member Author

@vados-cosmonic for PublishPlan, the need for pub is because it is instantiated inside of wkg proper, this is also true for all the methods within and inner types (except for PublishPlan::len but that one makes sense to be exposed):

let mut plan = PublishPlan::from_paths(paths)?;

@mkatychev mkatychev requested a review from vados-cosmonic June 27, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants