Cache proof receipts across the proving pipeline#83
Open
biryukovmaxim wants to merge 4 commits into
Open
Conversation
87bfcd1 to
840074e
Compare
0e8f2fa to
7436ff7
Compare
78f8553 to
63d3059
Compare
Relocate ScheduledBundle and BundleBlocks from the aggregate-prover crate to the scheduler crate, so the bundle handle sits next to ScheduledBatch and ScheduledTransaction. The handle is now generic over its artifact type A (opaque, like ScheduledBatch's P::BatchArtifact) rather than the receipt, which keeps the scheduler free of settlement/proof-system structs and avoids a scheduler -> aggregate-prover dependency cycle. SettlementArtifact stays in aggregate-prover; call sites move from ScheduledBundle<Receipt> to ScheduledBundle<SettlementArtifact<Receipt>>. The scheduler gains kaspa-hashes and vprogs-l1-types for the bundle's metadata fields, and the settler crate gains a scheduler dependency. Neutralize the moved bundle docs so the scheduler crate no longer names downstream crates (aggregate prover, settler, proof-receipt store), matching ScheduledBatch's generic voice.
Wire the proof-receipt store into the transaction, batch, and aggregate provers so a replay (including a flip reorg back onto the same fork) reuses a cached receipt instead of re-proving. - Processor gains tx/batch/aggregator image-id accessors and an AggregatorArtifact type; artifacts are bounded Clone + Borsh so the scheduler can cache and restore them. - ScheduledTransaction and ScheduledBatch read/write their per-tx and per-batch receipts through the storage workers; ScheduledBundle owns the aggregate (settlement) receipt, keyed off its start coordinate with a batch as the storage gateway. - Receipts are keyed by (checkpoint, block_hash, image_id, ...) so competing forks stay distinct; the pruning worker drops a checkpoint's receipts across all forks and programs in one prefix scan. - The proving workers prove on a cache miss, wait for the receipt to be durable, then publish the artifact; the transaction worker proves inline so only one proof occupies the GPU at a time. ChainBlockMetadata supplies the block hash; a test-utils-gated BatchMetadata for u64 stands in for tests. Round-trips the tx/batch/agg receipt paths through the storage workers in a new scheduler test.
Return ScheduledBundle (and BundleBlocks) to the aggregate-prover crate so the scheduler stays bundle-agnostic: it owns the per-tx/per-batch/ aggregate receipt store but knows nothing about how bundles are formed. ScheduledBatch gains concrete read_agg_receipt / write_agg_receipt gateway methods that take a bare start coordinate and build the AggregatorKey internally, so the bundle forwards its own coordinate rather than reaching into the receipt store. The generic submit_*_receipt helpers and the ReceiptLookup trait stay private, keeping that machinery off the scheduler's public surface. Drop kaspa-hashes and vprogs-l1-types from the scheduler and the scheduler dependency from the settler (both only pulled in for the bundle), move the receipt-cache round-trip test alongside the bundle, and scrub "bundle" from the scheduler's receipt-store docs.
read_agg_receipt/write_agg_receipt/agg_key took from_block and seq_commit as adjacent [u8; 32] positional args, which a call site could transpose silently. Replace the trio with a named AggReceiptCoord struct so the fields disambiguate at every call site.
63d3059 to
8182f73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires the proof-receipt store into the transaction, batch, and aggregate provers so a replay — including a flip reorg back onto the same fork — reuses a cached receipt instead of re-proving. Built on top of
proving-pipeline-agg; this PR is just that delta.Why
The proof-receipt store (keyed by semantic proving coordinates) already exists, but nothing read or wrote it. On a reorg the provers re-collected SMT proofs / lane proofs and re-proved work they had already done. This connects the cache to the pipeline so identical coordinates resolve to the stored receipt.
What
Processorgainstx_image_id/batch_image_id/aggregator_image_idaccessors and anAggregatorArtifactassociated type. The tx/batch/aggregator artifacts are now boundedClone + Borsh{Serialize,Deserialize}so the scheduler can serialize them into the store and hand a cached value back out.ScheduledTransaction↔ per-tx receiptScheduledBatch↔ per-batch receiptScheduledBundle↔ aggregate (settlement) receipt — keyed off the bundle's own start coordinate, with its first batch as the storage gateway (the aggregate prover holds no store of its own). The methods are generic overS/P, so the publishedScheduledBundle<A>handle stays free of storage generics and the settler queue type is untouched.Prefix, followed byblock_hash,image_id, and (per kind)merge_idx/seq_commit. The same checkpoint on two competing chains yields distinct keys, yet both share the prefix.ChainBlockMetadatasupplies the block hash viaBatchMetadata::block_hash; atest-utils-gatedBatchMetadata for u64stands in for tests.TransactionEffectsderivesClone + Default + Borsh.Tests
New
scheduling/scheduler/tests/proof_receipt_cache.rsround-trips the tx/batch/aggregate receipt paths through the storage workers (miss → store → hit), covering the cache-hit path the dev-mode sims don't reach../cargo-check-all.sh(clippy-D warningsover the workspace + excluded crates) is clean. Real-proof verification on the GPU box: thecudatest-suitee2e and thesimreorg tests (l2_flow_*_reorgs_*, which exercise the cache-hit replay path) run with--features cudaandRISC0_DEV_MODEoff.