feat(l1): add benchmark-grade custom block workloads#71
Open
ilitteri wants to merge 2 commits into
Open
Conversation
Extend `custom block`/`custom batch` from ETH-transfers-only into a synthetic workload generator for benchmarking L1 execution and zkVMs. Workloads (--tx): eth-transfer, erc20-transfer, uniswap-v2-swap, contract-deploy, keccak, mulmod, ecrecover, sstore-fresh, sload-cold. Each produced block contains only the requested workload's transactions: contract provisioning (deploys, mints, liquidity) runs at build time on a scratch chain and is folded into genesis, never into produced blocks. Generation is deterministic (--seed), distributed across --n-senders seed-derived genesis-funded accounts, and supports --gas-target block filling, --tx-gas/--deploy-code-size/--prestate-slots knobs, cache persistence with --cached re-run, and the existing --no-zkvm/--repeat profiling path. Compute and state workloads share one EVM bytecode loop template (EEST JumpLoop shape with a gas-bounded clean exit); the Uniswap-V2-style AMM is compiled from committed source with solc 0.8.x. This also fixes the prior erc20-transfer panic and the multi-block batch nonce-reuse bug (sender nonces and slot cursors now thread across blocks). Bumps the ethrex dependencies (ziskos 0.15.0 -> 0.16.1) and adapts to the new ethrex APIs: the Crypto trait is threaded explicitly through the VM/witness layer (code_hash, Code::from_bytecode, tx.sender, get_transactions_with_sender, Evm::new_for_l1/l2, prepare_block, execute_block), witness conversion is two-step (into_execution_witness + GuestProgramState::from_witness), Holesky was removed, and EMPTY_KECCACK_HASH was renamed. The bump is required because the older ziskos pinned ZisK patch repositories that have been deleted upstream. See docs/custom_blocks.md.
Benchmark every custom L1 workload in the ZisK zkVM (execution / step counts, no proving) across 1/10/100/1000 transactions per block, run on a 32-core CPU host with ZisK 0.16.1. The headline metric is ZisK steps per gas, which spans ~130x across workloads (contract-deploy 0.89 to mulmod 116.9): gas mis-prices zkVM cost in both directions, with state growth cheap in steps and 256-bit arithmetic expensive. Full tables, methodology and the raw CSV are included.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR upgrades ethrex-replay custom into a deterministic, benchmark-oriented L1 synthetic workload generator (pure-workload blocks with build-time provisioning folded into genesis), while migrating the replay/witness pipeline to the newer ethrex APIs (notably the explicit Crypto plumbing).
Changes:
- Added a new
workloadsmodule implementing multiple benchmark workloads (transfers, ERC20, Uniswap V2-style swaps, deploys, and compute/state loop templates) with deterministic seed-driven generation and batch-safe nonce/state threading. - Reworked custom L1 block/batch generation to build blocks in-process, fold setup transactions into genesis, support
--gas-target, persist/reload caches, and ensure produced blocks contain only workload transactions. - Migrated witness conversion/execution code to new ethrex APIs (
decode_witness_headers,into_execution_witness,GuestProgramState::from_witness, explicitNativeCryptousage), and added end-to-end tests + documentation/benchmark artifacts.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
tests/custom_workloads.rs |
New E2E tests validating workload purity, determinism, caching, gas targeting, and sender distribution. |
src/workloads/mod.rs |
Core workload enum + deterministic generation context (GenCtx) and EIP-1559 tx builder helper. |
src/workloads/eth_transfer.rs |
Baseline ETH transfer workload implementation. |
src/workloads/erc20.rs |
ERC20 deploy/mint provisioning (folded) + per-block transfer tx generation. |
src/workloads/uniswap.rs |
Uniswap V2-min style provisioning (folded) + per-block router swap tx generation. |
src/workloads/loops.rs |
Loop-template workloads + contract-deploy workload wiring. |
src/workloads/bytecode.rs |
Minimal bytecode “assembler” + runtime/initcode builders for compute/state loop workloads. |
src/tx_builder.rs |
Removed legacy tx builder (superseded by workloads). |
src/cli.rs |
Major custom L1 generation rewrite: sizing (--n-txs/--gas-target), caching, folding setup into genesis, mempool draining, and new workload options. |
src/cache.rs |
Added Cache::write_named to support deterministic custom cache filenames. |
src/run.rs |
Updated witness decode/convert and guest state construction to new ethrex APIs + explicit NativeCrypto. |
src/rpc/mod.rs |
Updated constant rename to EMPTY_KECCAK_HASH. |
src/rpc/db.rs |
Updated sender recovery/code hash/code construction and LEVM calls to pass NativeCrypto. |
src/lib.rs |
Exported cache and introduced public workloads module. |
src/fetcher.rs |
Adapted to updated get_block_number() return type. |
Cargo.toml |
Added ethrex-crypto dependency and tokio dev-dependency features for tests; introduced zisk-build-elf feature. |
README.md |
Updated custom command description and linked docs. |
docs/custom_blocks.md |
New documentation describing workloads, sizing, determinism, caching, and architecture. |
docs/zisk_benchmark.md |
New ZisK execution benchmark report for all workloads. |
scripts/zisk_bench_matrix.sh |
Benchmark runner script to generate the matrix CSV. |
scripts/zisk_bench_results.csv |
Added benchmark result dataset. |
fixtures/contracts/uniswap_v2_min/UniswapV2Min.sol |
Added minimal Uniswap V2-style contracts used for the swap workload. |
Comments suppressed due to low confidence (1)
src/run.rs:140
run_txallocates a freshArcforNativeCryptomultiple times (for the wrapper DB and again per-transaction when constructing the EVM). This adds avoidable heap allocations in a hot path; creating a singleArcand cloning it is enough.
let mut wrapped_db = GuestProgramStateWrapper::new(guest_program_state, Arc::new(NativeCrypto));
#[cfg(feature = "l2")]
let fee_config = FeeConfig::default();
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
417
to
420
| #[derive(Parser)] | ||
| #[command(group(ArgGroup::new("block_size").args(["n_txs", "gas_target"]).multiple(false)))] | ||
| pub struct CustomBlockOptions { | ||
| #[command(flatten)] |
|
|
||
| let mut txs = vec![ | ||
| build_eip1559( | ||
| 0, |
| .map_err(|err| eyre::eyre!("failed to encode freeMint calldata: {err}"))?; | ||
| for (i, sender) in senders.iter().enumerate() { | ||
| // The deployer already consumed nonce 0 with the deploy. | ||
| let nonce = if i == 0 { 1 } else { 0 }; |
| .map_err(|err| eyre::eyre!("failed to encode freeMint calldata: {err}"))?; | ||
| for (i, sender) in senders.iter().enumerate() { | ||
| // The deployer already consumed nonce 0 with the deploy. | ||
| let nonce = if i == 0 { 1 } else { 0 }; |
| Ok(Self { | ||
| chain_id, | ||
| seed, | ||
| next_nonces: vec![0; senders.len()], |
| self.next_nonces[i] = store | ||
| .get_nonce_by_account_address(latest, sender.address()) | ||
| .await? | ||
| .unwrap_or(0); |
| let i = self.next_sender; | ||
| self.next_sender = (self.next_sender + 1) % self.senders.len(); | ||
| let nonce = self.next_nonces[i]; | ||
| self.next_nonces[i] += 1; |
| .map_err(|err| eyre::eyre!("invalid router initcode: {err}"))?; | ||
|
|
||
| // Per-sender nonce tracking for the setup transactions. | ||
| let mut nonces = vec![0u64; senders.len()]; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ethrex-replay customcould only generate blocks of ETH transfers (and--tx erc20-transferpanicked). To benchmark ethrex and the zkVM backends with controlled, reproducible load, we need synthetic blocks that stress execution at different levels — compute, state reads/writes, contract calls, deployments, and realistic DeFi.What changed
Turns
custom block/custom batchinto a benchmark-grade workload generator.Workloads (
--tx):eth-transfer,erc20-transfer,uniswap-v2-swap,contract-deploy,keccak,mulmod,ecrecover,sstore-fresh,sload-cold.Pure blocks invariant: every produced block contains only the requested workload's transactions. Provisioning (token deploys, mints, liquidity) runs at build time on a scratch chain and is folded into genesis — never into a produced block. Contracts we author (the compute/state loop contract, the cold-read prestate) are injected into genesis directly; third-party contracts (ERC20, Uniswap) are provisioned by executing real setup transactions and folding the resulting state, so layouts are correct by construction.
Determinism & sizing:
--seedmakes generation byte-reproducible;--n-sendersdistributes transactions across seed-derived, genesis-funded accounts;--gas-targetfills a block to a gas budget (alternative to--n-txs);--tx-gas,--deploy-code-size,--prestate-slotstune the loop/deploy/state workloads. Runs persist their cache and can be re-run with--cached. The existing--no-zkvm/--repeatprofiling path works on synthetic blocks.Design notes: the compute/state workloads share one EVM bytecode loop template (EEST
JumpLoopshape with a gas-bounded clean exit borrowed from spamoor's gas burner). Theuniswap-v2-swapAMM is a minimal Uniswap-V2-style implementation (exact constant-product math + 0.3% fee, real router → pair → token call graph) compiled from committed source (fixtures/contracts/uniswap_v2_min/UniswapV2Min.sol) with solc 0.8.x; tokens reuse the existingTestTokenfixture.Bug fixes: the
erc20-transferpanic, and multi-block batch nonce reuse (NonceTooLow) — sender nonces and storage-slot cursors now thread across batch blocks; mempool is drained between blocks.ethrex bump (required)
Bumps the ethrex dependencies (ziskos 0.15.0 → 0.16.1) and migrates to the new ethrex APIs. This is required: the previously pinned ziskos depended on ZisK patch repositories that have been deleted upstream, so the ZisK guest could not build. The migration threads the new
Cryptotrait explicitly through the VM/witness layer (code_hash,Code::from_bytecode,tx.sender,get_transactions_with_sender,Evm::new_for_l1/l2,prepare_block,execute_block), splits witness conversion into two steps (into_execution_witness+GuestProgramState::from_witness), drops Holesky, and renamesEMPTY_KECCACK_HASH. Adds theethrex-cryptodependency.ZisK benchmark
docs/zisk_benchmark.mdreports ZisK execution cost for every workload across 1/10/100/1000 txs (run on a 32-core CPU host, ZisK 0.16.1, execution-only — no proving). Headline (ZisK steps per gas, ~130× spread):Gas mis-prices zkVM cost in both directions: state growth is gas-expensive but step-cheap, while 256-bit arithmetic (
mulmod) is gas-cheap but ~130× step-expensive. Raw data inscripts/zisk_bench_results.csv; the runner isscripts/zisk_bench_matrix.sh.How to test
Notes
zisk-build-elffeature; plainziskkeeps the lightweight compile-only stub so CI is unaffected.revinstead ofbranch = "main"so builds are reproducible (cargo installcurrently drifts onto incompatible upstream commits).