node, aggregation: flat 8-raw XMSS prove tail (~4s p95) vs lantern/qlean on 8-validator subnet

[ch4r10t33r]

Context

Follow-up to #899, scoped to the current ansible-devnet topology:

• 8 validators per subnet, flat aggregation only — num_children = 0, no recursive child proofs yet
• Aggregator collects raw gossip XMSS signatures for one att_data and calls rec_xmss_aggregate once
• Goal: lean_pq_sig_aggregated_signatures_building_time_seconds p95 ≤ 1.3 s when aggregating a full 8/8 subnet committee

This issue deliberately excludes recursive-aggregation optimizations (child caches, greedy child selection, etc.) — those paths are not exercised today.

---

Live cross-client snapshot (head ≈ 2280, 2026-05-26)

Aggregator (subnet)	build count	mean build
lantern_0 (sn 2)	423	0.27 s
qlean_4 (sn 3)	1298	1.01 s
zeam_8 (sn 0)	169	1.34 s
ethlambda_8 (sn 1)	4044	1.38 s

On zeam_8 (post-#925 / #933, 0xpartha/zeam:local):

Percentile	Build time
p50	~0.5 s
p75	~2.3 s
p95	~4.0 s
mean	1.34 s

Phase metrics on zeam_8 show xmss_prove ≡ build histogram (169/169 samples, identical sums). Snapshot / prep / commit are all sub-ms. The tail is entirely inside xmss_aggregate → rec_xmss_aggregate.

---

What the zeam code actually does (flat, no children)

Prove path (every non-trivial att_data)

submitAggregateOnInterval → aggregateImpl → aggregateForSlots → aggregateUnlocked
  → computeSingleAggregatedSignature → prepareAggregateAttData → runAggregateAttDataFfi
  → AggregatedSignatureProof.aggregate → xmss.aggregateSignatures → xmss_aggregate (Rust)
  → rec_xmss_aggregate

Key locations:

• pkgs/node/src/forkchoice.zig:2091 — aggregateUnlocked (sequential per-att_data loop)
• pkgs/types/src/block.zig:794 — prepareAggregateAttData
• pkgs/types/src/block.zig:1018 — runAggregateAttDataFfi (starts lean_pq_sig_* timer)
• pkgs/types/src/aggregation.zig:78 — AggregatedSignatureProof.aggregate
• pkgs/xmss/src/aggregation.zig:178 — xmss_aggregate FFI + zeam_xmss_rec_aggregate_prove_seconds
• rust/multisig-glue/src/lib.rs:126 — clones every raw PK + sig before prove

Only fast path today

if (!has_gossip and selected_children.items.len == 1) {
    // SSZ clone lone child — no STARK
}

Not applicable when aggregating raw subnet attestations (has_gossip = true, num_children = 0). Every flat 8-raw build goes through the full STARK pipeline. There is no zeam fast path for N raw + 0 children.

Trivial-input filter (aggregator only)

isAggregatorTrivialInput (forkchoice.zig:3781): skips when num_children = 0 and num_gossip_sigs < min_aggregation_inputs (default 2).

So zeam proves for 2–8 raw sigs, never 0–1. The build-time bimodal distribution (p50 ~0.5 s vs p95 ~4 s) is consistent with rec_xmss_aggregate cost scaling with num_raw, not with recursive depth.

Aggregation trigger timing

Worker runs on slot interval with window {slot-1, slot} (chain.zig:4462-4472). Logs frequently show agg start … gossip_sigs=subnet0=1/8 — the worker often proves before all 8 subnet attestations have arrived, then may prove again on a later trigger with more sigs. That produces a mix of small-N (fast) and large-N (slow) samples in the same histogram.

---

What is NOT the problem (please do not optimize these first)

Previously suggested item	Why irrelevant on current subnet
Child proof deserialize cache	num_children = 0 always — deserialize loop never runs (multisig-glue/src/lib.rs:143-177)
Greedy child selection tuning	No known_payloads / peer child inputs in flat mode
Recursive aggregation parallelism	Not deployed
Rayon thread auto-tune (#925)	Already done — 12 Rayon threads on 16 vCPU aggregators
ThinLTO / multisig-release	Production aggregators already build via -Dprover=dummy → multisig-release (build.zig:56, rust/Cargo.toml:34-37)
Parallel per-att_data ThreadPool scope	Explicitly removed in #925 — caused ThreadPool × Rayon oversubscription
Coalesce / in-flight skip rate	Affects slot scheduling, not per-prove STARK cost inside one histogram sample

---

Root-cause hypothesis (code-grounded)

Two independent factors explain the zeam histogram shape:

1. Input size variance: num_raw at snapshot time ranges from 2 to 8 because aggregation fires on the slot tick, not when 8/8 sigs are present. Small-N proves ≈ 0.5 s; full 8-raw ≈ 2–4 s.

2. Prove cost gap vs lantern: For the same flat multisig construction, lantern mean 0.27 s vs zeam 1.34 s on live devnet suggests zeam pays substantial overhead beyond leanMultisig's core prove — or lantern uses a lighter aggregation construction. Need direct comparison (see investigation plan).

Known zeam per-prove overhead in code:

• Rust: deep-clone all 8 XmssPublicKey + XmssSignature on every FFI call (multisig-glue/src/lib.rs:130-135)
• Zig prep: fromBytes + handle materialization for every gossip sig every prove (block.zig:847-866)
• Timer scope: lean_pq_sig_* includes bitfield merge + prove + serialize; live data shows prove dominates (identical to zeam_xmss_rec_aggregate_prove_seconds)

---

Investigation plan

1. Instrument num_raw on the build histogram (required first)

Add a label (or companion histogram) on prove entry:

lean_pq_sig_aggregated_signatures_building_time_seconds{num_raw="N"}

Without this, p95 tuning is blind — we cannot tell whether the tail is "8-raw steady state" or "partial input + reprove".

2. lean-bench parity on aggregator hardware

Run lean-bench aggregate.flat_{2,4,8}_r2 (referenced in pkgs/metrics/src/lib.zig:937) on the same host class as zeam_8 and compare to zeam_xmss_rec_aggregate_prove_seconds.

Outcome	Next step
Bench flat_8 ≈ zeam p95 (~4 s)	Problem is leanMultisig prove scaling — upstream perf or different security params (LOG_INV_RATE_PROD = 2, aggregation.zig:132)
Bench flat_8 ≈ lantern mean (~0.3 s)	Problem is zeam FFI / wrapper overhead — profile clones + serialize
Bench flat_8 ≈ 1.3 s	Problem is partial-input mix — fix aggregation timing (item 3)

3. Aggregation timing policy (if tail = partial inputs)

Options to evaluate once num_raw is labeled:

• Defer prove until subnet gossip coverage ≥ K/8 (e.g. 6/8 or 8/8) within the slot window
• Or: prove early with partial input for fork-choice weight, but do not count partial proves toward the p95 SLO for published aggregates

Code touchpoints: submitAggregateOnInterval / aggregateImpl slot window (chain.zig:4462), pruneTrivialFromAggregateSnapshot threshold (forkchoice.zig:3913).

4. Cross-client construction comparison (lantern / qlean)

lantern mean 0.27 s is the benchmark to beat. Questions for lantern/qlean teams:

• Same rec_xmss_aggregate / leanMultisig STARK at LOG_INV_RATE=2?
• Same metric semantics on lean_pq_sig_aggregated_signatures_building_time_seconds?
• Do they wait for full committee before proving?

zeam has no lantern/qlean source in-tree; this comparison is external but essential — ethlambda (1.38 s) is no longer the right reference point.

5. FFI clone reduction (if bench shows overhead gap)

If lean-bench flat_8 is fast but zeam is slow with identical num_raw:

• Avoid deep-cloning PKs/sigs in multisig-glue when handles are prove-duration-borrowed
• Reuse Zig-side fromBytes handles across reproves of the same snapshot sigs

---

Acceptance criteria

• num_raw (and ideally num_children) exposed on aggregate build metrics
• lean-bench flat_8 vs zeam_xmss_rec_aggregate_prove_seconds documented on aggregator hardware
• Root cause classified: prove scaling vs FFI overhead vs partial-input timing
• With 8/8 raw input (steady state, no children): p95 build ≤ 1.3 s on zeam aggregator
• Mean build within 1.5× of lantern on the same subnet / hardware class (currently ~5×)

---

Related

• node, aggregation: zeam aggregate-build is 5–7x slower than ethlambda on devnet (2–3s vs 0.4s) #899 — original cross-client gap (zeam/ethlambda parity largely achieved on mean; tail remains)
• node, aggregation: PR #902 metrics reveal real FFI worker cost is ~30x the standard build histogram, and ~50% of slot triggers are skipped as in_flight #907 — single-raw STARK cost finding; flat mode still hits full prove for N≥2 raw
• node, types, xmss: keep aggregate worker under 4s slot budget (#907) #925 — Rayon auto-tune + sequential proves (orchestration fixed; prove cost unchanged)

[ch4r10t33r]

Correction / superseded recommendations from prior #899 thread analysis

Reviewing my earlier suggestions on #899 against the live measurements + code in this issue — most are already in place or do not apply on the current flat-aggregation devnet. Recording corrections so the investigation plan above stays the canonical next-step list.

Prior suggestion	Status	Reason
Verify Docker uses --profile multisig-release (ThinLTO)	Already done	build.zig:56 selects multisig-release via -Dprover=dummy; rust/Cargo.toml:34-37 has lto = "thin", codegen-units = 1 for that profile. Aggregator images already build with ThinLTO.
Add phase3 commit + per-input-size metrics	Half done	Phase split exists (snapshot/att_data_prep/xmss_prove/compute_ffi/commit on zeam_pq_sig_aggregated_signatures_building_phase_seconds, pkgs/metrics/src/lib.zig:93). Still missing: num_raw / num_children attribution on the prove histogram — exactly item 1 of the plan above.
Pre-warm child proofs between slots / streaming child cache	N/A on current topology	num_children = 0 on every flat 8-validator subnet build (rust/multisig-glue/src/lib.rs:143-177 deserialize loop never executes). No recursive path to optimise yet.
mimalloc global allocator, panic = "abort", lto = "fat", PGO, -Ctarget-cpu=native	Defer until classified	All are constant-factor wins on the STARK prove kernel. Without num_raw attribution we cannot tell whether the ~4 s p95 is "8-raw steady state needs faster prove" (where these help) or "partial-input reprove" (where the fix is the aggregation timing policy in item 3 of the plan). Item 1 + item 2 (lean-bench parity) must run first or the A/B is uninterpretable.
Bump Rayon to 15-of-16	Defer	Same reason — unknown whether prove kernel is the bottleneck for the tail samples vs partial-input mix.
Pipeline phase 1 of att_data[i+1] with tail of att_data[i]	Likely premature	Phase metrics show snapshot / prep / commit are sub-ms; only xmss_prove has measurable cost. No idle Rayon time to overlap into. Revisit only if recursive aggregation re-enables and child setup grows.

What's actually next (per investigation plan above)

Starting work in this order:

1. Instrument num_raw + num_children on the prove path (companion labeled histogram on zeam_xmss_rec_aggregate_prove_* — leaves the spec'd lean_pq_sig_aggregated_signatures_building_time_seconds unchanged so cross-client dashboards keep working). PR to follow.
2. Once deployed: classify the p95 tail samples against the three branches in the investigation-plan decision table (bench=p95 → prove scaling; bench=lantern-mean → FFI overhead; bench≈1.3 s → partial-input timing).
3. Drive items 3 / 4 / 5 of the plan based on the classification, not before.

[ch4r10t33r]

Deferred-aggregation deploy: measured on 2026-05-27 devnet

PR #941 deployed to 0xpartha/zeam:local (commit 7931e696) and scraped on zeam_8 (16 vCPU Hetzner, log_inv_rate=2).

Headline comparison

Metric	Pre-PR baseline	Prune-only (cf336cf3)	Deferred (7931e696)
Samples	78	86	84
Head slot	53	61	56
Mean prove time	1830 ms	2320 ms	1638 ms
stark phase mean	1830 ms	2320 ms	1638 ms
marshal phase mean	0.4 ms	0.5 ms	0.28 ms
Recursive share	47%	60%	39%
Flat share	53%	40%	61%

Mean down ~30% vs the prune-only run (which had been an unlucky chain shape). Recursive share dropped from 60% → 39%. Flat-only proves now dominate work count and individual flat means match the lean-bench extrapolation (~426 ms mean across all flat buckets, ~3.6 ms/sig).

Per-bucket breakdown

Flat (num_children = 0), 51 of 84 proves:

num_raw	count	mean (ms)
2	3	271
5	4	359
6	2	343
7	4	355
8	8	405
9–15	8	401
16–31	21	509

The flat path is healthy. phase="stark" matches prove_seconds exactly (1638 ms total / 1638 ms stark), confirming wrapper overhead is sub-ms.

Recursive (num_children > 0), 33 of 84 proves:

Bucket	count	mean (ms)
1/1	4	1485
5/3-4	3	3942
9-15/2	2	2649
9-15/3-4	4	3411
16-31/2	4	2793
16-31/3-4	13	4762
(others)	3	1485–2780

The 16-31/3-4 bucket alone is 15% of proves but ~45% of total prove time (62 s / 137.5 s). These are the cases where our local node has raws for ~16 validators and known_payloads contains 3–4 peer-published aggregates covering different validators we never received raws for. Greedy correctly picks all four because each adds coverage raws don't have, and the resulting STARK is genuinely a multi-child merge.

Verdict on hypothesis

Confirmed for the dominant pre-deploy bottleneck (eager-delete-raws → forced recursive merge of own freshly-stored aggregate). The deferred-aggregation flow eliminated that class. Mean dropped, flat share grew, and flat-only proves now match bench cost.

Incomplete for the cross-aggregator coordination case. ~40% of proves still take the recursive path because peers' gossipped aggregates cover validators we don't have local raws for, and merging them is genuinely required to publish a unified proof per att_data. That's a different root cause from #899 / #940 's original framing.

Next step: cap children per call (follow-up PR)

PR coming on branch perf/issue-940-cap-children (builds on top of #941):

• New CLI knob --max-aggregation-children (default 0) caps the number of peer children passed to rec_xmss_aggregate per call. Excess children are dropped lowest-coverage first.
• With default 0, the worker is flat-only. Peer aggregates remain in latest_known_aggregated_payloads for the block proposer's compactAttestations step at proposal time — information is still on the wire, just not re-published by us.
• Predicted effect (extrapolated from current data): drops the 16-31/3-4 bucket entirely, mean drops to ~426 ms (the flat-only mean), p95 drops well under 1 s.
• Operators who would rather publish maximally-covering aggregates per tick can raise to 1 (allows one peer child to be merged, ~1.5 s per call) or higher.

Acceptance criteria status

• Phase split (marshal / stark / post) exposed on zeam_xmss_rec_aggregate_phase_seconds
• Input-size attribution exposed on zeam_xmss_rec_aggregate_prove_by_input_seconds{num_raw, num_children}
• Root cause classified — eager-delete-raws structural lifecycle bug, fixed
• 8-validator flat steady-state: p95 build ≤ 1.3 s achievable today (current flat 16-31 mean 509 ms, p95 likely <800 ms on its own)
• Cross-aggregator merge tail: still ~4.5 s for */3-4 cases; the follow-up cap PR closes this acceptance criterion
• Mean within 1.5× of lantern on same subnet/hardware — pending re-run of cross-client comparison after cap PR lands