Skip to content

perf(store): precompute account state forest mutations during block application#2341

Open
kkovaacs wants to merge 17 commits into
nextfrom
krisztian/upsert-accounts-optimization
Open

perf(store): precompute account state forest mutations during block application#2341
kkovaacs wants to merge 17 commits into
nextfrom
krisztian/upsert-accounts-optimization

Conversation

@kkovaacs

Copy link
Copy Markdown
Contributor

Summary

Closes #2111.

Precompute account-state forest mutations before applying a block to SQLite. This allows the database layer to reuse the vault and storage-map roots produced by the forest instead of reconstructing the same SMTs from persisted account data.

Changes

Store

  • Add a two-phase account-state forest update flow:
    • compute forest mutations and resulting account roots without changing the forest;
    • pass the precomputed roots into the SQLite transaction;
    • apply the prepared mutations after the transaction commits.
  • Use precomputed vault and storage-map roots for partial and non-genesis full-state public account updates.
  • Continue persisting raw vault assets and storage-map values in SQLite.
  • Validate the resulting account commitment against the block update.
  • Fetch the latest account row once per update instead of repeating the query.
  • Index computed forest roots by lineage to avoid repeated scans.
  • Use the same compute/apply machinery for startup forest reconstruction and test helpers.
  • Preserve storage-map Create replacement semantics, including empty maps.
  • Add structured errors for forest preparation and rebuild failures.

Commit consistency

  • Hold the canonical state and account-state forest write locks across database commit and state publication.
  • Apply forest, nullifier-tree, and account-tree updates before advancing the in-memory blockchain tip.
  • Treat any persistent-tree failure after SQLite commits as fatal because the block cannot be safely retried.
  • Add failure-injection tests covering every post-commit stage and successful publication ordering.

Benchmarks and stress testing

  • Add a Criterion benchmark tracking storage-map recreation cost at increasing map sizes.
  • Extend seed-store to:
    • create an exact requested number of accounts;
    • populate deterministic large storage maps;
    • generate dedicated partial account-update blocks;
    • seed oversized public accounts at genesis when their full update would exceed the protocol limit.
  • Report account creation, note emission, and account update blocks separately.
  • Add end-to-end coverage for large-map creation and partial updates.

Benchmark results show that a single update to a storage map with 500k entries now takes about 50ms.

Changelog

[[entry]]
scope       = "node"
impact      = "changed"
description = "Applying blocks with changes to accounts having large storage maps is now significantly faster."

@kkovaacs
kkovaacs marked this pull request as ready for review July 15, 2026 14:08
// The DB is now committed. Keep both write locks held while advancing the forest and
// canonical in-memory state so readers cannot observe different block heights.
let InnerState { nullifier_tree, blockchain, account_tree } = inner;
Self::finalize_committed_state(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this fn name and complete_post_commit_update could maybe indicate failure is fatal. Ignore if you can't think of better fn names.

) where
ForestError: Display,
NullifierError: Display,
AccountError: Display,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is quite awkward (taking closures, Display trait bound, the PostCommiteStateComponent enum) and AFAICT its really only to support the tests. Is there another way we could factor this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: avoid duplicate SMT work in State::apply_block by precomputing account state forest mutations

2 participants