fix(hermes): share VAA bytes across a slot's proofs via Arc#3795
Open
sunoj wants to merge 1 commit into
Open
Conversation
`construct_message_states_proofs` cloned the full VAA (~1-2KB) into the `WormholeMerkleMessageProof` of every message in a slot. Since a slot contains all price feeds (~6000), the same VAA was duplicated ~6000x per slot and then retained `cache-size-slots` deep in the message cache. With the default `cache-size-slots = 1600` this dominated memory: a Hermes instance subscribed to all Pythnet feeds grew to ~20GB of resident memory (and kept climbing). The VAA bytes accounted for the bulk of it. Store the VAA behind an `Arc<VaaBytes>` so every proof in a slot shares a single allocation (refcount bump instead of a deep copy). The bytes are cloned once when building update data for an API response, which is a read-time/per-request cost rather than a stored, per-feed cost. Measured on a node subscribed to all mainnet feeds (cache-size-slots=1600): resident memory plateaued at ~6.5GB instead of climbing past 20GB.
|
Someone is attempting to deploy a commit to the Pyth Network Team on Vercel. A member of the Team first needs to authorize it. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🚩 Version bump not included for hermes server
Per REVIEW.md guidelines on version bumps: the hermes server crate (apps/hermes/server/Cargo.toml) version is 0.12.0 and was not bumped in this PR. Since this is an internal memory optimization with no public API change and no behavioral change, a version bump is arguably not required — but the reviewer should confirm whether the project's release process expects a bump for internal-only changes.
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
construct_message_states_proofsclones the full VAA (~1–2 KB) into theWormholeMerkleMessageProofof every message in a slot. A slot contains all price feeds, so the same VAA is duplicated across ~6000MessageStates per slot, and each is then retainedcache-size-slotsdeep in the message cache.With the default
cache-size-slots = 1600, a Hermes instance subscribed to all Pythnet mainnet feeds grew to ~20 GB resident and kept climbing until OOM. The duplicated VAA bytes accounted for the bulk of that.Fix
Store the VAA behind
Arc<VaaBytes>so every proof in a slot shares a single allocation (a refcount bump instead of a deep copy). The bytes are cloned once, lazily, when building update data for an API response — a per-request read cost rather than a stored, per-feed cost.No public API or wire-format change;
WormholeMerkleMessageProofonly derivesClone/PartialEq/Debug, all of whichArc<Vec<u8>>satisfies.Measurement
On a node subscribed to all mainnet feeds,
cache-size-slots = 1600:(At
cache-size-slots = 64the patched build plateaus at ~2.4 GB and is stable.)cargo build --releaseand the existingstate::cache/wormhole_merkletests pass.