Skip to content

[wip] bloatnet: reduce word-level compress sampling#21742

Open
AskAlexSharov wants to merge 1 commit into
performancefrom
alex/merge_domain_sample_36
Open

[wip] bloatnet: reduce word-level compress sampling#21742
AskAlexSharov wants to merge 1 commit into
performancefrom
alex/merge_domain_sample_36

Conversation

@AskAlexSharov

Copy link
Copy Markdown
Collaborator

to speedup merge: increase sampling of DomainCompressCfg and HistoryCompressCfg
@sudeepdino008 do you thing we must go this way? Need cherry-pick some bug-fix?

@sudeepdino008

sudeepdino008 commented Jun 11, 2026

Copy link
Copy Markdown
Member

tldr:


  • So this PR "works for bloatnet, but not for mainnet"; -- I'd suggest to go ahead with it on bloatnet, maybe put it in "bloatnet specific args".
  • the algorithm change - [wip] seg: 2-3x faster domain merge compression via Aho-Corasick matcher and cover-DP optimizations #21625 -- saw 2x speedup on bloatnet -- it worked in my testing (byte identical etc.) -- but I don't want to push this version in main atleast; in performance -- maybe we can do it (upto you)
  • acceptable for mainnet too but only with the bug fix.
  • I don't have stats for history merge...but definitely will be faster.

@AskAlexSharov

Copy link
Copy Markdown
Collaborator Author

@sudeepdino008 seems you saw on mainnet +13% size on SF=4. Right? (not +43%)

@sudeepdino008

Copy link
Copy Markdown
Member

@sudeepdino008 seems you saw on mainnet +13% size on SF=4. Right? (not +43%)

with the #21628 fix, it was 13%, without the fix 43%.
I'll add the fix into performance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants