Add simdjson chunked parse + domain rewards optimization#77
Merged
Conversation
- Introduced `domain_service_daily_rewards` summary table and refresh function to process daily relay and reward aggregates by domain and service. - Added functions to fetch supplier and reward statistics grouped by domain. - Implemented performance indexes to optimize query execution for claims, blocks, and supplier service configurations. - Updated GraphQL schema to include extracted domains and remove unused Relay entity and references. - Extended block processing to compute and store domain-based reward summaries.
- Introduced `createModToAcctTransfersTableFn` to define and manage the `mod_to_acct_transfers` table outside of SubQuery, removing auto-generated GiST indexes and enabling selective index addition. - Added a `rawBulkInsert` utility for optimized batched inserts into non-SubQuery-managed tables. - Replaced GraphQL `ModToAcctTransfer` entity with direct SQL bulk insert. - Refactored `_handleEventClaimSettled` to use `bulkInsertModToAcctTransfers` for improved idempotency and batch handling. - Updated associated imports, dependencies, and schema files for consistency.
- Add vendor/simdjson submodule (fork: jorgecuesta/simdjson_nodejs, branch: buffer-support) with Buffer support and findChunkBoundaries C++ state machine - Add simdjson to vendor-config.yaml and root package.json - Update subql-cosmos with simdjson-based HttpClient.ts - Add benchmarks for JSON parse strategies (benchmarks/json-parse/) - Production verified: 40s parse for 1.5GB/6.7M events (stream-json was crashing)
Alann27
approved these changes
Mar 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
simdjson Details
vendor/simdjsonsubmodule (fork of simdjson_nodejs with Buffer support +findChunkBoundaries)block_results@679773 total=199s (headers=77s body=80s concat=2s parse=40s) size=1542.8MBTest plan
benchmarks/json-parse/