Repository navigation
indexer: incremental supply counters, keyset pagination, holders, order book (closes #2129) - #2130
Merged
Merged
Conversation
Supply statistics (closes the indexer part of #2129): - get_all_statistic now resolves each counter (circulating supply, staked, burned, preminted) with an indexed per-statistic latest lookup instead of a window scan over the whole statistics history, making /statistics/coin and /statistics/token/{id} O(1) per counter. The counters themselves were already maintained incrementally and reorg-safe per block height, so no new supply tables are needed; total minted remains derivable as circulating supply + burned. - storage version bumped to 27: existing deployments rescan from genesis, which rebuilds the new caches in the same pass. Keyset pagination for /pool and /transaction: - ml.latest_pool_data_cache holds the newest state of each pool (including its creation height), maintained by the scanner and rolled back on reorgs, so /pool no longer scans the whole per-block pool history; the pledge sort now orders by a numeric column instead of a textual byte comparison. - both endpoints accept an opaque URL-safe cursor (next_cursor field in the response, empty cursor = first page); the offset protocol keeps working. - an index on ml.transactions.tx_global_index backs the transaction cursor. Holders endpoints: - the scanner maintains ml.address_amount (latest balance per address and asset, numeric column, rolled back on reorgs). - new endpoints /statistics/coin/holders and /statistics/token/{id}/holders list holders by balance with cursor pagination. Order book endpoint: - the scanner maintains ml.latest_orders_cache (remaining ask/give balances per order, rolled back on reorgs; concluded orders drop out via zero balances, frozen orders are excluded from reads). - new endpoint /order/pair/{pair}/book?side=ask|bid aggregates remaining balances into price levels (remaining give/ask ratio) with cursor pagination; /order/pair/{pair} stays unchanged.
|
🔍 OpenCodeReview found 11 issue(s) in this PR.
📄
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Addresses the first automated review pass: - items=0 is now rejected by the shared pagination parameter parser: the cursor endpoints fetch items+1 rows and read the last returned row to build the next cursor, so an accepted zero page size made all four cursor endpoints panic remotely (requests return 400 instead). - the in-memory pool listings now filter pools by a non-zero staker balance, matching the postgres implementation (decommissioned is not equivalent: pledge/rewards can be zero while the staker balance is not, and vice versa). - the holders' address and the pool id columns are declared with the C collation so ordering and cursor comparisons are plain byte order, deterministic across database clusters and identical to the in-memory backend's ordering. - the order book price decimal field is computed with exact integer arithmetic (u128 mantissa scaled in a u256) instead of f64; the fraction string is built by the shared decimal formatter. An f64 remains only as an unreachable-in-practice fallback if the exact mantissa overflows u128. - endpoint tests: zero page size is rejected on /pool and /transaction.
Collaborator
Author
OCR cycle log
|
- set-based reorg rollback for the holders table (constant round trips instead of per-row queries) - cap order book aggregation at 10k deepest orders per query - validate the holders cursor tie-break address; guard reduce_rational against zero denominators - exact integer price formatting via 256-bit truncation (no float fallback inconsistency) - tests: items=0 rejected on transaction/pool listings, exact holder page content, baseline truncation assert
- order-book reads return (entries, has_more) with the 10k cap enforced in both backends; the endpoint surfaces truncation as an additive "truncated" field - deterministic tie-breaks everywhere: order ids and pool ids sort by their encoded (bech32) address strings in byte order in both backends (in-memory storage now keeps the chain config for that); explicit COLLATE "C" in the holders ORDER BY - in-memory pool listings sort by the same keys as postgres (creation height then encoded address; pledge then encoded address) - tests assert the raw backend order instead of re-sorting
- encode order-id/pool-id sort keys once per entry in the in-memory backend (decorate-sort-undecorate) instead of once per comparison - the pledge listing decodes the balance once and treats an unparseable balance as absent instead of expecting in a production sort - the order-book handler defensively skips zero-balance entries so a future storage change cannot corrupt the level ordering - cursor-walk endpoint tests are bounded by the known item count
- the holders and pools keyset predicates expand the row comparison so the byte-order (C) collation of the tie-break is explicit rather than relying on the column default; the pool ORDER BYs pin it too - latest_orders_currencies_index now covers the book read's ordering so the pair scan needs no separate sort - the order-book price decimal is rendered exactly (u256 digit extraction) with no floating-point fallback; an impossible scaling overflow becomes an internal server error - the storage test pins the equal-balance tie-break and its cursor boundary
- VARIANTS gets a compile-time exhaustiveness guard so a new statistic variant cannot be silently omitted from the latest-state lookup - the orders-cache reorg restore names its columns instead of SELECT * - a transactions cursor index that does not fit an i64 is rejected as an invalid cursor (client error) before reaching the storage layer - the holders test baseline checks itself against the page limit; the order book handler documents that a walk is not a consistent snapshot of a moving book
- cursors carry an endpoint tag and reject unknown fields, so a cursor minted by one listing can no longer be decoded by another (with a test) - reduce_rational pins its zero-denominator invariant with a debug assertion - is_initialized documents that true does not mean the storage version is current; the caller must compare versions and reinitialize on mismatch
- a pools cursor height that does not fit an i64 is rejected as an invalid cursor (client error) before reaching the storage layer - the in-memory sort-key helpers propagate an encoding error instead of panicking on a mismatched chain config, matching the postgres backend - the decorated sorts compare references instead of cloning string keys
Collaborator
Author
Security review of the PR diffScope: the full diff against Verdict: no new remotely exploitable vulnerabilities found. Basis:
Residual notes (informational, no action required): a paginated walk over a moving book is not a consistent snapshot — documented on the handler; This review covers the diff only; it does not claim coverage of pre-existing indexer code, deployment configuration, or the upstream dependency stack. |
nullPointerEnjoyer
marked this pull request as ready for review
October 3, 2026 01:48
nullPointerEnjoyer
requested review from
ImplOfAnImpl,
anyxem and
erubboli
as code owners
October 3, 2026 01:48
- Canonicalize the address id when minting and decoding holders cursors: Address::from_string keeps the input verbatim and bech32 accepts an all-uppercase encoding, which would corrupt the byte-order keyset comparison. Re-encode via Address::new like the pools listing does. - Extract sorted_live_pools() shared by the offset and keyset pool listings, removing the duplicated decorate/filter/sort block, and document the in-memory backend's full-scan cost characteristics. - Comment the intentional <= conflict guard on the latest_pool_data_cache upsert (same-height reorg replacement). - Document the items=0 -> 400 behavior change across paginated v2 endpoints in the changelog, and that the order book truncated flag is informational (levels beyond the cap are unreachable by cursor walk).
Make the order book cursor tag side-specific so a cursor minted for one side is rejected on the other, guard next-cursor minting against an empty page instead of expecting, log on the zero-denominator price fallback, document the floored decimal price and the ignored transaction cursor tie-break, and use expect with context in tests. The postgres pool cache upsert becomes a monotone envelope update (min creation height, max latest state) so late lower-height writes correct the creation height like the in-memory backend and the rollback path; covered by a new shared pool_data_keyset block and a rollback-coupling note in top_address_amounts.
Cap the encoded cursor length before the base64 decode, filter zero-balance pools before the bech32 sort-key encoding, compute the order book price scale by exponentiation, fail the holders listing with a server error when a stored address cannot be re-encoded canonically, collate the pool listing tie-break explicitly, and document the offset-mode items guard and the coin ticker case-insensitivity. No behavioral change for well-formed requests.
Mirror the holders cursor tuple to the returned row shape (address, amount) across the trait, both backends and call sites, reuse canonical_address_string on the decode side, guard the reduce against a zero gcd, reject empty trading-pair segments with the accurate error, extend the pledge index with the tie-break column, skip empty versioned maps in the in-memory pool listing, and drop unused test bindings. No behavioral change for well-formed requests.
Propagate an undecodable pool staker balance as a storage error instead of silently shrinking the in-memory listing, fold the versioned balance delete and the holders re-derivation into one helper so the ordering dependency is structural, include the failing address/order in balance parse errors, check the creation block height cast, note the DISTINCT ON omission semantics, document pool listing page stability and the canonical address encoding contract, and expect instead of unwrap in the orders test setup.
Share one constant query text between the cursor and no-cursor top-addresses variants so the ordering and the cursor predicate cannot drift apart, include the offending identifier and the underlying error in address-decode failures instead of the bare AddressableError, skip empty versioned maps without expect in the in-memory pledge listing, document the transaction listing ordering contract, and simplify the statistics test bootstrap. No behavioral change for well-formed storage.
Re-encode pool-id cursors through Address::new so keyset boundaries use canonical bech32; document orders-cache rollback invariant; propagate staker-balance decode errors in sorted_live_pools; validate cursor key counts; add context to decode errors.
Reject holders cursors above the max representable amount; make reduce_rational return Option so a zero denominator surfaces as an internal error; validate pools cursor key count; add debug asserts to the transactions pagination bounds; document holders upsert staleness policy; compile-time parameter-count check for the top-holders queries; expect the pools test web server.
Compile-time placeholder checks now read the actual query constants instead of a duplicated literal; balance-write cache upsert rationale documented; in-memory sort-key errors carry context; statistics VARIANTS ordering contract documented; holders cursor doc specifies atom-string encoding; order book aggregation cost documented; test web servers expected consistently.
Cursor decode rejects key vectors above a small constant bound; the order book cursor walk stops (no next_cursor) when the storage cap truncated the aggregation, so a partial book is never continued; the book cursor rejects a non-empty tie-break id; the transactions pagination debug assert checks the fetch bound instead of restating the page condition; scanner statistics tests fail loudly on a missing key; the in-memory id sort keys share one helper.
Move query-text constants before the executable statements in the holders query (items_after_statements is denied on production code); restore the reorg statistics helper to treat a missing statistic as zero, which is the intended meaning after a rollback, with the rationale documented.
erubboli
approved these changes
Oct 5, 2026
nullPointerEnjoyer
added a commit
that referenced
this pull request
Oct 6, 2026
…tion indexer: incremental supply counters, keyset pagination, holders, order book (closes #2129)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2129.
Supply statistics
/statistics/coinand/statistics/token/{id}were slow becauseget_all_statisticresolved each counter with a window scan over the wholeml.statisticshistory (the primary key is(statistic, coin_or_token_id, block_height), which cannot serve a filter oncoin_or_token_idalone). This PR rewrites it as an indexed per-statistic latest lookup (DISTINCT ONover the four counters), making both endpoints point queries.The counters themselves were already maintained incrementally and reorg-safe by the scanner (height-versioned rows +
del_statistics_above_heightrollback), so no new supply tables are added — this deliberately deviates from the issue's sketch ofml.supply/ml.token_supplytables. Total minted stays derivable ascirculating_supply + burned. The storage version is bumped 26 → 27, so existing deployments rescan from genesis, which rebuilds the new caches in the same pass (no separate backfill).Keyset pagination for /pool and /transaction
ml.latest_pool_data_cacheholds the newest state of each pool (creation height included), maintained same-pass by the scanner and rolled back on reorgs;/poolno longer runs aROW_NUMBERwindow over the entire per-block pool history. Side effect: the pledge sort now orders by a numeric column instead of a textual byte comparison (which mis-ordered amounts of different widths)./pooland/transactionaccept an opaque URL-safe cursor (next_cursorin the response; an emptycursor=parameter starts the listing). The offset protocol keeps working unchanged; when both are sent the cursor takes precedence. Atx_global_indexindex backs the transaction cursor.Holders endpoints
The scanner now maintains
ml.address_amount(latest balance per address and asset, numeric column, reorg-rolled-back). New endpoints:GET /api/v2/statistics/coin/holdersGET /api/v2/statistics/token/{token_id}/holdersBoth list holders by balance, descending, with cursor pagination.
Order book endpoint
The scanner now maintains
ml.latest_orders_cache(remaining ask/give balances per order, numeric columns, reorg-rolled-back; concluded orders leave the book via zero balances, frozen orders are excluded from reads). New endpoint:GET /api/v2/order/pair/{pair}/book?side=ask|bidIt aggregates the remaining balances into price levels (remaining
give/askratio, quoted in the quote currency per base unit) with cursor pagination. The existing/order/pair/{pair}endpoint is unchanged.Tests
Deployment note
Version bump → one-time rescan from genesis on upgrade;
postgres_max_connectionsunchanged (the rewrite strictly reduces query load; the new endpoints add only small indexed reads).