Repository navigation
feat(indexer): api-server v2 cursor pagination, holders and order book - #4
Conversation
Track api-server 1.4.1 (mintlayer-core PR #2130) without breaking existing method signatures. New endpoints and types: - Statistics holders listings: ListCoinHolders / ListTokenHolders plus the Holder type (address + amount); token ids are exact case-sensitive bech32 and are path-escaped. - Order book: GetOrderBook / OrderBookPager with OrderBookPage / OrderBookLevel / OrderBookPrice. Required side, side-specific cursors, exact reduced-rational price plus floored decimal rendering, and the truncation invariant (Truncated true => NextCursor nil at the 10000-order cap; the paginator stops there). Cursor pagination: - Shared CursorPage[T] envelope and Pager[T] (NewPager, NextPage, Walk, NextCursor, Truncated). Walk and NextPage share one consumption position, so they can be mixed without duplicating or dropping items; the pager stops on a nil cursor, a truncated page, or a server-issued empty cursor. Pre-1.4.1 bare-array responses are accepted and wrapped. - ListPoolsPage / PoolsPager (cursor only with the default by_height sort), ListTransactionsPage / TransactionsPager (WithOffsetMode, newest block first, BlockID documented as the confirming block hash), both holders pagers, OrderBookPager. List options and validation: - WithItems (validated 1..=100), WithOffset, WithCursor (opaque, never constructed client-side), WithSort (whitelisted), WithOffsetMode, WithSide. Inapplicable options fail client-side with *RequestError naming the option; pagers validate at construction. - Documented semantics: page stability only once the scanner is caught up; a cursor silently overrides the offset position server-side on pools/holders/order book; cursor + non-default sort (pools) and cursor + offset_mode (transactions) are server 400s, propagated. Typed errors: - HTTPError.Message/Kind populated from the server error body; ErrorKind classification status-gated and case-insensitive; ErrInvalidCursor / ErrInvalidNumItems / ErrTokenNotFound sentinels matched via errors.Is. Docs: README, docs/indexer.md (cursor pagination guide, holders and order book reference, error table), CHANGELOG, and the runnable examples/indexer-paging. Table-driven tests against canned fixtures cover full walks, both book sides, truncated and last pages, option validation, and every error shape.
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 10 selected item(s). |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| func TokenHoldersPager(c *Client, tokenID string, opts ...ListOption) *Pager[Holder] { | ||
| p, err := applyListOptions(opts) | ||
| if err == nil { | ||
| err = p.reject("TokenHoldersPager", paramSide|paramSort|paramOffsetMode) | ||
| } | ||
| if err == nil && tokenID == "" { | ||
| err = &RequestError{Option: "tokenID", Reason: "must not be empty"} | ||
| } | ||
| return NewPager(func(ctx context.Context, cursor *string) ([]Holder, *string, bool, error) { |
There was a problem hiding this comment.
If applyListOptions fails, p is nil, but the error is only captured and the function still constructs the pager with a valid fetch closure instead of returning failHolder like the sibling constructors (CoinHoldersPager, PoolsPager, TransactionsPager, OrderBookPager). On first use, NextPage calls the closure with p == nil, and fetchTokenHolders's p.reject dereferences the nil *listParams, panicking. Even when tokenID is empty (err set, p non-nil), the deferred *RequestError contract is silently lost and a doomed network request is issued instead. Return failHolder when err != nil.
Suggestion:
| func TokenHoldersPager(c *Client, tokenID string, opts ...ListOption) *Pager[Holder] { | |
| p, err := applyListOptions(opts) | |
| if err == nil { | |
| err = p.reject("TokenHoldersPager", paramSide|paramSort|paramOffsetMode) | |
| } | |
| if err == nil && tokenID == "" { | |
| err = &RequestError{Option: "tokenID", Reason: "must not be empty"} | |
| } | |
| return NewPager(func(ctx context.Context, cursor *string) ([]Holder, *string, bool, error) { | |
| if err != nil { | |
| return fail[Holder](err) | |
| } | |
| return NewPager(func(ctx context.Context, cursor *string) ([]Holder, *string, bool, error) { |
| if err := p.reject("ListTransactionsPage", paramSide|paramSort); err != nil { | ||
| return nil, nil, false, err | ||
| } | ||
| page, err := getPage[Transaction](ctx, c, "/transaction", transactionsQuery(p)) |
There was a problem hiding this comment.
The documented contract says combining WithCursor with WithOffsetMode is rejected by the server with 400 "Bad request", but fetchTransactions only rejects paramSide|paramOffsetMode — a cursor+offset_mode combination passes client-side validation and incurs a guaranteed round-trip 400. The ListOption layer's stated goal is to surface misuse as *RequestError before network traffic; consider rejecting paramOffsetMode when p.set¶mCursor != 0 (and paramCursor when offset mode is set) here.
Suggestion:
| if err := p.reject("ListTransactionsPage", paramSide|paramSort); err != nil { | |
| return nil, nil, false, err | |
| } | |
| page, err := getPage[Transaction](ctx, c, "/transaction", transactionsQuery(p)) | |
| if err := p.reject("ListTransactionsPage", paramSide|paramSort); err != nil { | |
| return nil, nil, false, err | |
| } | |
| if p.set&(paramCursor|paramOffsetMode) == (paramCursor | paramOffsetMode) { | |
| return nil, nil, false, &RequestError{Option: "WithOffsetMode", Reason: "cannot be combined with WithCursor"} | |
| } | |
| page, err := getPage[Transaction](ctx, c, "/transaction", transactionsQuery(p)) |
- TokenHoldersPager: return fail on constructor error instead of capturing a nil params pointer (nil-deref panic on the first NextPage when an option like WithItems(0) was rejected) - ListTransactionsPage: reject cursor + offset_mode client-side with a RequestError instead of a guaranteed server 400 round trip; docs and changelog updated to match - getPage: trim whitespace before the bare-array sniff so a legacy offset response with leading whitespace decodes as a page instead of a silent empty envelope
| func (p *Pager[T]) NextCursor() *string { | ||
| if !p.started { | ||
| return nil | ||
| } | ||
| return p.cursor | ||
| } |
There was a problem hiding this comment.
NextCursor returns the pager's internal cursor pointer directly. A caller that mutates the pointed-to string (or mutates the *string returned in CursorPage.NextCursor by the *Page methods) silently corrupts the pager's resume position and any value the caller persists, despite the documented guarantee that the value is "never constructed or modified client-side". Return a defensive copy of the value instead.
Suggestion:
| func (p *Pager[T]) NextCursor() *string { | |
| if !p.started { | |
| return nil | |
| } | |
| return p.cursor | |
| } | |
| func (p *Pager[T]) NextCursor() *string { | |
| if !p.started || p.cursor == nil { | |
| return nil | |
| } | |
| v := *p.cursor | |
| return &v | |
| } |
A caller mutating the pointed-to string previously corrupted the pager's resume position; the value is now copied before it leaves the pager.
- every well-known ErrorKind now has an errors.Is sentinel (adds ErrBadRequest, ErrInvalidOffsetMode, ErrInvalidPoolsSortOrder, ErrInvalidTokenID, ErrInvalidOrderPair); docs tables updated - Pager.NextPage documents that the returned slice aliases the pager's internal page buffer and must not be modified
PoolsPager rejects non-default sorts client-side with a RequestError; the server 400 only applies to ListPoolsPage with an explicit cursor.
| func (c *Client) fetchPools(ctx context.Context, p *listParams) ([]Pool, *string, bool, error) { | ||
| if err := p.reject("ListPoolsPage", paramSide|paramOffsetMode); err != nil { | ||
| return nil, nil, false, err | ||
| } | ||
| page, err := getPage[Pool](ctx, c, "/pool", poolsQuery(p)) |
There was a problem hiding this comment.
Validation inconsistency between the two public entry points for the same listing: PoolsPager rejects a non-default sort up front with a *RequestError, but ListPoolsPage/fetchPools perform no equivalent check, so WithCursor combined with WithSort(SortByPledge) reaches the wire and only fails at request time with HTTP 400 "Bad request". This breaks the documented ListOption contract that unsupported combinations "fail the request with a *RequestError before any network traffic". Consider mirroring the pager-side check in fetchPools (e.g. reject paramCursor+paramSort when sort != SortByHeight), or explicitly document that only the pager validates this combination.
Suggestion:
| func (c *Client) fetchPools(ctx context.Context, p *listParams) ([]Pool, *string, bool, error) { | |
| if err := p.reject("ListPoolsPage", paramSide|paramOffsetMode); err != nil { | |
| return nil, nil, false, err | |
| } | |
| page, err := getPage[Pool](ctx, c, "/pool", poolsQuery(p)) | |
| func (c *Client) fetchPools(ctx context.Context, p *listParams) ([]Pool, *string, bool, error) { | |
| if err := p.reject("ListPoolsPage", paramSide|paramOffsetMode); err != nil { | |
| return nil, nil, false, err | |
| } | |
| if p.set¶mCursor != 0 && p.sort != "" && p.sort != SortByHeight { | |
| return nil, nil, false, &RequestError{Option: "WithSort", Reason: "cursor pagination requires the default by_height sort"} | |
| } | |
| page, err := getPage[Pool](ctx, c, "/pool", poolsQuery(p)) |
…example - document that ListPoolsPage propagates the server 400 for cursor + non-default sort (api-server contract) while PoolsPager validates the combination client-side - example walks now return errors and main exits non-zero on failure so the demo doubles as a CI-observable smoke check
The api-server issues next_cursor only alongside items (it fetches items+1 to detect the next page), so an empty page cannot legitimately continue. Treating an empty fetched page as terminal bounds Walk against a non-compliant peer that returns an empty page with a non-null cursor forever. TestPagerNilItemsWithCursor pins the new contract.
1 participant
Summary
Brings the indexer client up to api-server 1.4.1 (mintlayer-core PR #2130, master commit
2684078c546). No breaking changes to existing method signatures — offset-based listings (ListPools,ListTransactions, per-block transaction ids, …) keep working as-is.New endpoints & types
ListCoinHolders/ListTokenHolders+Holdertype (address,amount). Token ids are exact case-sensitive bech32, preserved verbatim and path-escaped; unknown ids surface 404Token not found.GetOrderBook/OrderBookPager+OrderBookPage/OrderBookLevel/OrderBookPrice. Requiredside(WithSide), pair format{base}_{quote}validated client-side (coin ticker matches case-insensitively server-side, token ids exact). Aggregated price levels carry the exact reduced rational plus the floored decimal rendering. The documented truncation invariant is enforced client-side:Truncated == true⇒NextCursor == nil(server capORDER_BOOK_MAX_ORDERS= 10000), and the paginator stops there.GetCoinStatistics/GetTokenStatistics) documented: all four counters (circulating_supply,preminted,burned,staked) are always present, unwritten counters render as"0".Cursor pagination
CursorPage[T]envelope (items/next_cursor) and genericPager[T](NewPager,NextPage,Walk,NextCursor,Truncated).WalkandNextPageshare one consumption position, so they can be mixed on the same pager without duplicating or dropping items. The pager stops on anilcursor, on a truncated page, and normalizes a server-issued empty cursor to end-of-listing. Pre-1.4.1 bare-array responses are accepted and wrapped.ListPoolsPage/PoolsPager,ListTransactionsPage/TransactionsPager, both holders pagers,OrderBookPager.by_heightsort; pagers reject other sorts at construction, single-page methods propagate the server 400.WithOffsetMode("legacy"|"absolute")selects the offset listing (global listing is newest block first, transactions in block order per block); cursor +offset_modeis a server 400.Transaction.BlockIDdocumented as the confirming block's hash (empty for mempool transactions). Per-block listing stays offset-based and unchanged.Options & validation
WithItems(validated 1..=100 =MaxNumItems; the server rejectsitems=0on every paginated v2 endpoint),WithOffset,WithCursor(opaque, non-empty; cursors are never constructed or mutated client-side),WithSort(whitelisted),WithOffsetMode,WithSide. Inapplicable options fail client-side with*RequestErrornaming the option.Typed errors
HTTPErrornow carriesMessageandKindparsed from the server's{"error": ...}body, with status-gated, case-insensitive classification anderrors.Issupport for theErrInvalidCursor/ErrInvalidNumItems/ErrTokenNotFoundsentinels.Tests
Table-driven tests against canned JSON fixtures (
indexer/testdata/): first page → full cursor walk (pools, holders, global transactions), both sides of an order book, a truncated book (iteration stops, no continuation request), last page (next_cursor: null), Walk/NextPage interleavings, empty-cursor and empty-items pages, option validation, and the exact error shapes (400invalid cursor/invalid num items, 404Token not found, mislabeled 5xx ⇒ErrorKindOther).Docs & examples
README indexer section,
docs/indexer.md(cursor pagination guide, holders/order book reference, error table), CHANGELOG[Unreleased]entry, and the runnableexamples/indexer-paging(coin holders walk, both-side order-book walk, global transaction listing with cursor and withoffset_mode, pools walk noting the sort/cursor restriction).Verification
gofmtclean,go build ./...,go vetclean on all touched packages,go test ./...green (28 files changed).Pagerguard, path escaping of pair/token-id input). Two findings were rejected as contrary to the spec'd behavior (client-side 1..=100 clamping; server-400 propagation for the single-page cursor conflicts) — both are explicitly documented. Note:go vet ./wasm/reports two pre-existingunsafe.Pointerwarnings onmaster(unrelated to this PR; addressed by the pending wasm-fix branch).