[kylebernhardy]

⊙ Problem

❓ Your call: The final post-review delta is limited to documentation and removal of a redundant test assertion; production code remains the reviewed implementation. Native continuation is warranted because the published library cannot reach later ranked matches; keep this branch scoped to that capability and its safety qualification. Harper owns the separate authorization and REST integration. The merged Add structured native filters and prepare 0.5.0 is a foundation, with no competing traversal PR identified.

Offset search cannot reach matches after position 10,000. Autocomplete returns at most 100 raw hits, so a host that rejects those hits can miss eligible matches later in the ranking. Add bounded deep-result traversal and qualify autocomplete limits owns this native capability; Support full-text and autocomplete continuation in Table.search and REST owns the separate Harper integration.

💡 Solution

Add searchPage() to index and reader facades for ordinary, prefix, fuzzy-prefix and Boolean queries. Pages use score-descending / UTF-8 ID-ascending order, exact snapshot totals and authenticated continuations. Each page is bounded independently of the total traversable matches. Existing search() windows remain compatible.

⚖️ Alternatives

❓ Your call: Adopted the corrective planning review's threshold hybrid: identities up to 64 KiB retain a bounded synchronous hash; larger identities hash asynchronously with bounded preparation admission. Rejected a new 256 KiB traversal-query cap because legal candidate/filter requests already reach approximately 8 MiB. The threshold selects execution and does not reduce accepted query capacity. Every token boundary is HMAC-protected on both paths.

Ordinary searches reuse their packed request instead of copying it for traversal. One small encoder metadata object remains; the reviewer explicitly accepts that measured-path tradeoff because the local samples do not resolve its cost.

Use an existing dedicated read-only handle rather than a new native cursor registry. The caller owns its lease and closes it. Raising the offset window would still retain winners proportional to depth; ID-ordered export would change ranking. The chosen collector rescans matches each page, approximately N²/P visits for N matches and page size P. Reduce repeated matching work during deep ranked traversal tracks a separately measured optimization.

🔧 Changes

Product and architecture tour

Ranked traversal and snapshot ownership

Keep the reader open and do not reload it between pages. Writer publications and merges can proceed while the reader serves its original indexed snapshot. Reload onto a changed snapshot yields E_CURSOR_EXPIRED; same-snapshot reload preserves continuations. Writer-handle continuation is also supported but publication can expire it. Tokens cannot move to another facade, process or reopened handle. Source records and permissions remain the host's responsibility; this library snapshot does not freeze authoritative source data.

Ranked traversal request flow

Each page uses the retained snapshot and returns an authenticated boundary.

flowchart LR
    Q[Query and continuation] --> F[Facade validation and preparation]
    F --> N[Existing native search entry point]
    N --> A[Expensive-search admission]
    A --> S[Retained reader snapshot]
    S --> C[Bounded search-after collector]
    C --> R[Hits and exact raw total]
    R --> T[Authenticated continuation]

Loading

The traversal collector retains page-size-plus-one candidates per segment and applies the exclusive score/ID boundary. The ordinary stable collector stays independent. Matching-loop deadline checks can stop traversal; individual Tantivy query/scorer operations remain uninterruptible.

Token integrity and bounded preparation

Version-3 tokens authenticate version/handle, snapshot, score, ID and fixed query digest using a 32-byte per-object HMAC key. Snapshot identity is SHA-256 over an explicitly framed, ordered Tantivy segment-ID/deletion-generation map. The engine derives it from the captured searcher for both direct Rust and Node callers, without mutable lease state. Maximum encoded continuations are 5,602 characters; older unreleased token versions reject explicitly. Constant-time comparison rejects finite boundary edits, splicing, query/reader mismatch and old tokens. Page size, exact-total preference and execution budget remain outside query identity as documented. Integrity does not grant authorization.

Large-hash preparation respects configured search queue limits and ceilings derived from the 8 MiB request bound. Reservations release before native submission; remaining time and close state are rechecked after hashing. Preparation-specific E_QUEUE_FULL messages distinguish this stage from native saturation; native status counters exclude it. WebCrypto shares libuv capacity and may copy input, so configured frame bytes are not total RSS guarantees.

Capacity and compatibility

The package prepares 0.6.0 with native ABI 9 / query API 4. Direct Rust SearchRequest literals now require traversal and after; Engine::search enforces the checked 10,000-hit offset window. Rust deep traversal uses traversal: true on the same method and retains the original searcher. The draft token format changes within that unreleased version set; generated addon declarations remain unchanged. No analyzer or source-index migration is introduced. Checked ordinary-window validation is shared by decoding and direct Rust execution. Search responses reject computed non-finite scores before exposing hits/tokens. Existing term/clause, exact-prefix expansion, candidate, byte and execution ceilings remain unchanged; Make query term and clause limits configurable with bounded resource policy and Qualify higher candidate, prefix-expansion and payload capacity profiles track separate policy work.

❓ Your call: Accept the direct Rust source/API changes in 0.6: required traversal fields and the checked offset window. Deeper Rust traversal uses the same engine method with traversal: true; the source break accompanies the 0.5-to-0.6 package change.

Changed-file coverage

File	Meaningful change
.github/workflows/ci.yml	Run the bounded traversal correctness smoke in the existing platform/Node CI matrix.
Cargo.lock	Synchronize the native package version and lock the SHA-256 dependency graph.
Cargo.toml	Prepare native package 0.6.0 and exact-pin RustCrypto sha2 0.10.9 without changing Tantivy.
DESIGN.md	Index the snapshot traversal design note.
README.md	Document continuation usage, lease ownership, costs, compatibility and benchmark commands.
benchmarks/ordinary-search.mjs	Reproduce paired ordinary-search measurements at small and larger result sizes.
benchmarks/traversal-driver.mjs	Compare ordinary and traversal first-page calls on the same snapshot, scored query, exact total and page size with alternating samples.
benchmarks/traversal-preparation.mjs	Measure legal large-query submission, completion and event-loop delay through the public API.
benchmarks/traversal.mjs	Qualify original snapshots, ranked sequences, large payloads, concurrent writes/searches and byte recovery; CI smoke uses the normal native budget.
dependencies.md	Explain the exact-pinned SHA-256 dependency and why snapshot identity belongs in the engine rather than a new stateful Rust API.
docs/limits.md	Document numeric bounds, semantic omissions, correlated resource costs, host duties and scoped observations.
docs/native-backend-implementation.md	Document snapshot traversal, bounded WebCrypto/libuv preparation and traversal checkpoints while preserving explicit ordinary/Tantivy cancellation limits.
package-lock.json	Synchronize package and optional platform artifact versions.
package.json	Prepare JavaScript and optional platform package 0.6.0.
src/DESIGN.md	Record collector/scoring, HMAC token, bounded preparation and lifecycle invariants.
src/engine.rs	Add the separate bounded search-after collector, derive SHA-256 segment/deletion identity, check snapshot expiry before query construction, reuse the alive bitset and enforce finite-score/response validation; test scoring, limits, deadline and snapshot behavior, plus deterministic 1/256-segment qualification.
src/error.rs	Add the explicit cursor expiration error contract.
src/lib.rs	Advertise coordinated ABI/API versions and traversal/filter capabilities.
src/native.rs	Encode traversal responses and route traversal work through expensive-search admission.
src/protocol.rs	Decode bounded 64-hex snapshot boundaries, share checked page/window guards, and permit autocomplete traversal while retaining legacy bounds.
test/error-codes.test.mjs	Check error metadata and shared boundary constants.
test/fixtures/traversal-preparation-child.mjs	Use actual large requests and libuv crypto contention without stubbing modules.
test/load-addon-resolution.test.mjs	Align native-loader compatibility fixtures with the versioned contract.
test/native-autocomplete-traversal.test.mjs	Verify later eligible prefix hits, scored sequence parity, Boolean/filter behavior, snapshot retention and explicit expansion/capability failures.
test/native-expensive-budget.optimized.test.mjs	Exercise both phrase and traversal timeout/ordinary-capacity/native-saturation/close/reopen phases, including preparation release before native submission.
test/native-traversal-preparation.test.mjs	Run isolated real crypto preparation admission, concurrency, deadline, close and recovery regressions.
test/native-traversal.test.mjs	Verify deep traversal, token misuse, stale-prefix expiry precedence, Unicode candidate-set identity/caller immutability/original capacity and frame boundaries, SHA-256 token shape/old-format rejection, response limits and cross-process binding.
test/native.test.mjs	Check updated capabilities, native numeric/filter/analyzer/trace boundaries and finite-score recovery.
ts/addon.d.ts	Keep generated addon metadata/declarations aligned with the versioned native contract.
ts/codec.ts	Encode/decode traversal frames and centralize reusable bounded wire constants; canonicalize traversal candidate sets after native count/byte and original full-frame admission checks.
ts/errors.ts	Expose cursor expiration to consumers.
ts/load-addon.ts	Require the matching native compatibility version.
ts/native.ts	Expose searchPage; authenticate version-3 boundaries with per-object HMAC and bounded SHA-256 snapshot fields; lazily supply native candidate limits to normalization; bound preparation and recheck deadline/close before native submission.

| CONTRIBUTING.md | Match the supported Node versions and document synchronous encoding and identity hashing. |

| docs/native-checkpoint-publication.md | Link checkpoint rules to their implementation. |

| docs/native-reset-quiescence.md | Separate historical close behavior from current teardown; clarify other-reader ownership and close error recovery. |

| docs/release-packaging.md | Use a direct release-steps heading. |

| docs/terminal-writer-admission.md | Mark the former dirty-close behavior as historical and link source and regression tests. |

✅ Verification

• Documentation accuracy pass: checked public types, numeric bounds, preparation, close and reset against source; independent Gemini document review found no actionable corrections. Local Markdown targets, npm run format:check, and git diff --check pass. This follow-up changes documentation only.

• npm run format:check, npm run lint, and NPM_CONFIG_USERCONFIG=/dev/null npm test: passed; 250 Rust/Node/worker/release/offline-consumer checks. The npm override is process-local to avoid this machine's npm script policy; user configuration is unchanged.

• cargo deny check passed advisories, bans, licenses and source policy; npm audit --audit-level=high reported zero vulnerabilities.

• SHA-256 preimage vectors were independently calculated with Python. Native tests distinguish absent deletion generations, generation zero, later deletes and segment replacement. The public token-version assertion fails before correction and passes afterward.

• npm run test:optimized-native: two passing phrase/traversal contention tests, including native queue saturation and capacity recovery.

• Public boundary-tampering and preparation-admission regressions fail before the fixes and pass afterward. Coverage includes finite score/ID/snapshot edits, splicing, old tokens, concurrent initial tokens, large-identity command/byte pressure, real crypto-pool timeout, close and subsequent recovery.

• Deep traversal checks original versions beyond 10,000 hits during publications; autocomplete checks hits beyond 100, mixed scores/UTF-8 ties, Boolean/filter/candidate shapes, retained snapshots and explicit 50/51-term expansion outcomes. Independent Rust collector equivalence and forced-merge/deadline tests pass.

• Full and smoke traversal qualification passed locally with large IDs/versions and concurrent writes/ordinary searches. One full sample used 16,384 records / 12 actual segments. Platform CI runs the smaller correctness profile; Linux/Windows results remain pending, not a local success claim.

• The native SHA-256 follow-up qualification compared identical 512-record fixtures at exactly 1 and 256 segments, page sizes 1 and 32, against pre-SHA f9ec4a4. Three alternating optimized native process pairs verified every scored sequence and total; 256-segment paired full-walk changes were +0.8–2.8% at size 1 and +1.3–2.3% at size 32. These local observations are an accepted integrity cost, not an isolated hash measurement or production bound. The ignored qualification has an explicit reproduction command in the limits guide.

• One optimized macOS arm64 / Node 24 large-query before/after sample measured synchronous submission median 5.72 → 3.12 ms and completion median 321 → 325 ms. The benchmark includes encoding and native work: this shows reduced synchronous preparation, not an overall search speedup or production SLO.

No companion Harper documentation PR is needed for this standalone package change: its API documentation is in this package's README and limits guide. Harper product API/documentation follow its separate integration issue; no live Harper end-to-end result is claimed. The independent review's stale backend contract and 64-bit fingerprint findings are addressed by corrected execution/cancellation documentation and SHA-256 snapshot identity. Corrective review and the closing full review by Claude and Cursor Composer, with Harper domain adjudication, completed at 7bdb596. The original documentation/fingerprint findings and the many-segment performance-qualification follow-up are resolved. The final post-review delta documents the Rust 0.6 compatibility changes and permanent queue-capacity cases, and removes one redundant assertion; it changes no production behavior, passes all 250 checks, and was not independently code-reviewed. A subsequent documentation-only pass corrects the copied API types and reader-lifecycle wording, simplifies the traversal explanation, and adds source links; its independent document check found no actionable corrections. The 48-byte header increase is an accepted, documented change within the unreleased token format. Package publication and Harper integration qualification remain open; platform CI must rerun on the updated head.

Comment generated by kAIle (GPT-6)

Release target: main — prepare 0.6.0; no backport requested.
Related PRs: #48 independent
Complexity: complicated

_{Review-Coverage: authored=codex; ran=claude,cursor-composer; adjudicated=domain; blocked=gemini(auth); declined=cursor-grok,cursor-kimi,cursor-muse; rounds=17; full=11 @ 104cf5b}

_{Review-Attention: deep ~35m (critical: codec.ts; decisions: do-less-alternative, stateless-snapshot-identity, exact-total-every-page, hybrid-hash-scheduling, rust-api-break) @ 104cf5b}