Recover incomplete full-text highlight batches

[kylebernhardy]

⊙ Problem

Highlighted full-text searches can fail with 500 Full-text index returned incomplete highlights even when each record fits the native limits. The trace span limit applies across a batch: 60 records with 17 matches each fit its 1,024 spans, while adding one match per record exceeds it. The affected adapter is already shipped in v5.3.2.

❓ Your call: Ship this recovery in the next v5.3 patch? The proposal fixes existing highlighted searches without changing the native package or adding a public configuration option; deferring it leaves those searches failing.

💡 Solution

Discard incomplete multi-record trace results and retry sequential halves. The 60 × 18 example now returns all 60 records with complete highlights. Each child retains the original reader lease, execution deadline, cancellation and transaction checks.

Outcome	Behavior
Complete trace	Accumulate its authorized highlight fields
Incomplete multi-record trace with allowance remaining	Discard partial output and split into disjoint halves
Incomplete singleton	Preserve the existing incomplete-highlights message and status 500
Additional-call allowance exhausted	Return status 500 with advice to reduce the result limit or omit $highlights
Cancellation, closed transaction or native rejection	Stop; do not continue splitting

⚖️ Alternatives

• A smaller fixed record batch cannot guarantee fitting a content-dependent span limit.
• Reusing a partial native prefix would require an ordering/completeness contract the current wrapper does not provide.
• Increasing native limits or tracing halves concurrently would increase work or change a separate package's contract.

❓ Your call: Keep one allowance of 256 additional trace calls per search request? This bounds amplification across every leaf and source batch, at the cost of rejecting some recoverable large requests. Scaling that allowance is an independent, reversible policy change.

🔧 Changes

The private highlight path in resources/indexes/fullTextQueryIndex.ts retains the existing record/source-byte batching and allocates a bounded work stack only after an incomplete trace; normally complete traces allocate no recovery stack:

fullTextQueryIndex.ts:773

traced = await reader.traceMatches(…, batch, originalDeadline)
assertSearchActive()
if (traced.complete) → authorized highlight fields
else if (batch.length < 2) → existing 500
else if (remainingAdditionalTraceCalls < 2) → actionable 500
else → reserve two calls; visit left half, then right half

⚠️ Look hardest: An incomplete parent contributes no spans. Only complete child results can reach a successful response; the original reader lease and deadline cover the entire split sequence.

resources/indexes/DESIGN.md records these invariants. resources/FULL_TEXT.md explains the bounded recovery and the two failure messages. No schema, native binary, wrapper dependency, or harper-pro pointer changes are included. No companion public documentation PR is proposed: this is an internal recovery fix with unchanged search API and limits; the existing core full-text reference documents its failure behavior.

The shared allowance spans every leaf and source batch; the liveness callback centralizes cancellation and transaction checks before and after native work.

✅ Verification

• Observed: build, focused lint/format checks, design-document indexing and diff checks passed.
• Observed: 112 focused tests passed against the final ordinary build. unitTests/resources/fullTextQueryIndex.test.js covers recursive splitting, partial-output rejection, field filtering, the shared allowance, distinct failure messages, cancellation, transaction invalidation, native errors and the original deadline and the exact left-first trace order.
• Observed: unitTests/resources/nativeFullTextTableSearch.test.js exercises real Table.search recovery. unitTests/resources/nativeFullTextDerivedIndexLifecycle.test.js proves the real native 1,020/1,080-span boundary and complete halves. Both pin the runtime limits needed for this regression. The real Table.search regression fails against the original adapter.
• Observed: all 12 final-build integration tests passed in 25 seconds. integrationTests/resources/full-text-search.test.ts covers structured searches and public REST with all 60 records and 18 spans each.
• Observed: Claude and public Gemini code reviews completed four rounds, including two whole-artifact passes. The final commit received a narrow delta review, not a fresh whole-artifact pass. Final adjudication did not converge: it retains the deliberate 256-call tradeoff and flags the single liveness callback allocation per highlighted search as an unmeasured performance concern. The author retains the callback to centralize repeated cancellation and transaction checks; no callback-cost measurement or zero-cost claim is made. The deadline-fixture scheduling concern and test-only native-limit contract nit also remain disclosed; these reviews are not an all-findings-retired claim.
• Observed: the published native 0.5.0 module reports maxTraceRecords: 100, maxTraceSourceBytes: 1048576 and maxTraceSpans: 1024; the real-native premise assertions passed in the executed suites.
• Broad suites are not green: resource tests reported 4,290 passing, 54 pending and one subscription-replay failure; that case passed its focused rerun. Main unit tests reported 7,006 passing, 216 pending and 51 failing. One verified owned harness was terminated after a zombie-process stall; its later reported pass is intervention-assisted, so this is not an unassisted full-main qualification. Full integration reported 2,211 passing, 57 failing, 187 cancelled and 18 skipped. Recorded failures include npm package-output parsing, deployment setup, Unix-socket path limits, certificate/restart assertions and other unqualified failures; no blanket claim that they reproduce on base is made.
• Hosted CI remains blocked: the original Integration Tests 6/6 job and its single targeted retry each reported 307 passing and two failing tests out of 309. Both failures are QA-822 purge-log assertions at lines 573 and 736. The isolated fixture has no full-text index or highlight calls, and its test/resource/schema/config inputs are unchanged from base; this establishes isolation, not a proven baseline failure. No further retry or storage fix is included in this patch.
• Moving the disposable qualification runtime out of the core tree removed an author-created scanner obstruction; all 13 watch-path tests and the design-document gate then passed. This does not resolve the other broad-suite failures.

Comment generated by kAIle (GPT-6); posted via @kylebernhardy.

Release target: v5.3 — repairs the adapter already shipped in v5.3.2; proposed next patch milestone.
Related PRs: none found (partial scan: two large candidates could not be fully matched)
Complexity: complicated

_{Review-Coverage: authored=codex; ran=gemini,claude; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=4; full=2 @ 357150c}

_{Review-Attention: study ~20m (critical: fullTextQueryIndex.ts; decisions: recovery-budget-shape, incomplete-highlight-semantics, do-less-alternative) @ 357150c}

[github-actions]

Release cherry-pick v5.3: cancelled

Cherry-pick branch cherry-pick/v5.3/pr-3247 was deleted — this PR was closed without merging.

[gemini-code-assist]

Code Review

This pull request implements a retry mechanism for full-text search highlight tracing when a multi-record native trace exceeds its span budget. Incomplete multi-record traces are now discarded and retried in sequential halves under the original query deadline and cancellation, up to a maximum of 256 additional trace calls per search request. The PR also updates the design and user documentation, and adds comprehensive unit, integration, and lifecycle tests to verify this recovery behavior. There are no review comments, and I have no feedback to provide.

[kriszyp]

My review mentions that:
Halving is a sound Harper-side mitigation, but it discards every completed record in a failed batch and can spend up to 257 calls where the native side could report the completed-record prefix (or a per-record truncated flag) and let Harper retry only the remainder in one extra call. Worth filing against HarperFast/fulltext as the long-term fix; the singleton case (one record with > 1024 matches, e.g. a long text/plain blob and a frequent term) is unrecoverable from Harper at all and will keep 500-ing until that lands.

Is this worth addressing in the native side?

_{🤖 Reviewed with Claude}

[kriszyp]

Status-code consistency: this failure is a deterministic function of the request (result limit × match density), and the message tells the client how to change the request. That is the shape of a 4xx, and it is exactly how the sibling oversize-source check is classified (ClientError(..., 400) in traceEntryBatches/traceSourceBatches). As a 500 it will trip client/proxy retry policies for a request that can never succeed unchanged, and it will show up in error-rate alerts as a server fault.

The DESIGN note says 500 is retained deliberately; if the reason is "class name is the public error code and the singleton case was already ServerError", consider at least making this new branch a ClientError (400 or 422) since it is a new message anyway and nobody is coding against it yet. If there is a reason it must stay 500, a one-line rationale in the DESIGN paragraph would stop the next reviewer from asking the same question.

[kriszyp]

After this change this is the only unrecoverable branch, and it fires for a legitimate, deterministic input: a single record whose selected highlight fields hold more than 1024 matches of the query terms (easy with a text/plain blob and a common word, since maxTraceSourceBytes is far larger than 1024 tokens). The message gives the client nothing to act on, unlike the budget message two lines down, and it cannot be told apart from a native fault by an operator.

Suggest two small things: (1) append the same advice ("omit $highlights or narrow the query"), and (2) log once with the record id/field so an operator can see which record is un-highlightable. Longer term this needs a native-side fix (per-record truncation flag or caller-supplied span cap) — worth filing against the Fulltext package, since nothing Harper does with batching can recover a singleton.

[kriszyp]

Nit: this is declared before the two early returns (!highlighting, highlightFields.size === 0). Moving it below them keeps the no-highlight path allocation-free and reads as "budget belongs to the tracing phase". Trivial, flagging only because this runs per request.