Skip to content

fix(ncbivirus): honor Query.Limit as a walk ceiling + harden reconciliation#55

Merged
001TMF merged 1 commit into
mainfrom
fix/ncbivirus-limit-walk-ceiling
Jun 14, 2026
Merged

fix(ncbivirus): honor Query.Limit as a walk ceiling + harden reconciliation#55
001TMF merged 1 commit into
mainfrom
fix/ncbivirus-limit-walk-ceiling

Conversation

@001TMF

@001TMF 001TMF commented Jun 14, 2026

Copy link
Copy Markdown
Owner

fix(ncbivirus): honor Query.Limit as a walk ceiling + harden reconciliation

Why

The ncbi-virus connector ignored NormalizedQuery.Limit entirely, violating the documented
contract (query.go: "connectors honor plan.Query.Limit as a walk ceiling so a huge upstream is
never fully fetched"
). Every search paged through and fetched one GET per matching accession,
so a filtered viral query took minutes or hung under NCBI's unauthenticated rate limit. (Surfaced
while making the connector usable for a retrieval benchmark.)

Now pinakes search --limit 1 returns the authoritative count in ~1 request — a constrained Ebola
query went from >75s timeout → ~3s, reporting the real count as a clean authoritative_count.

Honor the limit

  • Bound the list walk and the per-accession fetch by Limit, but only when limit < authority
    (strict) — a limited walk is then always incomplete → BestEffort, never a false Complete.
  • Truncate to the first Limit accessions in imposed (sorted) order.
  • Expose Completeness.AuthoritativeCount on a limit-bounded BestEffort (known from the page-0
    x-ncbi-total-count header), so a caller asking only for a count gets the real total cheaply.

Pre-existing false-Completes the gate surfaced (both fixed here)

The cross-model determinism gate, probing the reconciliation around the limit change, found two
latent false-Complete paths unrelated to the limit:

Bug Scenario Fix
Over-listed + 404 page enumerates more distinct accessions than the total claims; a 404 brings retrieved count back to the total the Complete gate now also requires observedDistinct == authority
Covered-but-not-exhausted cursor walk stopped at len(ids) >= total with a next-page token still pending → under-reporting total Completes early (and hash depends on server page order) walk now drains the cursor to nextToken==""; pageBudget still bounds runaways

Verification

  • 6 non-vacuous regression tests, each proven by revert (limit-bounds-walk, over-rejection
    guard at limit>=total, list-walk early stop, limit==under-reported-total, over-listed+404,
    covered-but-not-exhausted cursor).
  • make ci green.
  • Cross-model determinism gate, 4 rounds: rounds 1–3 each caught a real HIGH false-Complete
    (fixed); round 4 → NO DEFECTS FOUND.

Connectors-only (no engine//idl//schema/) → no spine-change label.

🤖 Generated with Claude Code

…ation

The ncbi-virus connector ignored NormalizedQuery.Limit entirely, violating the documented
walk-ceiling contract (query.go: "connectors honor plan.Query.Limit as a walk ceiling so a
huge upstream is never fully fetched"). A search therefore always paged through and fetched
EVERY matching record (one GET per accession) — a filtered viral query could take minutes or
hang under NCBI's unauthenticated rate limit. Now `pinakes search --limit 1` returns the
authoritative count in ~1 request.

Honor the limit:
- Bound the list walk and the per-accession fetch by Limit, but ONLY when limit < authority
  (strict): a limited walk is then always incomplete -> BestEffort, never a false Complete.
- Truncate to the first Limit accessions in imposed (sorted) order.
- On a limit-bounded BestEffort, expose the authoritative total via Completeness.AuthoritativeCount
  (it is known from the page-0 x-ncbi-total-count header), so a caller asking only for a count
  gets the real number from a cheap limited fetch.

The cross-model determinism gate then surfaced two PRE-EXISTING false-Complete paths in the
reconciliation (latent, unrelated to the limit change), both fixed here:
- Over-listed + 404: a page enumerating MORE distinct accessions than the total claims, where a
  404 brings the retrieved count back to the total, falsely Completed. The Complete gate now also
  requires observedDistinct == authority (distinct accessions enumerated, captured before
  truncation).
- Covered-but-not-exhausted cursor: the walk stopped at len(ids) >= total even with a next-page
  token still pending, so an under-reporting total Completed early (and the hash depended on
  server page order). The walk now drains the cursor to nextToken=="" (pageBudget still bounds
  runaways); observedDistinct == authority downgrades any over-/under-enumeration.

Six non-vacuous regression tests (each proven by revert). make ci green. Verified by the
cross-model gate across four rounds (rounds 1-3 each caught a real HIGH; round 4 clean).
Connectors-only change (no engine/idl/schema).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@001TMF 001TMF merged commit 5be76e1 into main Jun 14, 2026
3 checks passed
@001TMF 001TMF deleted the fix/ncbivirus-limit-walk-ceiling branch June 14, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant