Skip to content

Add distributed IVF_RQ builds with a shared rotation model - #85

Merged
everySympathy merged 2 commits into
daft-engine:mainfrom
everySympathy:codex/distributed-ivf-rq
Oct 12, 2026
Merged

everySympathy merged 2 commits into
daft-engine:mainfrom
everySympathy:codex/distributed-ivf-rq

Conversation

@everySympathy

@everySympathy everySympathy commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Add IVF_RQ to distributed vector index creation. The coordinator trains IVF centroids and generates one RaBitQ rotation model, then every worker uses those shared models to write its segment. This makes separately built RQ segments searchable together and mergeable through optimize_indices.

RQ defaults to 1 bit per dimension; num_bits configures its bit width. Omitting num_bits continues to use 8 bits for PQ. Lance validates RQ configurations, including dimensions divisible by 8. Model generation stays internal, and caller-supplied rabitq_model is rejected.

Creation still covers the full snapshot, replacement uses one atomic commit, and incremental coverage belongs to optimize_indices. The README explains lossy RQ query results and refinement with original vectors.

Validation:

  • Native full suite: 535 passed, 10 skipped, 2 xfailed, 2 xpassed.
  • Ray vector and concurrency suites: 60 passed.
  • Full pre-commit: passed.
  • L2, cosine and dot build/query smoke checks: passed.
  • GitHub CI: all 6 checks passed (Python 3.10–3.13, Ray, style).
  • Real RQ tests cover shared rotations, default/2/4-bit builds, segment merging, incremental coverage, atomic replacement, rejected existing names, invalid dimensions, and worker failures preserving the old index.

@everySympathy
everySympathy merged commit b7b0d67 into daft-engine:main Oct 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant