diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 45702e6..2f3551d 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -38,9 +38,11 @@ backends. `UnsupportedCapabilityError`, `EmbeddingRequiredError`, `BackendNotInstalledError`); capability protocols. - `backends/` — 15 adapters: `memory`, `chroma`, `faiss`, `sqlite_vec`, - `duckdb`, `lancedb`, `qdrant` (all tested here) and `pgvector`, `pinecone`, - `weaviate`, `milvus`, `redis`, `elasticsearch`, `mongodb`, `turbopuffer` - (correct-by-construction; need a server/account to exercise). + `duckdb`, `lancedb`, `qdrant`, `milvus` (embedded; tested whenever their + client is installed, milvus via milvus-lite) and + `pgvector`, `pinecone`, `weaviate`, `redis`, `elasticsearch`, `mongodb` + (tested against local containers, no accounts) and `turbopuffer` + (correct-by-construction: no local emulator exists). - `providers.py` + `data/providers.yaml` — the ~21-provider registry + `recommend_backend` decision framework. `requirements.py` — `check_requirements` / `setup_guide` / `install_backend`. @@ -83,10 +85,16 @@ too; the root `conftest.py` skips backend modules whose SDK is missing. Parametrized suites (`tests/test_core.py`, `tests/test_hybrid.py`) run every reachable backend via the `client` fixture in `tests/conftest.py`: embedded backends always, server backends only when their port answers. Bring the -servers up with `docker compose -f tests/docker-compose.yml up -d` and install -their clients (`uv pip install -e ".[pgvector,redis,elasticsearch,weaviate,mongodb,milvus]"`). -CI installs only the `test` extra, so backend-specific tests skip there — -run them locally before changing an adapter. +servers up with `DOCKERHUB_MIRROR=mirror.gcr.io docker compose -f tests/docker-compose.yml up -d` +(the mirror avoids Docker Hub's anonymous pull limit) and install their clients +(`uv pip install -e ".[pgvector,redis,elasticsearch,weaviate,mongodb,milvus,pinecone]" "pinecone<10"`). +No server backend needs an account: pinecone runs on Pinecone Local (which +needs the pinecone SDK < 10) and mongodb on MongoDB Atlas Local. Only +turbopuffer has no emulator. See `tests/README.md`. +CI installs the `test` and `ci-backends` extras (qdrant-client, lancedb, +pgvector's client), so the embedded qdrant and LanceDB suites run there; +server backends still skip in CI — run them locally before changing an +adapter. ## 3. Core contracts (the design the refactor should converge on) diff --git a/README.md b/README.md index 5934ea4..bf4fd37 100644 --- a/README.md +++ b/README.md @@ -234,7 +234,15 @@ uv pip install -e ".[test,dev]" python -m pytest --doctest-modules -o doctest_optionflags='ELLIPSIS IGNORE_EXCEPTION_DETAIL' ``` -That is exactly what CI runs; package doctests are included. The backend-parametrized suites skip server backends that aren't running. To exercise them, start the containers with `docker compose -f tests/docker-compose.yml up -d` and install their clients with `uv pip install -e ".[pgvector,redis,elasticsearch,weaviate,mongodb,milvus]"`. +That is what CI runs (CI also installs the `ci-backends` extra); package doctests are included. The backend-parametrized suites skip server backends that aren't running. Every server backend except turbopuffer runs locally without an account, Pinecone and MongoDB Atlas included, through their official emulators: + +```bash +DOCKERHUB_MIRROR=mirror.gcr.io docker compose -f tests/docker-compose.yml up -d +uv pip install -e ".[test,dev,pgvector,redis,elasticsearch,weaviate,mongodb,milvus,pinecone]" "pinecone<10" +python -m pytest +``` + +See [`tests/README.md`](tests/README.md) for connection settings and caveats. **Design rationale.** diff --git a/pyproject.toml b/pyproject.toml index 58b075a..cc22362 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -86,6 +86,10 @@ dev = [ # suite (tests/test_async.py) has pytest-asyncio. # hatchling: tests/test_package.py checks the sdist file list (#28). test = ["pytest>=7.0", "pytest-cov>=4.0", "pytest-asyncio>=0.23", "hatchling"] +# Server-free backend clients CI installs so their tests run instead of +# skipping: qdrant (embedded :memory: + native async), lancedb (embedded, FTS +# hybrid), and pgvector's client for its server-free unit test. +ci-backends = ["qdrant-client", "lancedb", "pgvector", "psycopg[binary]"] docs = ["sphinx>=6.0", "sphinx-rtd-theme>=1.0"] [tool.hatch.build.targets.wheel] @@ -149,7 +153,7 @@ installer = "uv" # Install the `test` extra so CI has the async test runner (pytest-asyncio); # without it tests/test_async.py errors with "async def functions are not # natively supported" and every async test fails. -extras = "test" +extras = "test,ci-backends" [tool.wads.ci.testing] python_versions = ["3.10", "3.12"] diff --git a/tests/README.md b/tests/README.md index 9d6edf4..3b446c3 100644 --- a/tests/README.md +++ b/tests/README.md @@ -15,16 +15,27 @@ proves one uniform interface over many vector databases. loadable-extension support; `milvus` runs against the embedded **Milvus Lite** engine and is skipped if `milvus-lite` is not installed. - **Server backends** — `pgvector`, `redis`, `elasticsearch`, `weaviate`, - `mongodb`. Each needs a running container. A backend whose port is not open is + `mongodb`, `pinecone`, and `qdrant_server` (the qdrant adapter with `url=`). + Each needs a running container. A backend whose port is not open is **skipped**, so the suite stays green in a plain CI environment. +None of them needs an account. `pinecone` runs against **Pinecone Local**, the +official in-memory emulator, and `mongodb` against **MongoDB Atlas Local**. +Only `turbopuffer` has no local emulator; its maintainers recommend testing +against production with a throwaway namespace, so it is not in the suite. + ## Running against the server backends ```bash # Start every server backend (first run pulls images — a few minutes). -docker compose -f tests/docker-compose.yml up -d +# DOCKERHUB_MIRROR=mirror.gcr.io avoids Docker Hub's anonymous pull limit. +DOCKERHUB_MIRROR=mirror.gcr.io docker compose -f tests/docker-compose.yml up -d + +# Install every server backend's client. Pinecone Local speaks the pre-2026-07 +# API, so pin the pinecone SDK below 10 for these tests. +uv pip install -e ".[test,dev,pgvector,redis,elasticsearch,weaviate,mongodb,milvus,pinecone]" "pinecone<10" -# Wait until all five report (healthy), then run the suite. +# Wait until the services are up, then run the suite. docker compose -f tests/docker-compose.yml ps pytest tests/ @@ -50,11 +61,28 @@ The server backends use these defaults; override with environment variables: | `elasticsearch` | `VD_ELASTICSEARCH_URL`| `http://localhost:9200` | | `weaviate` | `VD_WEAVIATE_HOST` | `localhost` | | `mongodb` | `VD_MONGODB_URI` | `mongodb://localhost:27018/?directConnection=true` | +| `pinecone` | `VD_PINECONE_HOST` / `VD_PINECONE_API_KEY` | `http://localhost:5080` / `pclocal` | +| `qdrant_server` | `VD_QDRANT_URL` | `http://localhost:6333` | + +Only local servers are used: an override pointing elsewhere is skipped, because +the suite deletes every collection it can see. Set `VD_ALLOW_REMOTE_TESTS=1` to +run against a remote server you are happy to wipe. `mongodb` maps to host port **27018** (not 27017) to avoid colliding with a developer's native `mongod`. It uses the `mongodb-atlas-local` image because `$vectorSearch` requires an Atlas-capable deployment. +If Elasticsearch times out creating indexes, the host disk is probably above +its 90 % high watermark; on a throwaway test node, disable the check: + +```bash +curl -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' \ + -d '{"persistent":{"cluster.routing.allocation.disk.threshold_enabled":false}}' +``` + +`tests/test_live_server_regressions.py` holds server-only regression tests +(for example, a MongoDB collection recreated under a dropped name). + `milvus` needs no container — it is verified against the embedded Milvus Lite engine, which exercises the same adapter code path as a Milvus server (only the client constructor differs). diff --git a/tests/conftest.py b/tests/conftest.py index b4a09df..af765d8 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -13,13 +13,16 @@ ``sqlite3`` lacks loadable-extension support; ``milvus`` runs against the embedded Milvus Lite engine and is skipped if ``milvus-lite`` is absent. - **Server backends** (``pgvector``, ``redis``, ``elasticsearch``, - ``weaviate``, ``mongodb``) need a running container — see - ``tests/docker-compose.yml``. Each is TCP-probed and **skipped** when its + ``weaviate``, ``mongodb``, ``pinecone`` via Pinecone Local, and + ``qdrant_server``: the qdrant adapter with ``url=``) need a running + container — see ``tests/docker-compose.yml``; none needs an account. Each is TCP-probed and **skipped** when its container is down, so the suite stays green in a plain CI environment. Connection settings for the server backends are environment-overridable (``VD_PGVECTOR_DSN``, ``VD_REDIS_HOST``/``VD_REDIS_PORT``, -``VD_ELASTICSEARCH_URL``, ``VD_WEAVIATE_HOST``, ``VD_MONGODB_URI``). +``VD_ELASTICSEARCH_URL``, ``VD_WEAVIATE_HOST``, ``VD_MONGODB_URI``, +``VD_PINECONE_HOST``/``VD_PINECONE_API_KEY``, ``VD_QDRANT_URL``). An entry may +set ``"backend"`` to test another entry's adapter under a different setup. """ import hashlib @@ -48,7 +51,10 @@ ] #: Server backends — each needs a container (``tests/docker-compose.yml``). -#: ``probe`` is TCP-probed; the backend is skipped when the port is closed. +#: The server address (from ``connect_kwargs``, falling back to ``probe``) is +#: TCP-probed; the backend is skipped when the port is closed. Only local +#: servers are used unless ``VD_ALLOW_REMOTE_TESTS=1``: the ``client`` fixture +#: deletes every collection it can see, which would wipe a real account. #: ``connect_kwargs`` builds the :func:`vd.connect` arguments (env-overridable). SERVER_BACKENDS = { "pgvector": { @@ -78,6 +84,26 @@ "host": os.environ.get("VD_WEAVIATE_HOST", "localhost") }, }, + "pinecone": { + # Pinecone Local, the official in-memory emulator (no account): + # ghcr.io/pinecone-io/pinecone-local. It speaks the pre-2026-07 API, so + # these tests need the pinecone SDK < 10 until the emulator catches up. + "probe": ("localhost", 5080), + "connect_kwargs": lambda: { + "api_key": os.environ.get("VD_PINECONE_API_KEY", "pclocal"), + "host": os.environ.get("VD_PINECONE_HOST", "http://localhost:5080"), + }, + }, + "qdrant_server": { + # The qdrant adapter against a real Qdrant server (url=), next to the + # embedded "qdrant" entry above: exercises the network client paths. + "backend": "qdrant", + "probe": ("localhost", 6333), + "connect_kwargs": lambda: { + "url": os.environ.get("VD_QDRANT_URL", "http://localhost:6333"), + "check_compatibility": False, + }, + }, "mongodb": { # Host port 27018 — see tests/docker-compose.yml (avoids colliding # with a developer's native mongod on the default 27017). @@ -119,15 +145,74 @@ def _sqlite_ext_supported() -> bool: return False +_LOCAL_HOSTS = {"localhost", "127.0.0.1", "::1"} + + +def server_address(name: str) -> tuple: + """ + The ``(host, port)`` a server entry will connect to. + + ``host`` is ``None`` when it can't be determined (a keyword-style or + multi-host DSN, ...); callers must then treat the target as remote. + """ + import re + from urllib.parse import parse_qs, urlsplit + + entry = SERVER_BACKENDS[name] + default_host, default_port = entry["probe"] + kwargs = entry["connect_kwargs"]() + for key in ("url", "dsn", "uri", "host"): + value = kwargs.get(key) + if not isinstance(value, str): + continue + if "://" in value: + parts = urlsplit(value) + if "," in parts.netloc: # multi-host URI + return None, default_port + query_host = parse_qs(parts.query).get("host") + try: + port = parts.port or default_port + except ValueError: + return None, default_port + if query_host: + return None, port # host given in the query string + return parts.hostname or default_host, port + if key == "dsn": # keyword DSN, e.g. "host=db port=5432 dbname=vd" + found = re.search(r"\bhost(?:addr)?\s*=\s*'?([^\s']+)", value) + port = re.search(r"\bport\s*=\s*'?(\d+)", value) + if not found or "," in found.group(1): + return None, default_port + return found.group(1), int(port.group(1)) if port else default_port + if key == "host": + return value, int(kwargs.get("port", default_port)) + return default_host, default_port + + def _unavailable_reason(name: str) -> str | None: """Return a skip reason for backend ``name``, or ``None`` if it can run.""" if name == "sqlite_vec" and not _sqlite_ext_supported(): return "sqlite3 was built without loadable-extension support" if name == "milvus" and importlib.util.find_spec("milvus_lite") is None: return "milvus-lite not installed (embedded Milvus engine unavailable)" + if name == "pinecone": + import pinecone + + major = str(getattr(pinecone, "__version__", "0")).split(".")[0] + if major.isdigit() and int(major) >= 10: + return ( + "Pinecone Local speaks the pre-2026-07 API; the pinecone SDK " + ">= 10 cannot drive it. Install 'pinecone<10' to run these." + ) if name in SERVER_BACKENDS: - host, port = SERVER_BACKENDS[name]["probe"] - if not _tcp_open(host, port): + host, port = server_address(name) + if host not in _LOCAL_HOSTS and os.environ.get("VD_ALLOW_REMOTE_TESTS") != "1": + return ( + f"{name!r} points at {host or 'a host that could not be parsed'}, " + f"not a local server; the tests " + f"delete every collection they see. Set VD_ALLOW_REMOTE_TESTS=1 " + f"to allow it." + ) + if host is None or not _tcp_open(host, port): return ( f"{name!r} server unreachable at {host}:{port} " f"— start it with tests/docker-compose.yml" @@ -135,6 +220,11 @@ def _unavailable_reason(name: str) -> str | None: return None +def backend_of(name: str) -> str: + """The ``vd.connect`` backend for a fixture entry (entries may alias one).""" + return SERVER_BACKENDS.get(name, {}).get("backend", name) + + def _connect_kwargs(name: str) -> dict: """Return the :func:`vd.connect` kwargs for backend ``name``.""" if name in SERVER_BACKENDS: @@ -194,7 +284,7 @@ def embedder(): def backend_name(request): """Each reachable backend, one at a time; unreachable ones are skipped.""" name = request.param - if name not in vd.list_backends(): + if backend_of(name) not in vd.list_backends(): pytest.skip(f"backend {name!r} is not installed") reason = _unavailable_reason(name) if reason: @@ -206,7 +296,7 @@ def backend_name(request): def client(backend_name, embedder): """A fresh, connected client for each backend (with an embedder).""" connection = vd.connect( - backend_name, embedder=embedder, **_connect_kwargs(backend_name) + backend_of(backend_name), embedder=embedder, **_connect_kwargs(backend_name) ) _drop_all_collections(connection) yield connection diff --git a/tests/docker-compose.yml b/tests/docker-compose.yml index f51898d..ad16612 100644 --- a/tests/docker-compose.yml +++ b/tests/docker-compose.yml @@ -11,10 +11,17 @@ # # milvus is intentionally absent: the milvus adapter is verified against the # embedded Milvus Lite engine (no server), which exercises the same code path. +# +# No accounts needed: pinecone runs as Pinecone Local (the official in-memory +# emulator) and mongodb as MongoDB Atlas Local. turbopuffer has no emulator. +# +# Docker Hub rate-limits anonymous pulls. Set DOCKERHUB_MIRROR=mirror.gcr.io +# (Google's public Docker Hub mirror) to pull the Docker Hub images from there: +# DOCKERHUB_MIRROR=mirror.gcr.io docker compose -f tests/docker-compose.yml up -d services: pgvector: - image: pgvector/pgvector:pg17 + image: ${DOCKERHUB_MIRROR:-docker.io}/pgvector/pgvector:pg17 environment: POSTGRES_USER: vd POSTGRES_PASSWORD: vd @@ -28,7 +35,7 @@ services: retries: 20 redis: - image: redis/redis-stack-server:latest + image: ${DOCKERHUB_MIRROR:-docker.io}/redis/redis-stack-server:latest ports: - "6379:6379" healthcheck: @@ -52,7 +59,7 @@ services: retries: 30 weaviate: - image: cr.weaviate.io/semitechnologies/weaviate:1.28.4 + image: ${DOCKERHUB_MIRROR:-docker.io}/semitechnologies/weaviate:1.28.4 command: ["--host", "0.0.0.0", "--port", "8080", "--scheme", "http"] environment: AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true" @@ -70,7 +77,7 @@ services: retries: 30 mongodb: - image: mongodb/mongodb-atlas-local:latest + image: ${DOCKERHUB_MIRROR:-docker.io}/mongodb/mongodb-atlas-local:latest # Host port 27018 (not 27017) to avoid colliding with a developer's # native mongod. The container still listens on 27017 internally. ports: @@ -80,3 +87,21 @@ services: interval: 5s timeout: 5s retries: 30 + + qdrant: + # The qdrant adapter with url= (the "qdrant_server" test entry). Embedded + # qdrant needs no container and is always tested. + image: ${DOCKERHUB_MIRROR:-docker.io}/qdrant/qdrant:latest + ports: + - "6333:6333" + - "6334:6334" + + pinecone: + # Pinecone Local: in-memory, no account (api_key="pclocal"). It speaks the + # pre-2026-07 API, so run the pinecone tests with the pinecone SDK < 10. + image: ghcr.io/pinecone-io/pinecone-local:latest + environment: + PORT: "5080" + PINECONE_HOST: localhost + ports: + - "5080-5090:5080-5090" diff --git a/tests/test_async_native.py b/tests/test_async_native.py index 2e2a1f7..172a8d1 100644 --- a/tests/test_async_native.py +++ b/tests/test_async_native.py @@ -393,3 +393,38 @@ async def test_hybrid_search_async_native_accepts_sync_lexical_callable( ] assert set(ids) == {"a", "b", "c"} assert ids.index("a") < ids.index("b") or ids[0] == "b" + + +async def test_qdrant_native_against_live_server(qdrant_available): + """With a Qdrant server up, connect_async(url=) is native and matches sync.""" + from tests.conftest import _connect_kwargs, _unavailable_reason + + reason = _unavailable_reason("qdrant_server") + if reason: + pytest.skip(reason) + kwargs = _connect_kwargs("qdrant_server") + sync = vd.connect("qdrant", **kwargs) + name = "vd_async_live" + if name in sync: + sync.delete_collection(name) + try: + async with await vd.connect_async("qdrant", **kwargs) as client: + assert client.native_async is True + col = await client.create_collection(name, dimension=2, metric="l2") + await _populate(col) + await col.add_documents( + [vd.Document(id=f"d{i}", text="t", vector=[0.1, 0.01 * i]) + for i in range(300)], batch_size=128) + assert await col.count() == 303 + assert len({k async for k in col.keys()}) == 303 + native_hits = [(h["id"], round(h["score"], 6)) async for h in + col.search([0.7, 0.3], limit=3, + filter={"k": {"$in": [1, 3]}})] + sync_hits = [(h["id"], round(h["score"], 6)) for h in + sync.get_collection(name).search( + [0.7, 0.3], limit=3, filter={"k": {"$in": [1, 3]}})] + assert native_hits == sync_hits and [i for i, _ in native_hits] == ["a", "c"] + finally: + if name in sync: + sync.delete_collection(name) + sync.close() diff --git a/tests/test_live_server_regressions.py b/tests/test_live_server_regressions.py new file mode 100644 index 0000000..4a68163 --- /dev/null +++ b/tests/test_live_server_regressions.py @@ -0,0 +1,70 @@ +""" +Regression tests that need a live server backend. + +Each test is skipped unless its server answers (start them with +``docker compose -f tests/docker-compose.yml up -d``). They cover behaviour the +parametrized contract suites only hit by accident of test ordering. +""" + +import pytest + +import vd +from tests.conftest import _connect_kwargs, _unavailable_reason, backend_of + + +def _live_client(name): + if backend_of(name) not in vd.list_backends(): + pytest.skip(f"backend {name!r} is not installed") + reason = _unavailable_reason(name) + if reason: + pytest.skip(reason) + return vd.connect(backend_of(name), **_connect_kwargs(name)) + + +def test_mongodb_search_after_drop_and_recreate_same_name(): + """A recreated collection gets a fresh vector index (#29). + + Atlas briefly lists the dropped collection's index as DOES_NOT_EXIST; the + adapter used to take that as "exists", skip creation, and time out. + """ + client = _live_client("mongodb") + name = "vd_regress_recreate" + try: + for _ in range(2): + if name in list(client.list_collections()): + client.delete_collection(name) + col = client.create_collection(name, dimension=2) + col["a"] = vd.Document(id="a", text="x", vector=[1.0, 0.0]) + assert [h["id"] for h in col.search([1.0, 0.0], limit=1)] == ["a"] + finally: + if name in list(client.list_collections()): + client.delete_collection(name) + client.close() + + +def test_qdrant_get_collection_recovers_metric_from_server(): + """A client that didn't create an l2 collection must still score it as l2. + + get_collection used to assume cosine for collections created elsewhere, + returning raw Euclidean distances as (lower-is-better) scores. + """ + maker = _live_client("qdrant_server") + name = "vd_regress_metric" + try: + if name in maker: + maker.delete_collection(name) + col = maker.create_collection(name, dimension=2, metric="l2") + col["a"] = vd.Document(id="a", text="x", vector=[1.0, 0.0]) + col["b"] = vd.Document(id="b", text="y", vector=[0.0, 1.0]) + expected = [(h["id"], round(h["score"], 6)) for h in col.search([0.9, 0.1])] + reader = _live_client("qdrant_server") + other = reader.get_collection(name) + assert (other.metric, other.dimension) == ("l2", 2) + got = [(h["id"], round(h["score"], 6)) for h in other.search([0.9, 0.1])] + assert got == expected + assert all(0 < s <= 1 for _, s in got) # canonical l2 score range + reader.close() + finally: + if name in maker: + maker.delete_collection(name) + maker.close() diff --git a/tests/test_pinecone_unit.py b/tests/test_pinecone_unit.py new file mode 100644 index 0000000..f57bb4f --- /dev/null +++ b/tests/test_pinecone_unit.py @@ -0,0 +1,18 @@ +"""Server-free tests for the pinecone adapter's collection-name mapping.""" + +import pytest + +pytest.importorskip("pinecone") + +from vd.backends.pinecone import _collection_name, _index_name # noqa: E402 + + +@pytest.mark.parametrize("name", ["docs", "my_docs", "my-docs", "a1_b2-c3", "a_-b"]) +def test_valid_names_round_trip(name): + assert _collection_name(_index_name(name)) == name + + +@pytest.mark.parametrize("name", ["a--b", "a-_b", "Docs", "a.b", "", "x" * 46]) +def test_ambiguous_or_invalid_names_are_rejected(name): + with pytest.raises(ValueError): + _index_name(name) diff --git a/vd/backends/mongodb.py b/vd/backends/mongodb.py index e07fbcc..5a8fb74 100644 --- a/vd/backends/mongodb.py +++ b/vd/backends/mongodb.py @@ -94,6 +94,9 @@ "l2": "euclidean", } +#: Search-index statuses Atlas reports for an index that is gone or going. +_GONE_INDEX_STATUSES = frozenset({"DOES_NOT_EXIST", "DELETING"}) + # --------------------------------------------------------------------------- # Helper: build a $vectorSearch aggregation pipeline @@ -283,7 +286,7 @@ def _ensure_search_index(self) -> None: if self._index_ready: return try: - existing = {idx["name"] for idx in self._coll.list_search_indexes()} + existing = self._live_search_index_names() except OperationFailure as exc: raise RuntimeError( "This MongoDB deployment does not support Atlas Vector Search. " @@ -298,9 +301,31 @@ def _ensure_search_index(self) -> None: f"Write a document first, or pass dimension= to " f"create_collection." ) - from pymongo.operations import SearchIndexModel + self._create_vector_index() + self._wait_until_queryable() + self._index_ready = True - similarity = _METRIC_TO_ATLAS_SIMILARITY.get(self.metric, "cosine") + def _live_search_index_names(self) -> set: + """ + Names of this collection's search indexes that actually exist. + + Right after a collection is dropped, Atlas still lists its search + indexes for a moment with status ``DOES_NOT_EXIST`` (or ``DELETING``). + A same-named collection created in that window must not mistake them + for its own (#29). + """ + return { + idx["name"] + for idx in self._coll.list_search_indexes() + if idx.get("status") not in _GONE_INDEX_STATUSES + } + + def _create_vector_index(self) -> None: + """Submit the Atlas vector search index definition for this collection.""" + from pymongo.operations import SearchIndexModel + + similarity = _METRIC_TO_ATLAS_SIMILARITY.get(self.metric, "cosine") + try: self._coll.create_search_index( SearchIndexModel( definition={ @@ -317,8 +342,10 @@ def _ensure_search_index(self) -> None: type="vectorSearch", ) ) - self._wait_until_queryable() - self._index_ready = True + except OperationFailure as exc: + # A concurrent creator won the race; the wait loop takes over. + if "already exists" not in str(exc).lower(): + raise def _wait_until_queryable( self, *, timeout: float = 120.0, poll_interval: float = 1.0 @@ -328,9 +355,17 @@ def _wait_until_queryable( deadline = time.monotonic() + timeout while time.monotonic() < deadline: - for idx in self._coll.list_search_indexes(self._vector_index): - if idx.get("queryable"): - return + listed = [ + idx + for idx in self._coll.list_search_indexes(self._vector_index) + if idx.get("status") not in _GONE_INDEX_STATUSES + ] + if any(idx.get("queryable") for idx in listed): + return + if not listed and self.dimension is not None: + # The index we waited on belonged to a dropped, same-named + # collection and has vanished: create ours (#29). + self._create_vector_index() time.sleep(poll_interval) raise RuntimeError( f"Atlas vector search index {self._vector_index!r} for collection " diff --git a/vd/backends/pinecone.py b/vd/backends/pinecone.py index 4264214..7fa0e15 100644 --- a/vd/backends/pinecone.py +++ b/vd/backends/pinecone.py @@ -238,6 +238,73 @@ def _to_pinecone_filter(ast: Optional[Filter]) -> Optional[dict]: return out +#: Pinecone index names: lowercase letters, digits and hyphens, max 45 chars. +_PINECONE_NAME_MAX = 45 + + +def _index_name(name: str) -> str: + """ + Map a ``vd`` collection name to a Pinecone index name. + + Pinecone allows only ``[a-z0-9-]`` while other backends (Milvus, SQL + tables) reject hyphens, so portable ``vd`` names use underscores. Each + ``_`` becomes ``--`` here and :func:`_collection_name` maps it back, so + ``"my_docs"`` is stored as the index ``"my--docs"``. A name that already + contains ``--`` would be ambiguous and is rejected. + + >>> _index_name("my_docs") + 'my--docs' + >>> _collection_name(_index_name("my_docs")) + 'my_docs' + """ + import re + + if "--" in name or not re.fullmatch(r"[a-z0-9_-]+", name): + raise ValueError( + f"Invalid Pinecone collection name {name!r}: use lowercase letters, " + f"digits, '_' and '-' (not '--'). vd stores '_' as '--' because " + f"Pinecone index names allow only [a-z0-9-]." + ) + mapped = name.replace("_", "--") + if _collection_name(mapped) != name: # e.g. "a-_b" and "a_-b" -> "a---b" + raise ValueError( + f"Invalid Pinecone collection name {name!r}: '-_' is ambiguous " + f"once '_' is stored as '--'. Write it as '_-' or separate them." + ) + if len(mapped) > _PINECONE_NAME_MAX: + raise ValueError( + f"Pinecone collection name {name!r} is too long: it maps to the " + f"{len(mapped)}-character index name {mapped!r} (max " + f"{_PINECONE_NAME_MAX}; each '_' counts twice)." + ) + return mapped + + +def _collection_name(index_name: str) -> str: + """Map a Pinecone index name back to its ``vd`` collection name.""" + return index_name.replace("--", "_") + + +def _ids_from_list_page(page) -> list[str]: + """ + Extract document ids from one item yielded by ``Index.list()``. + + Depending on the SDK version an item is a list of ids, a response with + ``.vectors`` (each with ``.id``), or a page with ``.ids``. + """ + if isinstance(page, str): + return [page] + if isinstance(page, list): + return [x if isinstance(x, str) else getattr(x, "id", x) for x in page] + vectors = getattr(page, "vectors", None) + if vectors is not None: + return [getattr(v, "id", v) for v in vectors] + ids = getattr(page, "ids", None) + if ids is not None: + return list(ids) + return [getattr(x, "id", x) for x in page] + + def _index_is_ready(index_description) -> bool: """ Return ``True`` when a Pinecone index description's status is ``"Ready"``. @@ -307,9 +374,12 @@ def __init__( metric: str = "cosine", cloud: str = "aws", region: str = "us-east-1", + plain_http: bool = False, ): self.name = name + self._index_name = _index_name(name) self._pc = pc + self._plain_http = plain_http self._embedder = embedder self.dimension = dimension self.metric = metric @@ -328,7 +398,14 @@ def native(self): def _get_index(self): """Return the Pinecone ``Index`` handle, resolving it once.""" if self._index is None: - self._index = self._pc.Index(self.name) + if self._plain_http: + # Pinecone Local serves each index over plain HTTP but reports + # its host as https://…; open it on the http:// address. + host = self._pc.describe_index(self._index_name).host + host = "http://" + host.split("://", 1)[-1] + self._index = self._pc.Index(host=host) + else: + self._index = self._pc.Index(self._index_name) return self._index def _ensure_index(self) -> None: @@ -354,9 +431,9 @@ def _ensure_index(self) -> None: f"dimension= to create_collection." ) existing_names = {idx.name for idx in self._pc.list_indexes()} - if self.name not in existing_names: + if self._index_name not in existing_names: self._pc.create_index( - name=self.name, + name=self._index_name, dimension=self.dimension, metric=_pinecone_metric(self.metric), spec=ServerlessSpec(cloud=self._cloud, region=self._region), @@ -470,20 +547,7 @@ def _keys(self) -> Iterator[str]: return iter(()) ids: list[str] = [] for page in index.list(): - # ``page`` may be a list of ids or a list-like page object with an - # ``ids`` attribute, depending on the SDK version. - if isinstance(page, list): - ids.extend(page) - else: - page_ids = getattr(page, "ids", None) - if page_ids: - ids.extend(page_ids) - else: - # Fall back: treat the object itself as iterable. - try: - ids.extend(list(page)) - except TypeError: - pass + ids.extend(_ids_from_list_page(page)) return iter(ids) def _count(self) -> int: @@ -586,6 +650,15 @@ class PineconeClient(AbstractClient): region : str Cloud region for serverless indexes, e.g. ``"us-east-1"`` (the default and the only region available on the Starter free tier). + host : str, optional + Control-plane URL. Omit for Pinecone's cloud. Set it to reach + **Pinecone Local**, the official in-memory emulator that needs no + account: ``docker run -p 5080-5090:5080-5090 -e PORT=5080 + -e PINECONE_HOST=localhost ghcr.io/pinecone-io/pinecone-local``, then + ``vd.connect("pinecone", api_key="pclocal", + host="http://localhost:5080")``. With an ``http://`` host, indexes + are opened over plain HTTP too. (Pinecone Local speaks the pre-2026-07 + API, so it needs the ``pinecone`` SDK < 10.) **config Additional keyword arguments passed through to :class:`AbstractClient`. @@ -610,6 +683,7 @@ def __init__( api_key: Optional[str] = None, cloud: str = "aws", region: str = "us-east-1", + host: Optional[str] = None, **config, ): super().__init__(embedder=embedder, **config) @@ -620,15 +694,23 @@ def __init__( "PINECONE_API_KEY environment variable. Get a key at " "https://app.pinecone.io." ) - self._client = Pinecone(api_key=resolved_key) + self._client = ( + Pinecone(api_key=resolved_key, host=host) + if host + else Pinecone(api_key=resolved_key) + ) self._cloud = cloud self._region = region + self._plain_http = bool(host) and host.startswith("http://") + #: Collections created without a dimension: no index exists until the + #: first write, but the name is taken (vd name -> metric). + self._pending: dict[str, str] = {} # ----- helpers ---------------------------------------------------------- # def _existing_index_names(self) -> set[str]: - """Return the set of index names currently registered in this account.""" - return {idx.name for idx in self._client.list_indexes()} + """Return the ``vd`` names of the indexes registered in this account.""" + return {_collection_name(idx.name) for idx in self._client.list_indexes()} # ----- AbstractClient interface ----------------------------------------- # @@ -671,7 +753,8 @@ def create_collection( ValueError If a collection (index) with that name already exists in the account. """ - if name in self._existing_index_names(): + _index_name(name) # validate before touching the account + if name in self._existing_index_names() or name in self._pending: raise ValueError( f"Collection {name!r} already exists in this Pinecone account." ) @@ -683,10 +766,13 @@ def create_collection( metric=metric, cloud=self._cloud, region=self._region, + plain_http=self._plain_http, ) if dimension is not None: # Eager index creation when the dimension is known up front. collection._ensure_index() + else: + self._pending[name] = metric return collection def get_collection(self, name: str) -> PineconeCollection: @@ -703,10 +789,23 @@ def get_collection(self, name: str) -> PineconeCollection: KeyError If no index with that name exists in the account. """ - indexes = {idx.name: idx for idx in self._client.list_indexes()} + indexes = { + _collection_name(idx.name): idx for idx in self._client.list_indexes() + } + if name not in indexes and name in self._pending: + return PineconeCollection( + name, + self._client, + embedder=self._embedder, + metric=self._pending[name], + cloud=self._cloud, + region=self._region, + plain_http=self._plain_http, + ) if name not in indexes: raise KeyError( - f"Collection {name!r} does not exist. " + f"Collection {name!r} does not exist (vd lists a Pinecone index " + f"named 'a--b' as collection 'a_b'). " f"Existing collections: {sorted(indexes)}" ) idx_info = indexes[name] @@ -727,6 +826,7 @@ def get_collection(self, name: str) -> PineconeCollection: metric=vd_metric, cloud=self._cloud, region=self._region, + plain_http=self._plain_http, ) def delete_collection(self, name: str) -> None: @@ -743,9 +843,12 @@ def delete_collection(self, name: str) -> None: KeyError If no index with that name exists in the account. """ - if name not in self._existing_index_names(): + exists = name in self._existing_index_names() + if not exists and name not in self._pending: raise KeyError(f"Collection {name!r} does not exist and cannot be deleted.") - self._client.delete_index(name) + if exists: + self._client.delete_index(_index_name(name)) + self._pending.pop(name, None) def list_collections(self) -> Iterator[str]: """ @@ -754,6 +857,8 @@ def list_collections(self) -> Iterator[str]: Returns ------- Iterator[str] - Index names in the order returned by ``Pinecone.list_indexes()``. + ``vd`` collection names (index names with ``--`` mapped back to + ``_``), sorted, including collections not yet written to. """ - return (idx.name for idx in self._client.list_indexes()) + names = self._existing_index_names() | set(self._pending) + return iter(sorted(names)) diff --git a/vd/backends/qdrant.py b/vd/backends/qdrant.py index 09101ef..9a1f30d 100644 --- a/vd/backends/qdrant.py +++ b/vd/backends/qdrant.py @@ -176,6 +176,21 @@ def _to_document(point) -> Document: ) +def _metric_and_dimension(info, default_metric: str = "cosine") -> tuple: + """ + Read ``(vd metric, dimension)`` from a Qdrant ``CollectionInfo``. + + A collection created by another client (or process) is only known to + this one through the server, so the metric that decides score semantics + must come from there, not from local bookkeeping. + """ + vectors = getattr(getattr(info.config, "params", None), "vectors", None) + if not isinstance(vectors, models.VectorParams): # named / multi-vector + return default_metric, None + reverse = {distance: metric for metric, distance in _DISTANCE.items()} + return reverse.get(vectors.distance, default_metric), vectors.size + + def _vectors_config(dimension: int, metric: str) -> "models.VectorParams": """The Qdrant vector config for a collection of ``dimension`` and ``metric``.""" return models.VectorParams( @@ -347,13 +362,20 @@ def create_collection( return collection def get_collection(self, name: str) -> QdrantCollection: - if not self._client.collection_exists(name) and name not in self._metrics: + exists = self._client.collection_exists(name) + if not exists and name not in self._metrics: raise KeyError(f"Collection {name!r} does not exist") + metric, dimension = self._metrics.get(name, "cosine"), None + if exists: + metric, dimension = _metric_and_dimension( + self._client.get_collection(name), metric + ) return QdrantCollection( name, self._client, embedder=self._embedder, - metric=self._metrics.get(name, "cosine"), + dimension=dimension, + metric=metric, ) def delete_collection(self, name: str) -> None: @@ -570,11 +592,15 @@ async def create_collection( return collection async def get_collection(self, name: str) -> NativeAsyncQdrantCollection: - if not await self._client.collection_exists(name) and name not in self._metrics: + exists = await self._client.collection_exists(name) + if not exists and name not in self._metrics: raise KeyError(f"Collection {name!r} does not exist") - return self._collection( - name, dimension=None, metric=self._metrics.get(name, "cosine") - ) + metric, dimension = self._metrics.get(name, "cosine"), None + if exists: + metric, dimension = _metric_and_dimension( + await self._client.get_collection(name), metric + ) + return self._collection(name, dimension=dimension, metric=metric) async def delete_collection(self, name: str) -> None: exists = await self._client.collection_exists(name) diff --git a/vd/data/skills/vd-setup-backend/SKILL.md b/vd/data/skills/vd-setup-backend/SKILL.md index 2288652..b7afe49 100644 --- a/vd/data/skills/vd-setup-backend/SKILL.md +++ b/vd/data/skills/vd-setup-backend/SKILL.md @@ -108,6 +108,26 @@ as above, is for local development only. | `milvus` / Zilliz | `MILVUS_URI`, `MILVUS_TOKEN` | no — pass `uri=`, `token=` | | `elasticsearch` | `ELASTICSEARCH_URL`, `ELASTIC_API_KEY` | no — pass `url=`, `api_key=` | +**Managed, but no account for development:** two managed backends have +official local stand-ins, so you can build and test before signing up. + +```bash +# Pinecone Local: in-memory emulator, any api_key works (use "pclocal"). +docker run -d -p 5080-5090:5080-5090 -e PORT=5080 -e PINECONE_HOST=localhost ghcr.io/pinecone-io/pinecone-local +# MongoDB Atlas Local: real $vectorSearch, on host port 27018 here. +docker run -d -p 27018:27017 mongodb/mongodb-atlas-local +``` + +```python +vd.connect("pinecone", api_key="pclocal", host="http://localhost:5080") +vd.connect("mongodb", uri="mongodb://localhost:27018/?directConnection=true") +``` + +Pinecone Local speaks the pre-2026-07 API, so it needs `pip install "pinecone<10"` +for now. turbopuffer has no emulator: use a throwaway namespace on a real +account. If Docker Hub refuses anonymous pulls (rate limit), prefix Docker Hub +images with Google's mirror, e.g. `mirror.gcr.io/mongodb/mongodb-atlas-local`. + ## 5. Connect ```python @@ -171,5 +191,6 @@ smoke_test(vd.connect("memory")) the live pricing pages; re-check there rather than quoting numbers. For contributors, the repo's `tests/docker-compose.yml` brings up pgvector, -Redis Stack, Elasticsearch, Weaviate and MongoDB Atlas Local together for the -live test suite. +Redis Stack, Elasticsearch, Weaviate, MongoDB Atlas Local, a Qdrant server and +Pinecone Local together for the live test suite, with no accounts +(`DOCKERHUB_MIRROR=mirror.gcr.io docker compose -f tests/docker-compose.yml up -d`).