ci: run SQL suite over subprocess + http + unix transports#1
Merged
Conversation
Parameterize ci/run-integration.sh by a TRANSPORT env (subprocess default, plus http and unix), and turn the integration job into a transport x os matrix so the haybarn SQL suite runs over all three VGI transports in CI. - run-integration.sh: keep the offline mocking (VGI_GOOGLE_MOCK=1 + a temp VGI_GOOGLE_DISCOVERY_DIR built from tests.mock_google) and export both so the booted worker inherits them for ALL transports. For http, inject `INSTALL httpfs FROM core; LOAD httpfs;` into the staged tests and boot the worker with `--http --port 0 --port-file`; for unix, boot with `--unix`. One cleanup() trap captures $? first, then removes the discovery dir AND kills the worker. Add a silent-skip guard (tee + PIPESTATUS) that fails the leg unless `All tests passed (N>0 assertions` and nothing was skipped. - pyproject.toml: add an `http` extra (vgi-python[http]) for waitress. - google_worker.py: PEP 723 header now depends on vgi-python[http]. - uv.lock: relocked with the http extra. - ci.yml: integration job is now transport: [subprocess, http, unix] x os: [ubuntu, macos], sets WORKER_CMD, `uv sync --frozen --extra http`. - ci/README.md: document the transport matrix, offline mocking across all transports, httpfs-on-http, the silent-skip guard, and the externalized pagination cursor (works over HTTP unchanged). The worker's externalized _ScanState cursor already round-trips over HTTP, so the full suite (including the pageToken page-boundary round-trip) is GREEN over all three transports with no worker changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds multi-transport (subprocess + http + unix) coverage for the haybarn SQL
suite, with the full suite GREEN over all three transports.
What changed
ci/run-integration.shis now parameterized byTRANSPORT(
subprocessdefault, plushttpandunix). The offline mocking(
VGI_GOOGLE_MOCK=1+ a tempVGI_GOOGLE_DISCOVERY_DIRbuilt fromtests.mock_google) is exported by the script and inherited by the workerfor all three transports — no separate mock server. For
httpit injectsINSTALL httpfs FROM core; LOAD httpfs;into the staged tests and boots theworker with
--http --port 0 --port-file <f>; forunixit boots with--unix <sock>. Onecleanup()trap captures$?first, then removes thediscovery dir and kills the worker. A silent-skip guard (tee +
PIPESTATUS) fails the leg unless the runner reportsAll tests passed (N>0 assertions …)and nothing was skipped.pyproject.toml: newhttpextra (vgi-python[http], waitress).google_worker.py: PEP 723 header depends onvgi-python[http].uv.lock: relocked with the http extra (uv sync --frozen --extra httpclean)..github/workflows/ci.yml: the integration job is now atransport: [subprocess, http, unix]×os: [ubuntu, macos]matrix; setsWORKER_CMD, runsuv sync --frozen --extra http. Test + lint jobs unchanged.ci/README.md: documents the transport matrix, offline mocking across alltransports, httpfs-on-http, the silent-skip guard, and the externalized
pagination cursor.
Worker
No worker code change needed: the worker's externalized
_ScanStatecursor(Google's
nextPageTokencarried in anArrowSerializableDataclass) alreadyround-trips correctly over the stateless HTTP transport, so the centerpiece
pageToken page-boundary round-trip (
google_drive(count := 5, page_size := 1)→ five distinct rows) passes over http unchanged.
Local validation
🤖 Generated with Claude Code