Initial commit: vgi-google — query Google APIs from SQL

A VGI worker (Python) exposing Google APIs as DuckDB table functions:

- Discovery-backed client factory (google-api-python-client + google-auth),
  building a client for any Google REST API from its discovery document, with
  a generic dotted-method caller, per-call timeout, and bounded retry/backoff
  on 429/5xx and rateLimitExceeded/quotaExceeded.
- Curated READ adapters as table functions: google_sheet, google_drive,
  google_calendar, google_youtube — each mapping the API response to a clean,
  typed Arrow schema (TIMESTAMPTZ / LIST<VARCHAR> / JSON-tagged columns).
- Generic escape hatch google_call(api, version, method, params_json) and the
  discovery helpers google_apis() / google_methods(api, version).
- Service-account-first auth (API key for public APIs) via the VGI secret
  provider, never inline; least-privilege scopes declared per adapter.
- Pagination via Google's nextPageToken carried as serializable scan state,
  one page per process tick; paged functions pinned to a single worker so a
  parallel scan cannot re-emit and duplicate rows.

Tested with pytest (parsers + auth + discovery + mock E2E) and a haybarn SQL
E2E driven against a canned, paginated transport (no live Google in the CI
gate). The centerpiece asserts the pageToken scan-state round-trips across a
batch boundary (every row exactly once), with a companion test showing why the
single-worker pin prevents duplication. ruff + mypy clean.

v1 is READ-only; Sheets write and the OAuth-user flow are documented follow-ups.
Worker MIT; google-api-python-client / google-auth are Apache-2.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: rustyconover

.github/workflows/ci.yml

@@ -0,0 +1,96 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+    paths-ignore: ['README.md']
+  pull_request:
+    branches: [main]
+    paths-ignore: ['README.md']
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Python ${{ matrix.python-version }} on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+        python-version: ["3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      # Install from PyPI rather than the local ../vgi-python path used for
+      # `uv run` during development, so CI is self-contained.
+      - name: Install dependencies
+        run: |
+          uv venv
+          uv pip install vgi-python "google-api-python-client>=2.100" "google-auth>=2.20" pyarrow pytest
+          uv pip install -e . --no-deps
+
+      # Parser + auth + discovery + mock E2E; NO live Google API.
+      - name: Run unit tests
+        run: uv run --no-sync pytest
+
+  e2e:
+    name: SQL end-to-end (haybarn, mock transport)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+
+      - name: Set up Python 3.13
+        run: uv python install 3.13
+
+      # Self-contained: install runtime deps into a venv from PyPI (not the local
+      # ../vgi-python dev path), so the worker can be launched as
+      # `.venv/bin/python google_worker.py` (plain python ignores the PEP-723
+      # header that pins the dev path).
+      - name: Install worker dependencies
+        run: |
+          uv venv
+          uv pip install vgi-python "google-api-python-client>=2.100" "google-auth>=2.20" pyarrow
+          uv pip install -e . --no-deps
+
+      - name: Install haybarn-unittest
+        run: |
+          uv tool install haybarn-unittest
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+
+      # Driven against the canned mock transport (VGI_GOOGLE_MOCK=1) — no live API.
+      - name: Run SQL end-to-end tests
+        run: make test-sql WORKER_STDIO=".venv/bin/python google_worker.py"
+
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+
+      - name: Install tooling
+        run: |
+          uv venv
+          uv pip install vgi-python "google-api-python-client>=2.100" "google-auth>=2.20" pyarrow ruff mypy
+          uv pip install -e . --no-deps
+
+      - name: Check linting
+        run: uv run --no-sync ruff check .
+
+      - name: Type check (mypy)
+        run: uv run --no-sync mypy vgi_google

.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.pyc
+.venv/
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+.claude/
+vendor/
+duckdb_unittest_tempdir/
+*.coverage
+.coverage
+.cache/
+uv.lock

.python-version

@@ -0,0 +1 @@
+3.13

CLAUDE.md

@@ -0,0 +1,120 @@
+# CLAUDE.md — vgi-google
+
+Guidance for working in this repo. vgi-google is a **VGI worker** (Python) that
+queries **Google APIs from DuckDB/SQL**: a discovery-driven core, curated table
+adapters (Sheets / Drive / Calendar / YouTube), and a generic `google_call`
+escape hatch. It is an **egress / commodity** connector — the value is upstream
+in Google's APIs (quotas/billing), not in the worker; the durable cleverness is
+the discovery-driven breadth, not a moat. Keep the README's honest framing
+intact.
+
+## What this is
+
+- A VGI worker on the `vgi-python` SDK, launched by DuckDB (`ATTACH ... (TYPE vgi,
+  LOCATION '...')`) as a stdio subprocess, or over HTTP via `serve.py`.
+- The SQL surface is seven **table functions** in the `google` catalog assembled
+  in `google_worker.py`: `google_sheet`, `google_drive`, `google_calendar`,
+  `google_youtube`, `google_call`, `google_apis`, `google_methods`.
+- v1 is **READ only**. Sheets write and the OAuth-user flow are documented
+  follow-ups, NOT built.
+
+## Layout
+
+- `vgi_google/discovery.py` — the discovery-backed client factory
+  (`build_service` via `google-api-python-client`'s `build` / `build_from_document`),
+  generic dotted-method traversal (`resolve_method`), `execute` (timeout +
+  bounded 429/5xx/quota retry), and discovery introspection (`list_apis`,
+  `fetch_discovery_doc`, `list_methods`).
+- `vgi_google/auth.py` — credential resolution (service-account-first, then API
+  key) via the VGI secret provider; env hatches for tests only.
+- `vgi_google/client.py` — `build_client(api, version, secrets, scopes)` ties auth
+  + discovery together, and holds the **test seam** `set_http_factory`.
+- `vgi_google/adapters/*` — one curated adapter per API (`api`/`version`/`scopes`/
+  `schema` + `fetch_page(service, args, cursor) -> Page`).
+- `vgi_google/tables.py` — the table functions; `_run_paged_tick` is the shared
+  one-page-per-tick driver; `_ScanState` is the serializable pagination cursor.
+- `vgi_google/schema_utils.py` — `afield` / `json_field` / `rows_to_batch` and
+  the pinned `TIMESTAMPTZ` / `LIST_VARCHAR` types + `parse_*` helpers.
+
+## VGI conventions that matter here
+
+- **Table functions take `name := value` named args; scalars are positional-only.**
+  Every function here is a table function. Positional args CANNOT have a default
+  (DuckDB always binds them) — `google_call`'s `params_json` is positional and
+  required; everything optional is a named `Arg("name", default=...)`.
+- **LIST / TIMESTAMPTZ / JSON returns REQUIRE explicit `arrow_type`.** Pinned once
+  in `schema_utils.py`. JSON columns are plain string Arrow fields tagged with
+  `metadata={b"logical_type": b"JSON"}`; read them in SQL with the `json`
+  extension's `->>` (the `.test` file `LOAD json;` first).
+- **Do NOT request secrets in `on_bind`.** Using `@bind_fixed_schema` AND calling
+  `params.secrets.get(...)` in a custom `on_bind` triggers a two-phase bind retry
+  that returns an EMPTY output schema (every batch comes back 0-row). Resolve
+  secrets lazily in `process()` via `params.secrets` instead — `auth.resolve`
+  tolerates a missing/empty accessor. (`google_call` keeps an `on_bind` only to
+  validate `params_json`, and does not touch secrets there.)
+- **Scan state must be serializable.** `_ScanState` extends
+  `ArrowSerializableDataclass`; the framework round-trips Google's
+  `nextPageToken` across `process()` ticks (and HTTP requests). One page per
+  tick; `count` caps total rows.
+- **Pin paged functions to one worker.** Every paged table function is
+  `@init_single_worker`. Without it, parallel scan instances each re-emit the
+  whole result and DUPLICATE rows (the bug that bit vgi-search / vgi-wikipedia).
+  `tests/test_mock_e2e.py::test_single_worker_pin_prevents_duplication` documents
+  why.
+- **Never crash the worker.** `discovery.execute` raises `GoogleApiError`;
+  `auth.resolve` raises `AuthError`; `process()` converts both (and `ValueError`)
+  into a clean `RuntimeError` → DuckDB error.
+
+## The discovery / test seam
+
+`build_from_document` builds a real client from a static discovery doc, and an
+injectable `httplib2`-style transport serves canned responses — so the SAME
+adapter code runs in tests as live. `client.set_http_factory(factory)` installs
+`(api, version) -> (http, discovery_doc)`; tests use `tests/mock_google.py`'s
+`http_factory`. The worker subprocess enables it via `VGI_GOOGLE_MOCK=1` (see
+`google_worker._maybe_install_mock`). `google_apis`/`google_methods` read static
+docs from `VGI_GOOGLE_DISCOVERY_DIR`.
+
+Note: `execute` json-decodes any raw bytes/str body, so adapters get a dict even
+when a minimal discovery doc omits a method's `response` schema.
+
+## Auth
+
+Service-account-first, via `params.secrets` only — never inline. Secret types:
+`google_service_account` (JSON key; optional `subject` for delegation, `scopes`
+to narrow) and `google_api_key`. Each adapter declares least-privilege `scopes`
+(documented in the README table). Env hatches `VGI_GOOGLE_SERVICE_ACCOUNT_FILE` /
+`VGI_GOOGLE_API_KEY` are for tests/local only.
+
+## Adding a curated adapter
+
+1. `vgi_google/adapters/<name>.py`: a class with `api`/`version`/`scopes`/`schema`
+   and `fetch_page(service, args, cursor) -> Page` mapping the API response to
+   dict rows + `next_cursor` (Google's `nextPageToken`).
+2. Register it in `adapters/__init__.py`; add a `Google<Name>Function` in
+   `tables.py` (`@init_single_worker @bind_fixed_schema`, a `_<Name>Args`
+   dataclass, `process` delegating to `_run_paged_tick`) and to `TABLE_FUNCTIONS`.
+3. Add a discovery doc to `tests/mock_google.py` `DISCOVERY_DOCS` + canned
+   responses, a parser test in `tests/test_adapters.py`, and `.test` coverage.
+
+## Testing (NO live Google in the CI gate)
+
+- `make test-unit` / `pytest` — parser tests (`test_adapters.py`), auth
+  (`test_auth.py`), discovery (`test_discovery.py`), mock E2E (`test_mock_e2e.py`,
+  incl. the pageToken round-trip + single-worker-pin proofs).
+- `make test-sql` — the real worker subprocess under DuckDB via `haybarn-unittest`,
+  launched with `VGI_GOOGLE_MOCK=1` by `scripts/run_sql_e2e.py`. The `.test` file
+  uses `LOAD vgi;` (NEVER `require vgi`), `require-env VGI_GOOGLE_WORKER`, ATTACHes
+  `${VGI_GOOGLE_WORKER}`, and globs `test/sql/*`.
+- `make lint` (ruff) + `make typecheck` (mypy `vgi_google`) must be clean.
+- Live smoke (`tests/test_live_smoke.py`) is GATED by `VGI_GOOGLE_LIVE=1`, not in
+  the CI gate.
+
+When changing pagination or a schema, re-run BOTH `make test-unit` and
+`make test-sql` — the scan-state round-trip is asserted in both.
+
+## Licensing
+
+Worker code is MIT (`LICENSE`). `google-api-python-client` and `google-auth` are
+Apache-2.0; httplib2 / google-auth-httplib2 are permissive. No GPL/AGPL. Quotas,
+billing and Google's ToS are the user's responsibility.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Query Farm LLC https://query.farm
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,52 @@
+# vgi-google worker -- dev and test targets.
+#
+# Usage:
+#   make test        # unit (pytest) + SQL end-to-end (haybarn-unittest, mock transport)
+#   make test-unit   # pytest only (fixture parsers + mock E2E, no live network)
+#   make test-sql    # SQL end-to-end only (haybarn glob, driven against the mock transport)
+#   make lint        # ruff
+#   make typecheck   # mypy
+#
+# The SQL suite drives the worker as a real subprocess over stdio: haybarn-unittest
+# ATTACHes `${VGI_GOOGLE_WORKER}`, then runs the .test files in test/sql/. The
+# worker is launched with VGI_GOOGLE_MOCK=1 so its adapters build from static
+# discovery docs and are served canned, paginated responses (tests/mock_google.py),
+# so NOTHING in the CI gate touches a live Google API.
+
+# Worker stdio command (overridable). The PEP-723 header in google_worker.py
+# pins the google client libs, so `uv run` gives the worker its dependencies.
+WORKER_STDIO   ?= uv run --python 3.13 google_worker.py
+
+# haybarn-unittest: the DuckDB sqllogictest runner (uv tool install haybarn-unittest).
+HAYBARN        ?= haybarn-unittest
+TEST_DIR        = .
+TEST_PATTERN    = test/sql/*
+
+.PHONY: test test-unit test-sql pytest lint typecheck
+
+test: test-unit test-sql
+
+test-unit: pytest
+
+pytest:
+	uv run --no-sync pytest -q
+
+# End-to-end SQL: a tiny Python launcher enables the mock transport (and a temp
+# discovery dir for the discovery helpers), then runs the haybarn glob with the
+# worker command exported.
+test-sql:
+	@command -v $(HAYBARN) >/dev/null 2>&1 || { \
+		echo "ERROR: $(HAYBARN) not found. Install it with:" >&2; \
+		echo "  uv tool install haybarn-unittest" >&2; \
+		echo "  (then ensure ~/.local/bin is on PATH)" >&2; \
+		exit 1; \
+	}
+	VGI_GOOGLE_WORKER="$(WORKER_STDIO)" \
+	HAYBARN="$(HAYBARN)" TEST_DIR="$(TEST_DIR)" TEST_PATTERN="$(TEST_PATTERN)" \
+		uv run --no-sync python scripts/run_sql_e2e.py
+
+lint:
+	uv run --no-sync ruff check .
+
+typecheck:
+	uv run --no-sync mypy vgi_google