Skip to content

Commit 9930f3d

Browse files
rustyconoverclaude
rustyconover
and
claude
committed
Package for PyPI publishing with CI extension tests
- Depend on the published vgi-python[http]>=0.8.0 (PyPI) instead of the local ../vgi-python and ../vgi-rpc path sources; the real distribution is vgi-python (imports as vgi), not the unregistered "vgi". Drop the unused sentry and oauth extras. - Add pyproject.toml (hatchling): dist name vgi-easter, MIT license, and the vgi-easter / vgi-easter-http console scripts. Add MIT LICENSE. - Add serve.py:main() for the HTTP entry point. - Add ci/ harness (run-integration.sh + preprocess-require.awk) that runs the test/sql sqllogictest suite against the worker through the real signed vgi community extension via a prebuilt haybarn-unittest. - Add .github/workflows/ci.yml (unit + extension integration, reusable) and publish.yml (token-based PyPI publish gated on a green ci.yml run). - Refresh README.md / CLAUDE.md for install-from-PyPI and the CI design. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent fabe507 commit 9930f3d

11 files changed

Lines changed: 410 additions & 41 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Copyright 2026 Query Farm LLC - https://query.farm
2+
#
3+
# Unit tests + the sqllogictest extension suite (test/sql/*.test) run against
4+
# the easter worker through the real signed `vgi` DuckDB community extension via
5+
# a prebuilt standalone `haybarn-unittest`. See ci/README.md for the design.
6+
#
7+
# Reusable (workflow_call) so publish.yml can gate releases on a green run.
8+
name: CI
9+
10+
on:
11+
push:
12+
pull_request:
13+
workflow_dispatch:
14+
workflow_call:
15+
16+
permissions:
17+
contents: read
18+
19+
concurrency:
20+
group: ci-${{ github.ref }}
21+
cancel-in-progress: true
22+
23+
env:
24+
# Haybarn release providing the prebuilt haybarn-unittest binary. Must be
25+
# ABI-compatible with the community-published vgi extension. See ci/README.md.
26+
HAYBARN_RELEASE: haybarn-v1.5.3-rc10
27+
28+
jobs:
29+
unit:
30+
runs-on: ubuntu-latest
31+
steps:
32+
- uses: actions/checkout@v4
33+
- uses: astral-sh/setup-uv@v6
34+
- name: Run unit tests
35+
run: uv run --python 3.13 pytest tests/ -q
36+
37+
integration:
38+
runs-on: ubuntu-latest
39+
steps:
40+
- uses: actions/checkout@v4
41+
- uses: astral-sh/setup-uv@v6
42+
43+
- name: Install the easter worker
44+
run: |
45+
uv venv --python 3.13 .venv
46+
uv pip install --python .venv .
47+
48+
- name: Download haybarn-unittest
49+
run: |
50+
gh release download "$HAYBARN_RELEASE" \
51+
--repo Query-farm-haybarn/haybarn \
52+
--pattern 'haybarn_unittest-linux-amd64.zip' \
53+
--output /tmp/haybarn-unittest.zip --clobber
54+
unzip -o -q /tmp/haybarn-unittest.zip -d /tmp/haybarn-unittest
55+
UNITTEST=$(find /tmp/haybarn-unittest -name 'haybarn-unittest' -type f | head -1)
56+
chmod +x "$UNITTEST"
57+
echo "HAYBARN_UNITTEST=$UNITTEST" >> "$GITHUB_ENV"
58+
env:
59+
GH_TOKEN: ${{ github.token }}
60+
61+
- name: Run extension integration suite
62+
run: ci/run-integration.sh
63+
env:
64+
HAYBARN_UNITTEST: ${{ env.HAYBARN_UNITTEST }}
65+
VGI_EASTER_WORKER: ${{ github.workspace }}/.venv/bin/vgi-easter

‎.github/workflows/publish.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Copyright 2026 Query Farm LLC - https://query.farm
2+
#
3+
# Token-based publishing (no OIDC / trusted publishing). The PyPI API token is
4+
# stored as the `PYPI_API_TOKEN` repository secret and passed to `uv publish`
5+
# via UV_PUBLISH_TOKEN (uv uses the `__token__` username automatically).
6+
#
7+
# Publishes when a GitHub Release is published; `workflow_dispatch` allows a
8+
# manual run from the Actions tab. The full CI suite (unit tests + the vgi
9+
# extension integration suite) must pass before anything is uploaded.
10+
name: Publish to PyPI
11+
12+
on:
13+
release:
14+
types: [published]
15+
workflow_dispatch:
16+
17+
permissions:
18+
contents: read
19+
20+
jobs:
21+
# Run the same unit + extension-integration suite as on push/PR, so a release
22+
# can't publish without the worker passing against the real vgi extension.
23+
ci:
24+
uses: ./.github/workflows/ci.yml
25+
26+
publish:
27+
needs: ci
28+
runs-on: ubuntu-latest
29+
steps:
30+
- uses: actions/checkout@v4
31+
- uses: astral-sh/setup-uv@v6
32+
- name: Build wheel and sdist
33+
run: uv build
34+
- name: Publish to PyPI
35+
env:
36+
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
37+
run: uv publish

‎CLAUDE.md‎

Lines changed: 43 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -48,20 +48,30 @@ classmethod whose params/return are annotated:
4848
- null handling is manual: `compute` iterates `year.to_pylist()` and maps
4949
`None -> None`.
5050

51-
## Dependencies & Python version
51+
## Packaging, dependencies & Python version
5252

53-
Requires **Python 3.13+**, managed with `uv`. Deps are declared inline as
54-
PEP 723 script metadata in `easter_worker.py` and `serve.py`:
53+
Requires **Python 3.13+**, managed with `uv`. This repo is a published PyPI
54+
distribution named **`vgi-easter`** (hatchling build; see `pyproject.toml`). The
55+
sole dependency is **`vgi-python[http]`** (PyPI; imports as `vgi`) — the `http`
56+
extra pulls in the HTTP-server transport. The distribution name on PyPI is
57+
`vgi-python`, *not* `vgi` (which is unregistered); the inline path sources for
58+
the sibling checkouts have been removed.
5559

56-
```python
57-
# dependencies = ["vgi[http,oauth]", "vgi-rpc[sentry]"]
58-
# [tool.uv.sources]
59-
# vgi = { path = "../vgi-python" }
60-
# vgi-rpc = { path = "../vgi-rpc" }
61-
```
60+
`pyproject.toml` packages the two flat modules (`easter_worker.py`, `serve.py`)
61+
explicitly via `[tool.hatch.build.targets.wheel] include` and registers two
62+
console scripts:
63+
64+
- `vgi-easter` → `easter_worker:main` (stdio transport)
65+
- `vgi-easter-http` → `serve:main` (HTTP server)
6266

63-
In development, `vgi` and `vgi-rpc` resolve against the sibling checkouts
64-
`~/Development/vgi-python` and `~/Development/vgi-rpc`.
67+
The same modules also carry inline PEP 723 metadata
68+
(`vgi-python[http]>=0.8.0`), so `uv run easter_worker.py` works from a checkout
69+
without installing.
70+
71+
```bash
72+
uv build # build wheel + sdist into dist/
73+
uv publish # upload to PyPI
74+
```
6575

6676
## Commands
6777

@@ -72,10 +82,10 @@ uv run --python 3.13 easter_worker.py
7282
# Run the HTTP server
7383
VGI_SIGNING_KEY=dev uv run --python 3.13 serve.py --host 0.0.0.0 --port 8000
7484

75-
# Unit tests (pytest). The --rootdir/-o flags stop pytest from picking up an
76-
# upstream pyproject that injects --mypy --ruff.
85+
# Unit tests (pytest). vgi-python resolves from PyPI; -o "addopts=" guards
86+
# against any inherited pytest addopts.
7787
uv run --python 3.13 \
78-
--with pytest --with pyarrow --with ../vgi-python --with ../vgi-rpc \
88+
--with pytest --with vgi-python \
7989
pytest tests/ --rootdir=. -o "addopts=" -q
8090
```
8191

@@ -107,13 +117,30 @@ sqllogictest files exercised through the real DuckDB VGI extension, gated on
107117
Run them with the DuckDB `unittest` binary built with the VGI extension, with
108118
`VGI_EASTER_WORKER` set to a worker LOCATION (stdio command or HTTP URL).
109119

120+
In CI this is automated without a C++ build: `ci/run-integration.sh` drives a
121+
prebuilt standalone `haybarn-unittest` and installs the signed `vgi` extension
122+
from the community channel, with `ci/preprocess-require.awk` rewriting each
123+
`require <ext>` into `INSTALL/LOAD`. `.github/workflows/ci.yml` runs the unit +
124+
integration suite on push/PR and is reused by `publish.yml` so nothing reaches
125+
PyPI without a green extension run. See `ci/README.md`.
126+
127+
### CI / publishing
128+
129+
- `.github/workflows/ci.yml` — unit tests + extension integration suite
130+
(reusable via `workflow_call`).
131+
- `.github/workflows/publish.yml` — on GitHub Release (or manual dispatch),
132+
runs `ci.yml` then `uv build && uv publish`. Token-based, no trusted
133+
publishing: needs the `PYPI_API_TOKEN` repo secret (passed as
134+
`UV_PUBLISH_TOKEN`).
135+
110136
## ATTACH syntax
111137

112138
The VGI extension auto-detects transport from LOCATION:
113139

114140
```sql
115-
-- stdio: DuckDB spawns the worker
116-
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'uv run --python 3.13 easter_worker.py');
141+
-- stdio: DuckDB spawns the worker (installed command, or `uvx vgi-easter`;
142+
-- from a checkout, `uv run --python 3.13 easter_worker.py`)
143+
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'uvx vgi-easter');
117144

118145
-- HTTP: worker running as a server (requires httpfs, which the extension auto-loads)
119146
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'http://localhost:8000');

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright 2026 Query Farm LLC - https://query.farm
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 43 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ that computes the date of **Western (Gregorian) Easter Sunday** for a given year
55
and exposes it to DuckDB as a SQL scalar function.
66

77
```sql
8-
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'uv run --python 3.13 easter_worker.py');
8+
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'uvx vgi-easter');
99

1010
SELECT easter_date(2025);
1111
-- 2025-04-20
@@ -51,33 +51,45 @@ The entire implementation lives in [`easter_worker.py`](easter_worker.py):
5151
## Requirements
5252

5353
- Python **3.13+**
54-
- [`uv`](https://docs.astral.sh/uv/) for dependency management
54+
- [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
5555

56-
Dependencies are declared inline as [PEP 723](https://peps.python.org/pep-0723/)
57-
script metadata in `easter_worker.py` and `serve.py`. During development they
58-
resolve against the sibling checkouts `../vgi-python` and `../vgi-rpc`.
56+
The only dependency is [`vgi-python`](https://pypi.org/project/vgi-python/) (the
57+
`http` extra adds the HTTP-server transport); it is published on PyPI, so no
58+
sibling checkouts are needed.
59+
60+
## Installing
61+
62+
```bash
63+
# Install from PyPI (provides the vgi-easter and vgi-easter-http commands)
64+
pip install vgi-easter
65+
# or run it ad hoc without installing
66+
uvx vgi-easter
67+
```
5968

6069
## Running
6170

62-
The worker supports both VGI transports.
71+
The worker supports both VGI transports. Two console scripts are installed:
72+
`vgi-easter` (stdio) and `vgi-easter-http` (HTTP server).
6373

6474
### stdio (DuckDB spawns the worker)
6575

6676
DuckDB runs the worker as a subprocess and talks to it over stdin/stdout. No
67-
server to manage:
77+
server to manage — point the LOCATION at the installed command (or `uvx
78+
vgi-easter` to fetch it on demand):
6879

6980
```sql
70-
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'uv run --python 3.13 easter_worker.py');
81+
ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'uvx vgi-easter');
7182
SELECT easter_date(2025);
7283
DETACH easter;
7384
```
7485

7586
### HTTP
7687

77-
Start the worker as an HTTP server (`serve.py` calls `EasterWorker.main_http()`):
88+
Start the worker as an HTTP server (`vgi-easter-http` calls
89+
`EasterWorker.main_http()`):
7890

7991
```bash
80-
VGI_SIGNING_KEY=dev uv run --python 3.13 serve.py --host 0.0.0.0 --port 8000
92+
VGI_SIGNING_KEY=dev vgi-easter-http --host 0.0.0.0 --port 8000
8193
```
8294

8395
Then attach over HTTP (the VGI extension auto-loads `httpfs`):
@@ -87,6 +99,16 @@ ATTACH 'easter' AS easter (TYPE vgi, LOCATION 'http://localhost:8000');
8799
SELECT easter_date(2025);
88100
```
89101

102+
### From a source checkout
103+
104+
The two modules also carry inline [PEP 723](https://peps.python.org/pep-0723/)
105+
metadata, so you can run them directly without installing:
106+
107+
```bash
108+
uv run --python 3.13 easter_worker.py # stdio
109+
VGI_SIGNING_KEY=dev uv run --python 3.13 serve.py --host 0.0.0.0 --port 8000 # HTTP
110+
```
111+
90112
## Testing
91113

92114
### Unit tests (pytest)
@@ -97,7 +119,7 @@ propagation:
97119

98120
```bash
99121
uv run --python 3.13 \
100-
--with pytest --with pyarrow --with ../vgi-python --with ../vgi-rpc \
122+
--with pytest --with vgi-python \
101123
pytest tests/ --rootdir=. -o "addopts=" -q
102124
```
103125

@@ -128,6 +150,15 @@ LOCATION (a stdio command or an HTTP URL) and run them with the DuckDB
128150
| `VGI_HTTP_PORT` / `VGI_HTTP_HOST` | HTTP bind address (defaults: `8000` / all interfaces). |
129151
| `VGI_WORKER_DEBUG` | Set to `1` for debug logging. |
130152

153+
## Publishing
154+
155+
This repo is a packaged distribution (`vgi-easter`) built with hatchling:
156+
157+
```bash
158+
uv build # writes dist/*.whl and dist/*.tar.gz
159+
uv publish # upload to PyPI (needs a token)
160+
```
161+
131162
## License
132163

133-
Copyright © Query.Farm. All rights reserved.
164+
MIT — see [LICENSE](LICENSE). Copyright 2026 Query Farm LLC — https://query.farm

‎ci/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# CI: the easter extension integration suite
2+
3+
[`.github/workflows/ci.yml`](../.github/workflows/ci.yml) runs the unit tests
4+
and this repo's sqllogictest suite (`test/sql/*.test`) against the easter VGI
5+
worker through the **real DuckDB `vgi` extension** on every push / PR — and is
6+
reused by [`publish.yml`](../.github/workflows/publish.yml) so nothing reaches
7+
PyPI without a green extension run.
8+
9+
## How it works (no C++ build)
10+
11+
Rather than building the vgi DuckDB extension from source, CI drives a
12+
**prebuilt** standalone `haybarn-unittest` (the DuckDB/Haybarn sqllogictest
13+
runner, published in Haybarn's releases) and installs the **signed** vgi
14+
extension from the Haybarn community channel:
15+
16+
1. **Install the worker** — `uv pip install .` into a venv. The `vgi-easter`
17+
console-script (with its venv shebang) is a self-contained stdio worker the
18+
extension can spawn.
19+
2. **Download the runner** — `haybarn_unittest-linux-amd64.zip` from the pinned
20+
Haybarn release.
21+
3. **Preprocess** — the standalone runner links none of the extensions the
22+
tests gate on, so [`preprocess-require.awk`](preprocess-require.awk) rewrites
23+
each `require <ext>` into an explicit signed `INSTALL <ext> FROM
24+
{community,core}; LOAD <ext>;`. `require-env` and everything else pass
25+
through untouched.
26+
4. **Run** — [`run-integration.sh`](run-integration.sh) stages the preprocessed
27+
tree, points `VGI_EASTER_WORKER` at the installed `vgi-easter` command, warms
28+
the extension cache once, then runs the suite in a single `unittest`
29+
invocation. The CI log streams the runner's native report; any failed
30+
assertion exits non-zero and fails the job.
31+
32+
## Run it locally
33+
34+
```bash
35+
uv venv .venv --python 3.13 && uv pip install --python .venv .
36+
gh release download haybarn-v1.5.3-rc10 --repo Query-farm-haybarn/haybarn \
37+
--pattern 'haybarn_unittest-osx-arm64.zip' --output /tmp/hb.zip --clobber
38+
unzip -o /tmp/hb.zip -d /tmp/hb
39+
HAYBARN_UNITTEST=/tmp/hb/haybarn-unittest \
40+
VGI_EASTER_WORKER="$PWD/.venv/bin/vgi-easter" \
41+
ci/run-integration.sh
42+
```
43+
44+
(Swap the asset pattern for your platform: `haybarn_unittest-linux-amd64.zip`
45+
on CI.)
46+
47+
## Version pin (and its coupling)
48+
49+
`HAYBARN_RELEASE` in [`ci.yml`](../.github/workflows/ci.yml) pins the Haybarn
50+
release supplying `haybarn-unittest`; it must be ABI-compatible with the
51+
community-published `vgi` extension. The vgi extension is pulled live from the
52+
community channel (`INSTALL vgi FROM community`), which always serves the
53+
currently published build — so CI verifies the worker against what users can
54+
actually install today. Bump the pin deliberately and re-run the suite.

‎ci/preprocess-require.awk‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Copyright 2026 Query Farm LLC - https://query.farm
2+
#
3+
# Rewrite each `require <ext>` gate in this repo's sqllogictest files into an
4+
# explicit signed INSTALL+LOAD, so the prebuilt standalone `haybarn-unittest`
5+
# (which links none of these extensions) can run the suite. The vgi extension
6+
# comes from the signed community channel; httpfs/json/parquet/spatial from the
7+
# signed core channel. `require-env` and every other directive pass through
8+
# untouched. See ci/README.md.
9+
/^require[ \t]+vgi[ \t]*$/ {
10+
print "statement ok"; print "INSTALL vgi FROM community;"; print "";
11+
print "statement ok"; print "LOAD vgi;"; next
12+
}
13+
/^require[ \t]+(httpfs|json|parquet|spatial)[ \t]*$/ {
14+
ext = $2
15+
print "statement ok"; print "INSTALL " ext " FROM core;"; print "";
16+
print "statement ok"; print "LOAD " ext ";"; next
17+
}
18+
{ print }

0 commit comments

Comments
 (0)