ci: run versioning_http + gzip_fallback http tests (new vgi-rpc-java features)

Implement the three remaining HTTP integration features in coordination with
vgi-rpc-java (b946c2d) and wire them into CI:

- Sticky cookies (attach/versioning_http.test): the versioned catalog sets a
  vgi_sticky cookie on catalog_attach and asserts it on catalog_version (via the
  new CallContext.cookies()/setCookie()), proving the C++ cookie jar round-trips
  Set-Cookie -> Cookie. The minimal versioned fixture now also exposes only its
  default schema (vgi-python parity; the test asserts exactly one schema).

- gzip fallback (http/gzip_fallback.test): a zstd-disabled http worker
  (VGI_HTTP_DISABLE_ZSTD) advertises gzip only; the C++ client 415s on zstd,
  reads VGI-Supported-Encodings: gzip, and retries. Runs as a dedicated step
  since it needs VGI_TEST_WORKER itself to be the special worker.

- Bearer auth (Main.buildHttpConfig + BearerAuthenticator): implemented and
  validated 10/11 against the vgi-python worker, but bearer_token.test isn't
  greenable under the prebuilt haybarn-unittest (the community C++ extension
  raises on the no-auth-ATTACH 401 instead of deferring it — fails against the
  Python reference worker identically), so it stays skipped.

CI builds vgirpc from source via the composite include (settings.gradle.kts now
honours VGI_RPC_JAVA_DIR; the workflow checks out the pinned vgi-rpc-java) until
a vgirpc release ships these features. Main suite: 177 pass / 7 skip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: rustyconover

.github/workflows/integration.yml

@@ -32,6 +32,10 @@ env:
   # The Haybarn release providing the prebuilt haybarn-unittest binary. Must be
   # ABI-compatible with the community-published vgi extension (both v1.5.3).
   HAYBARN_RELEASE: haybarn-v1.5.3-rc7
+  # The vgi-rpc-java commit built from source (composite include) for the worker's
+  # HTTP features. Drop once a vgirpc release with these features is published and
+  # pinned in vgi/build.gradle.kts.
+  VGI_RPC_JAVA_REF: b946c2de63e176e16ad73b533a4eebdcccddd16d
 
 jobs:
   integration:
@@ -56,6 +60,16 @@ jobs:
           ref: ${{ env.VGI_REF }}
           path: vgi-upstream
 
+      # Build the vgirpc RPC layer from source (composite include) so the worker
+      # picks up the HTTP features the integration suite needs (gzip negotiation,
+      # sticky cookies) ahead of the next published vgirpc release.
+      - name: Checkout pinned vgi-rpc-java
+        uses: actions/checkout@v4
+        with:
+          repository: Query-farm/vgi-rpc-java
+          ref: ${{ env.VGI_RPC_JAVA_REF }}
+          path: vgi-rpc-java
+
       - name: Set up JDK 25
         uses: actions/setup-java@v4
         with:
@@ -73,6 +87,8 @@ jobs:
 
       - name: Build example worker
         run: ./gradlew --no-daemon :vgi-example-worker:installDist
+        env:
+          VGI_RPC_JAVA_DIR: ${{ github.workspace }}/vgi-rpc-java
 
       - name: Download haybarn-unittest
         run: |

CLAUDE.md

@@ -269,20 +269,28 @@ without a C++ build:
   unittest invocation** (like `make test_launcher`) so the CI log streams the
   native sqllogictest report (per-test progress + `All tests passed (… N
   assertions in M test cases)` — ~8700 assertions across the ~185 files), not a
-  rolled-up count. Green on both lanes: **176 pass / 8 skip**. Sets
+  rolled-up count. Green on both lanes: **177 pass / 7 skip** (+
+  `http/gzip_fallback.test` in a dedicated step). Sets
   `VGI_REQUIRE_LAUNCHER_TRANSPORT=1` (we *are* the launcher transport) so
   `launcher/options_smoke.test` runs — which required `Main.runWorker` to accept
   the launcher cache-key / fixture flags (`--describe`/`--no-describe`/
   `--threaded`/`--quiet`/`--debug`/`--log-level`) as no-ops instead of exiting
-  on unknown argv. Also boots a versioned_tables catalog worker over HTTP and
-  sets `VGI_VERSIONED_TABLES_HTTP_WORKER` so the 4
-  `attach/versioned_tables_*_http` tests run (they attach an `http://` worker and
-  pass against the Java worker). The remaining http-only tests stay skipped —
-  they need Java worker HTTP features not yet ported: `versioning_http`
-  (sticky-session `vgi_sticky` cookies), `bearer_token` (HTTP bearer auth),
-  `gzip_fallback` (zstd-disable negotiation). The whole-suite-over-http
-  `TRANSPORT=http` path stays local-only (function-backed table-function
-  streaming, e.g. `example.sequence`, still errors over http). **Excludes**
+  on unknown argv. Boots versioned_tables + versioned catalog workers over HTTP
+  (`VGI_VERSIONED_TABLES_HTTP_WORKER`/`VGI_VERSIONED_HTTP_WORKER`) so the 4
+  `versioned_tables_*_http` tests and `versioning_http` (sticky-cookie
+  round-trip) run, and runs `gzip_fallback` against a dedicated zstd-disabled
+  http worker. These use **HTTP features added to vgi-rpc-java** (`b946c2d` —
+  gzip codec negotiation via `VGI-Supported-Encodings`/415,
+  `CallContext.cookies()`/`setCookie()`), which CI **builds from source** through
+  the composite include (`VGI_RPC_JAVA_DIR`/`VGI_RPC_JAVA_REF`) until a vgirpc
+  release ships them. `bearer_token.test` stays skipped: bearer auth IS
+  implemented (`Main.buildHttpConfig` + `BearerAuthenticator`, validated 10/11
+  against the vgi-python worker) but the test isn't greenable under the prebuilt
+  haybarn-unittest — the community C++ extension raises on the no-auth-ATTACH 401
+  instead of deferring it to the first query, which fails against the Python
+  reference worker identically. The whole-suite-over-http `TRANSPORT=http` path
+  stays local-only (function-backed table-function streaming, e.g.
+  `example.sequence`, still errors over http). **Excludes**
   `bool_in_union.test` in addition to `nested_type_combinations` — CI surfaced
   that it characterizes a **pre-existing platform-dependent union-bool bug**:
   the worker reads uninitialized memory for boolean union variants after row 1

ci/run-integration.sh

@@ -89,16 +89,18 @@ case "$TRANSPORT" in
     # We are the launcher transport, so opt into the launcher-only tests
     # (launcher/options_smoke.test) — matches vgi's `make test_launcher`.
     export VGI_REQUIRE_LAUNCHER_TRANSPORT=1
-    # Also serve the versioned_tables catalog over HTTP: the four
-    # attach/versioned_tables_*_http tests attach an http:// worker regardless of
-    # the main transport, and pass against the Java worker. (The other http-only
-    # tests need worker HTTP features the Java port doesn't yet implement —
-    # versioning_http needs sticky-session cookies, bearer_token needs bearer
-    # auth, gzip_fallback needs zstd-disable negotiation — so their env vars stay
-    # unset and they skip.)
+    # Serve the versioned_tables + versioned catalogs over HTTP too: the
+    # attach/versioned_tables_*_http and attach/versioning_http tests attach an
+    # http:// worker regardless of the main transport. versioning_http exercises
+    # the sticky-cookie round-trip (vgi_sticky). (bearer_token / gzip_fallback
+    # run as a separate step below since they need VGI_TEST_WORKER itself to be a
+    # specially-configured http worker.)
     vth_port="$(boot_http_worker "${HERE}/wrappers/vgi-worker-versioned-tables")"
     export VGI_VERSIONED_TABLES_HTTP_WORKER="http://localhost:${vth_port}"
     echo "versioned_tables http worker on ${VGI_VERSIONED_TABLES_HTTP_WORKER}"
+    vh_port="$(boot_http_worker "${HERE}/wrappers/vgi-worker-versioned")"
+    export VGI_VERSIONED_HTTP_WORKER="http://localhost:${vh_port}"
+    echo "versioned http worker on ${VGI_VERSIONED_HTTP_WORKER}"
     ;;
   http)
     # Whole-suite-over-HTTP. For LOCAL use — NOT yet a CI lane: the Java worker's
@@ -151,3 +153,15 @@ rm -f "$STAGE/test/_warm.test"
 # and fails the job (via `set -e`).
 echo "Running suite (single invocation — native sqllogictest report) ..."
 "$HAYBARN_UNITTEST" "test/sql/integration/*"
+
+# gzip_fallback.test needs VGI_TEST_WORKER itself to be a zstd-disabled HTTP
+# worker (it attaches it directly and asserts the gzip codec fallback). That
+# conflicts with the main suite's launch: worker, so run it as a dedicated
+# single-test invocation against its own worker. Only on the launch lane (shm is
+# the same coverage; the http lane is local-only).
+if [ "$TRANSPORT" = "launch" ]; then
+  echo "Running http/gzip_fallback.test (zstd-disabled http worker) ..."
+  gz_port="$(VGI_HTTP_DISABLE_ZSTD=1 boot_http_worker "$VGI_WORKER_BIN")"
+  VGI_TEST_WORKER="http://localhost:${gz_port}" VGI_HTTP_DISABLE_ZSTD=1 \
+    "$HAYBARN_UNITTEST" "test/sql/integration/http/gzip_fallback.test"
+fi

settings.gradle.kts

@@ -9,7 +9,11 @@ rootProject.name = "vgi-java"
 // Composite-include the sibling vgi-rpc-java repo so :vgirpc is built from
 // source; falls back gracefully if the directory isn't present (CI may pin
 // to a published artifact instead).
-val vgiRpcJavaDir = file("../Development/vgi-rpc-java")
+// VGI_RPC_JAVA_DIR overrides the path (CI checks out the sibling repo inside
+// the workspace and points here so it can build vgirpc from source — actions/
+// checkout can't write the default ../Development sibling path).
+val vgiRpcJavaDir = System.getenv("VGI_RPC_JAVA_DIR")?.let { file(it) }
+    ?: file("../Development/vgi-rpc-java")
 if (vgiRpcJavaDir.isDirectory) {
     includeBuild(vgiRpcJavaDir) {
         dependencySubstitution {

vgi-example-worker/src/main/java/farm/query/vgi/example/Main.java

@@ -5,6 +5,9 @@
 import farm.query.vgi.SettingSpec;
 import farm.query.vgi.internal.SchemaUtil;
 import farm.query.vgi.Worker;
+import farm.query.vgirpc.AuthContext;
+import farm.query.vgirpc.http.HttpServer;
+import farm.query.vgirpc.http.auth.BearerAuthenticator;
 import farm.query.vgi.example.scalar.AddValuesFunction;
 import farm.query.vgi.example.scalar.BinaryPacketFunction;
 import farm.query.vgi.example.scalar.AnyMixedFunctions;
@@ -1064,10 +1067,26 @@ private static void runWorker(Worker w, String[] args) {
             try { w.runUnixSocket(java.nio.file.Path.of(unixSocket), idleTimeoutMs); }
             catch (Exception e) { e.printStackTrace(); System.exit(1); }
         } else if (http) {
-            try { w.runHttp(host, port); }
+            try { w.runHttp(buildHttpConfig(host, port)); }
             catch (Exception e) { e.printStackTrace(); System.exit(1); }
         } else {
             w.runStdio();
         }
     }
+
+    /**
+     * Build the HTTP server config from the environment. {@code VGI_TEST_BEARER_TOKEN}
+     * turns on bearer-token auth (the server accepts only that token; the C++
+     * extension presents it via the {@code bearer_token} ATTACH option). Other
+     * HTTP-feature knobs ({@code VGI_HTTP_DISABLE_ZSTD}) are applied here too.
+     */
+    private static HttpServer.Config buildHttpConfig(String host, int port) {
+        HttpServer.Config.Builder cb = HttpServer.Config.builder().host(host).port(port);
+        String bearer = System.getenv("VGI_TEST_BEARER_TOKEN");
+        if (bearer != null && !bearer.isEmpty()) {
+            cb.authenticator(BearerAuthenticator.fromMap(
+                    Map.of(bearer, new AuthContext("bearer", true, "vgi-test", Map.of()))));
+        }
+        return cb.build();
+    }
 }

vgi/src/main/java/farm/query/vgi/VgiService.java

@@ -241,7 +241,8 @@ default farm.query.vgi.protocol.TableBufferingDestructorResponse table_buffering
      * @param transaction_opaque_data optional in-flight transaction handle
      * @return the monotonically increasing catalog version; default {@code 0}
      */
-    default CatalogVersionResponse catalog_version(byte[] attach_opaque_data, @Nullable byte[] transaction_opaque_data) {
+    default CatalogVersionResponse catalog_version(byte[] attach_opaque_data, @Nullable byte[] transaction_opaque_data,
+                                                    CallContext ctx) {
         return new CatalogVersionResponse(0L);
     }
 

vgi/src/main/java/farm/query/vgi/internal/VgiServiceImpl.java

@@ -747,7 +747,17 @@ public ItemsResponse catalog_catalogs() {
      * @return the catalog version response
      */
     @Override
-    public farm.query.vgi.protocol.CatalogVersionResponse catalog_version(byte[] attach_opaque_data, byte[] transaction_opaque_data) {
+    public farm.query.vgi.protocol.CatalogVersionResponse catalog_version(byte[] attach_opaque_data,
+            byte[] transaction_opaque_data, CallContext ctx) {
+        // Versioned fixture: assert the routing cookie set at ATTACH is echoed
+        // back. Over HTTP a populated cookie jar that's missing vgi_sticky means
+        // the client failed to plumb Set-Cookie -> Cookie. Over subprocess the
+        // cookie map is empty, so the check is skipped. (vgi-python parity.)
+        if ("versioned".equals(worker.catalogName()) && ctx != null
+                && !ctx.cookies().isEmpty() && !ctx.cookies().containsKey("vgi_sticky")) {
+            throw new IllegalStateException(
+                    "expected cookie 'vgi_sticky' on follow-up request; got " + ctx.cookies().keySet());
+        }
         return new farm.query.vgi.protocol.CatalogVersionResponse(1L);
     }
 
@@ -866,6 +876,14 @@ public CatalogAttachResult catalog_attach(CatalogAttachRequest request, CallCont
         Map<String, String> tags = new java.util.LinkedHashMap<>(worker.catalogTags() == null ? Map.of() : worker.catalogTags());
         if (resolvedImpl != null) tags.put("vgi_resolved_implementation_version", resolvedImpl);
         if (resolvedData != null) tags.put("vgi_resolved_data_version", resolvedData);
+        // Versioned fixture: pin an HTTP routing cookie so the C++ cookie jar
+        // round-trips Set-Cookie -> Cookie; catalog_version asserts it on every
+        // follow-up RPC (attach/versioning_http.test). No-op on non-HTTP
+        // transports (set_cookie has no sink there), matching vgi-python's
+        // VersionedCatalog.
+        if ("versioned".equals(worker.catalogName()) && ctx != null) {
+            ctx.setCookie("vgi_sticky", java.util.UUID.randomUUID().toString().replace("-", ""));
+        }
         return new CatalogAttachResult(
                 sealer.sealAttach(attachId, authOf(ctx)),
                 true, true, false,  // supports_transactions, supports_time_travel, catalog_version_frozen
@@ -1056,6 +1074,10 @@ private List<SchemaDesc> workerSchemas() {
         String defaultSchema = worker.defaultSchema();
         result.add(new SchemaDesc(defaultSchema,
                 comments.getOrDefault(defaultSchema, "Default schema")));
+        // The minimal `versioned` fixture exposes only its default schema
+        // (vgi-python parity); the example worker's auxiliary `data` schema is
+        // not part of it. attach/versioning_http.test asserts exactly one schema.
+        if ("versioned".equals(worker.catalogName())) return result;
         java.util.LinkedHashSet<String> extras = new java.util.LinkedHashSet<>();
         for (var t : worker.catalogTables()) {
             if (!defaultSchema.equals(t.schema())) extras.add(t.schema());