feat(deltagraph): Bolt e2e boot test + QUICKSTART docs (Phase 4.3) (#337)

genezhang · claude · web-flow · commit 47185348d043 · 2026-05-16T20:38:06.000-07:00
Co-authored-by: Claude Opus 4.7 &lt;noreply@anthropic.com&gt;
diff --git a/Cargo.toml b/Cargo.toml
@@ -118,3 +118,12 @@ path = "tests/rust/e2e/mod.rs"
 name = "deltagraph_bin"
 path = "tests/rust/bin/deltagraph_smoke.rs"
 required-features = ["databricks"]
+
+# Bolt-level boot regression for `deltagraph` (Phase 4.3). Spawns the
+# subprocess against wiremock and completes the Bolt v5 handshake to
+# prove the server boots end-to-end in Databricks mode without
+# tripping any ClickHouse-specific init.
+[[test]]
+name = "deltagraph_bolt_e2e"
+path = "tests/rust/bin/deltagraph_bolt_e2e.rs"
+required-features = ["databricks"]
diff --git a/docs/deltagraph/QUICKSTART.md b/docs/deltagraph/QUICKSTART.md
@@ -0,0 +1,216 @@
+# DeltaGraph Quickstart
+
+DeltaGraph turns a Databricks SQL Warehouse into a Cypher-queryable graph
+database. Same Cypher you'd send to Neo4j; the server translates it to
+Spark SQL and executes against your warehouse over the Statement
+Execution API. Neo4j Browser, NeoDash, graph-notebook, and any other
+Bolt v5 client connect unchanged.
+
+This document walks through the manual setup: build, configure, point
+Neo4j Browser at the server, run sample Cypher.
+
+> ⚠️ **Status:** DeltaGraph ships in `v0.6.7-dev`. Phases 1.x–4.3 are
+> complete: dialect routing, executor, embedded API, FFI, `cg` CLI,
+> server binary, Bolt e2e. Phase 3 (catalog discovery) and Phase 5
+> (full release) are still ahead. Treat this as an early-adopter path
+> until v0.7.0 lands.
+
+## Prerequisites
+
+- A Databricks workspace with at least one **SQL Warehouse** running.
+- A **Personal Access Token** (PAT) with `SELECT` on the catalog/schema
+  you want to query. OAuth M2M is on the roadmap; PAT is the only auth
+  method today.
+- A **schema YAML** describing how your tables map to a graph
+  (`benchmarks/social_network/schemas/social_benchmark.yaml` is the
+  smallest example in the repo).
+- Rust 1.85+ to build from source. We do not yet ship pre-built
+  `deltagraph` release artifacts.
+
+## 1. Build
+
+```bash
+cargo build --release --features databricks --bin deltagraph
+```
+
+The `databricks` feature pulls in the executor and its `reqwest`
+client. The default build (no feature) produces only `clickgraph`,
+which talks to ClickHouse — that binary is unaffected by this work.
+
+## 2. Configure the environment
+
+The server reads these env vars at startup:
+
+| Variable                     | Required | Purpose                                                              |
+| ---------------------------- | -------- | -------------------------------------------------------------------- |
+| `DATABRICKS_HOST`            | yes      | Workspace hostname, no scheme. `dbc-abc123-def4.cloud.databricks.com` |
+| `DATABRICKS_WAREHOUSE_ID`    | yes      | Target SQL Warehouse ID (find under SQL Warehouses in the UI)         |
+| `DATABRICKS_TOKEN`           | yes      | Personal access token. **Env-only; never accepted as a flag.**       |
+| `DATABRICKS_CATALOG`         | no       | Default catalog for unqualified table names                          |
+| `DATABRICKS_SCHEMA`          | no       | Default schema for unqualified table names                           |
+| `GRAPH_CONFIG_PATH`          | yes      | Path to your graph-schema YAML                                       |
+
+The token deliberately has no CLI flag — exposing it on the command
+line would leak via `ps`, shell history, and CI log uploads.
+
+## 3. Start the server
+
+```bash
+./target/release/deltagraph
+```
+
+You'll see:
+
+```
+DeltaGraph v0.6.7-dev
+
+🧱 DeltaGraph mode: routing queries through a Databricks SQL Warehouse
+✓ Schema initialization complete (YAML mode, 1 schema(s) registered)
+✓ Successfully bound HTTP listener to 0.0.0.0:7475
+Successfully bound Bolt listener to 0.0.0.0:7687
+ClickGraph server is running
+  HTTP API: http://0.0.0.0:7475
+  Bolt Protocol: bolt://0.0.0.0:7687
+```
+
+By default the server starts in Neo4j compat mode (so Neo4j Browser
+recognises it as a Neo4j 5.x server) and accepts unauthenticated
+connections. The compat mode is the headline feature for the Browser
+demo; pass `--disable-neo4j-compat` if you want the raw ClickGraph
+identity.
+
+## 4. Point Neo4j Browser at it
+
+Open Neo4j Browser (web or desktop). Connect with:
+
+```
+Connection URI:  bolt://localhost:7687
+Authentication:  no auth   (or: Basic, username `neo4j`, any password)
+```
+
+You should land on the standard Browser welcome page. Click the
+schema icon in the sidebar to list node labels — they'll come from
+your YAML.
+
+### Sample queries (assuming the social-network schema)
+
+```cypher
+// Count all users
+MATCH (u:User) RETURN count(u) AS users;
+
+// Top followers
+MATCH (u:User)<-[:FOLLOWS]-(f:User)
+RETURN u.name AS user, count(f) AS followers
+ORDER BY followers DESC
+LIMIT 10;
+
+// Friends-of-friends (variable-length path)
+MATCH (me:User {user_id: 42})-[:FOLLOWS*2..2]->(fof:User)
+WHERE fof.user_id <> 42
+RETURN DISTINCT fof.name AS friend_of_friend
+LIMIT 25;
+```
+
+Each query is translated to Spark SQL locally, posted to your
+warehouse's Statement Execution API, and the result is mapped back
+into the Bolt response Browser expects. Use Browser's "Query Plan" or
+the equivalent HTTP probe (below) to see the actual SQL.
+
+## 5. Inspect the generated SQL (without executing)
+
+For debugging the translation:
+
+```bash
+curl -s -X POST http://localhost:7475/query \
+  -H "Content-Type: application/json" \
+  -d '{"query":"MATCH (u:User) RETURN u.name LIMIT 5","sql_only":true}' \
+  | jq -r .sql
+```
+
+This returns the Spark SQL the executor would have sent, without
+touching the warehouse. Equivalent to:
+
+```bash
+cg --schema schema.yaml --dialect databricks sql "MATCH (u:User) RETURN u.name LIMIT 5"
+```
+
+## What works today
+
+- Read queries: `MATCH`, `WHERE`, `RETURN`, `WITH`, `ORDER BY`, `LIMIT`,
+  `SKIP`, `DISTINCT`, `OPTIONAL MATCH`, `UNWIND`, `UNION ALL`.
+- Variable-length paths (`*1..n`), shortest path, multi-hop traversals.
+- Aggregations: `count`, `sum`, `avg`, `min`, `max`, `collect`.
+- String / numeric / date / list / map functions — all routed through
+  the dialect-aware `FunctionMapper` (e.g. `collect()` → `collect_list`,
+  `toInt64()` → `bigint`).
+- Pattern comprehension, list comprehension, `CASE` expressions.
+- Neo4j Browser, NeoDash, graph-notebook, neo4rs, any Bolt v5 client.
+
+## What's not in this iteration
+
+- **Writes** (`CREATE`, `SET`, `DELETE`, `MERGE`). DeltaGraph is read-only
+  against Databricks; embedded chdb is the only write target today.
+  Write support against Delta tables is not on the current roadmap and
+  has no committed timeline — track [the DeltaGraph plan](../design/DELTAGRAPH_PLAN.md)
+  for any change in scope.
+- **OAuth M2M.** PAT is the only supported auth.
+- **External-link result chunks.** Large result sets (>25 MB) currently
+  fail with an error from the Statement Execution API. The executor
+  uses `INLINE`/`JSON_ARRAY` disposition; switching to `EXTERNAL_LINKS`
+  for large results is a Phase 5 deliverable.
+- **`CALL` subqueries** (e.g. LDBC bi-16) — same gap as ClickGraph;
+  inherited from the shared planner.
+- **Schema discovery from Unity Catalog** (`SHOW TABLES IN catalog.schema`,
+  `DESCRIBE TABLE EXTENDED`). Phase 3 ships this; until then your YAML
+  is hand-authored or generated via `cg schema discover` against a
+  ClickHouse staging copy.
+
+## Pointing `cg` at the same warehouse
+
+The `cg` CLI also supports `--dialect databricks` for ad-hoc queries
+without the full server:
+
+```bash
+export DATABRICKS_HOST=...
+export DATABRICKS_WAREHOUSE_ID=...
+export DATABRICKS_TOKEN=...
+
+# Translate (no execution):
+cg --schema schema.yaml --dialect databricks sql "MATCH (u:User) RETURN u.name"
+
+# Execute against the warehouse (requires `cg` built with --features databricks):
+cargo install --path clickgraph-tool --features databricks --force
+cg --schema schema.yaml --dialect databricks query "MATCH (u:User) RETURN u.name LIMIT 5"
+```
+
+`cg` shares the same `DATABRICKS_*` env names, plus `CG_DATABRICKS_*`
+overrides if you want to scope settings to `cg` alone without leaking
+into a running `deltagraph` server.
+
+## Troubleshooting
+
+**`DATABRICKS_HOST not set`** — env var is missing or scrubbed. Check
+that you didn't `env -i` the shell, and that `direnv` (if used) loaded
+the right `.envrc`.
+
+**`401 Unauthorized` from the executor** — PAT is invalid, expired, or
+lacks `SELECT` on the catalog/schema. Verify with `curl` against
+`https://$DATABRICKS_HOST/api/2.0/sql/warehouses/$DATABRICKS_WAREHOUSE_ID`.
+
+**Browser shows "Could not connect"** — the server's Bolt port is bound
+to `0.0.0.0` by default. Check `lsof -i :7687` on the server host. If
+running in Docker, expose the port (`-p 7687:7687`).
+
+**Query returns "Cannot resolve column …"** — the Cypher property names
+in your query don't match the `property_mappings` in your schema YAML.
+This is the most common source of friction; double-check the YAML.
+
+**Slow first query** — Databricks SQL Warehouses scale to zero by
+default. The first query after idle takes 30–90s for the warehouse to
+warm up. Subsequent queries are sub-second.
+
+## Where to file feedback
+
+Issues go in the main ClickGraph repo with the `dialect:databricks`
+label. The DeltaGraph design doc is at
+[`docs/design/DELTAGRAPH_PLAN.md`](../design/DELTAGRAPH_PLAN.md).
diff --git a/src/server/mod.rs b/src/server/mod.rs
@@ -647,5 +647,101 @@ fn build_databricks_config() -> Result<crate::executor::databricks_sql::Databric
         crate::executor::databricks_sql::DatabricksConfig::new(hostname, warehouse_id, token);
     cfg.catalog = std::env::var("DATABRICKS_CATALOG").ok();
     cfg.schema = std::env::var("DATABRICKS_SCHEMA").ok();
+    // Test-only override for the executor's request base URL. Honored
+    // by the executor (`DatabricksConfig.base_url`) and used by the
+    // deltagraph subprocess test in tests/rust/bin/ to redirect HTTP
+    // at a wiremock URL. Production callers leave this unset — the
+    // executor falls back to `https://{hostname}`.
+    //
+    // Security guardrail: a misconfigured override would otherwise send
+    // the PAT in plaintext to whatever endpoint is named. We accept
+    // either an `https://` URL (any host, real workspaces) or `http://`
+    // restricted to loopback (wiremock-style tests). Anything else is
+    // rejected up front, and any successful override produces a single
+    // `log::warn!` so a stray production setting can't be silent.
+    if let Ok(raw) = std::env::var("DATABRICKS_BASE_URL") {
+        validate_databricks_base_url(&raw)?;
+        log::warn!(
+            "⚠ DATABRICKS_BASE_URL override in use ({raw}); the PAT will be sent there. \
+             This env var is intended for tests against a local mock — unset it for production."
+        );
+        cfg.base_url = Some(raw);
+    }
     Ok(cfg)
 }
+
+/// Reject overrides that would silently leak the PAT to a non-localhost
+/// plaintext endpoint. Loopback HTTP is allowed (wiremock); arbitrary
+/// HTTP is not. Anything that fails to parse or uses a non-http(s)
+/// scheme is rejected with a clear message rather than being passed
+/// through.
+#[cfg(feature = "databricks")]
+fn validate_databricks_base_url(raw: &str) -> Result<(), String> {
+    // Minimal parsing — we don't want to pull in the `url` crate just
+    // for this check. Match `https://...` unconditionally; `http://`
+    // only when the host segment is `localhost` or `127.0.0.1` (with
+    // optional port). Anything else is rejected.
+    if let Some(rest) = raw.strip_prefix("https://") {
+        if rest.is_empty() {
+            return Err("DATABRICKS_BASE_URL is empty after https://".to_string());
+        }
+        return Ok(());
+    }
+    if let Some(rest) = raw.strip_prefix("http://") {
+        let host = rest.split(['/', ':']).next().unwrap_or("");
+        if host == "localhost" || host == "127.0.0.1" {
+            return Ok(());
+        }
+        return Err(format!(
+            "DATABRICKS_BASE_URL rejected: `http://` is only allowed for loopback (localhost, \
+             127.0.0.1) to avoid leaking the PAT in plaintext. For real workspaces use https:// \
+             or unset the variable. Got host={host:?}"
+        ));
+    }
+    Err(format!(
+        "DATABRICKS_BASE_URL rejected: must start with `https://` (or `http://localhost…` for \
+         tests). Got: {raw:?}"
+    ))
+}
+
+#[cfg(all(test, feature = "databricks"))]
+mod databricks_base_url_validation_tests {
+    use super::validate_databricks_base_url;
+
+    #[test]
+    fn accepts_https_for_real_workspaces() {
+        assert!(validate_databricks_base_url("https://dbc-abc-def.cloud.databricks.com").is_ok());
+        assert!(
+            validate_databricks_base_url("https://dbc-abc-def.cloud.databricks.com/api/").is_ok()
+        );
+    }
+
+    #[test]
+    fn accepts_http_for_loopback_tests() {
+        // wiremock + mock server hosts — anything bound to loopback.
+        assert!(validate_databricks_base_url("http://127.0.0.1:55555").is_ok());
+        assert!(validate_databricks_base_url("http://localhost:55555/api/2.0/sql").is_ok());
+    }
+
+    #[test]
+    fn rejects_plaintext_http_to_external_hosts() {
+        // The whole point of the validator: a typo'd
+        // `DATABRICKS_BASE_URL=http://workspace.example.com` would
+        // POST the PAT in plaintext. Block it up front.
+        let err = validate_databricks_base_url("http://workspace.example.com").unwrap_err();
+        assert!(err.contains("loopback"), "got: {err}");
+        let err = validate_databricks_base_url("http://192.168.1.10:8080").unwrap_err();
+        assert!(err.contains("loopback"), "got: {err}");
+    }
+
+    #[test]
+    fn rejects_unknown_schemes() {
+        // ftp/gopher/no-scheme — anything that wouldn't reach the
+        // executor's reqwest client cleanly should fail loudly here
+        // rather than producing a confusing runtime error later.
+        assert!(validate_databricks_base_url("workspace.example.com").is_err());
+        assert!(validate_databricks_base_url("ftp://workspace.example.com").is_err());
+        assert!(validate_databricks_base_url("").is_err());
+        assert!(validate_databricks_base_url("https://").is_err());
+    }
+}
diff --git a/tests/rust/bin/deltagraph_bolt_e2e.rs b/tests/rust/bin/deltagraph_bolt_e2e.rs
