A self-hosted Databricks App that generates realistic, privacy-safe synthetic datasets — built from scratch, modeled on an uploaded CSV, or cloned from existing Unity Catalog tables — and lands the results wherever you need them: a CSV download, or a governed Unity Catalog table in Delta or Iceberg format.
Stack: Python Flask backend · React (Vite + Material UI) frontend · Databricks SQL Warehouse · Unity Catalog · Lakebase (optional, for audit & saved configs)
- Why we built this
- What you can do with it
- The three modes
- Step-by-step: generating your first dataset
- Column configuration reference
- Saved configurations
- Upload to Databricks UC (table export)
- Architecture
- Repository layout
- Deploy to Databricks
- Local development
- Environment variable reference
- Extending the generator
- Operational notes & limits
- Troubleshooting
- Tests and linting
- Contributing
- License
Every data team eventually hits the same wall: you need data you're not allowed to use.
- A new pipeline needs testing, but the only realistic data is production data — full of personal information you can't copy into a dev workspace.
- A vendor or partner needs a sample dataset to build against, but sharing real records is a compliance non-starter.
- A demo environment needs tables that look real — believable names, emails, amounts, dates — but contain nothing real.
- A load test needs 5 million rows shaped like your production schema, and nobody wants to hand-write a generator script for the 40th time.
The usual answers are bad. Masking scripts are one-off and brittle. Copying production data "just this once" is how incidents happen. Mock data libraries produce data that's shaped wrong — wrong cardinality, wrong distributions, wrong column names — so tests pass in dev and fail in prod.
SDG turns this into a point-and-click workflow inside your Databricks workspace. Pick a real table (or upload a CSV, or define a schema from scratch), choose per-column what to keep, mask, or synthesize, set a row count, and click Generate. The result lands as a governed Delta or Iceberg table in Unity Catalog — queryable immediately, owned by your governance model, and containing zero real records in the synthesized columns.
Because it runs as a Databricks App, there is no infrastructure to operate: no servers, no separate auth system, no credentials to manage. Your workspace's login is the front door, and the app's own service principal — with exactly the Unity Catalog grants you give it — is the only identity that touches data.
| Use case | How SDG handles it |
|---|---|
| Privacy-safe dev/test datasets | Clone a production table's shape; synthesize the PII columns (names, emails, phones), retain the rest |
| Demo environments | Generate believable customer/order/employee tables from scratch with realistic value distributions |
| Load & scale testing | Generate millions of rows shaped like your schema, written by the SQL warehouse (not the app container), as Delta or Iceberg |
| Sharing data externally | Produce a lookalike dataset of a sensitive table where every sensitive value is synthetic |
| Reproducible test fixtures | Save a column configuration once; re-apply it to regenerate the same shape of data anytime |
| Data contract prototyping | Define a schema column-by-column before the real source exists, and hand consumers a populated table |
┌────────────────────────────────────────────┐
Browser ── HTTPS ──────► │ Databricks Apps (auth proxy + container) │
(workspace login) │ │
│ React SPA ◄──► Flask API (gunicorn) │
│ │ │
└──────────────────────┼─────────────────────┘
│ service principal (OAuth)
┌──────────────────┬───────────┴────────┬──────────────────┐
▼ ▼ ▼ ▼
SQL Warehouse Unity Catalog UC Volume Lakebase
(reads + table (catalog browsing, (Parquet staging (audit logs,
writes via CTAS) grants, governance) for exports) saved configs)
Request flow for a typical warehouse-mode generation:
- User browses catalogs/schemas/tables (SQL warehouse, via the app SP) and previews rows.
- User configures columns (retain / mask / synthesize + type-specific rules) and a row count.
- The backend reads the source via Arrow, applies the column configuration (parallelized across processes), and keeps the full DataFrame in a server-side job store (30-minute TTL).
- The UI shows a 100-row sample; Download CSV streams the full dataset in 50k-row chunks; Upload to Databricks UC stages Parquet and runs a warehouse CTAS.
- If Lakebase is attached, the generation is recorded in the audit log.
Security model in one paragraph: no credentials live in code or config. The app runs as its own Databricks service principal, whose OAuth credentials are injected by the platform at runtime; Lakebase passwords are short-lived tokens minted on demand. Users reach the app only through the workspace's authenticated proxy. What the app can touch in Unity Catalog is bounded entirely by the grants you give its service principal. Identifiers are validated against a strict pattern and Lakebase SQL is parameterized. The server runs gunicorn with Flask debug mode off by default.
.
├── app.py # Entry point (local dev: python app.py)
├── app.yaml # Databricks Apps manifest: gunicorn command + environment
├── requirements.txt # Backend dependencies
├── backend/
│ ├── config.py # ALL configuration — every env var, one place
│ ├── errors.py # Typed exceptions + consistent JSON error handlers
│ ├── routes/ # Flask blueprints (thin: validate → service → JSON)
│ │ ├── databricks_routes.py # Catalog browsing, previews, CSV upload, saved configs
│ │ ├── final_data_route.py # Generation (sync + async), downloads, UC table export
│ │ └── generate_routes.py # Manual-schema generation
│ └── services/
│ ├── data_generator.py # Generator registry + parallel generation
│ ├── apply_config_service.py# Retain / mask / synthesize per column
│ ├── databricks_service.py # Warehouse connections, Arrow reads, UC export
│ ├── lakebase_audit_service.py # Postgres pool, audit log, saved configs
│ └── job_store.py # In-memory job store (TTL) for generated datasets
├── frontend/ # React + Vite + Material UI source
│ ├── package.json # Frontend dependencies (managed separately from Python's)
│ ├── vite.config.js # Dev server config (proxies /api → Flask on :8000)
│ ├── src/
│ │ ├── pages/ # One page per mode: BasicSourcePage, UploadCsvSourcePage,
│ │ │ # WarehouseSourcePage + the GuidedWizard landing flow
│ │ ├── components/ # Shared UI: ExportToUcTableDialog, column config panels,
│ │ │ # GlobalSnackbar (toasts), download button, tooltips
│ │ ├── hooks/ # Per-mode state machines (useBasicSourceState, ...)
│ │ ├── api/ # Typed API client for every backend endpoint
│ │ └── types/ # Shared TypeScript types (column configs, responses)
│ └── dist/ # Built SPA served by Flask (committed for easy deploys)
├── scripts/build.sh # One-command frontend build
├── tests/ # pytest suite (config, errors, generators, job store, validation)
└── .github/workflows/ # CI: ruff + pytest + frontend build
SDG offers three entry points, depending on where your schema comes from. All three end at the same place: a generated dataset you can preview, download as CSV, or export as a Unity Catalog table.
When to use it: you don't have a source table at all. You're prototyping, building a demo, or defining a data contract before the real data exists.
What it does: you name an entity (e.g. Customers), then add columns one at a time. Each column gets a name, a data type, and type-specific rules (ranges, formats, distributions — see the configuration reference). Set a row count, click Generate Data, and SDG fabricates the whole dataset.
Requires Databricks connectivity? No — generation itself is fully local to the app. You only need the workspace connection for the final Upload to Databricks UC step.
When to use it: you have a sample file (an export, a spreadsheet, a fixture from another system) and want a bigger and/or de-identified version of it.
What it does: you upload a CSV; SDG parses it, lists the columns, and infers types automatically — both from the values (numbers, dates, booleans) and from column names (a column called email_address is pre-configured as an email generator; first_name, phone, company get similar semantic treatment). You then choose per column whether to retain the original values, mask them, or synthesize replacements. You can also grow the dataset: ask for more rows than the file has, and SDG resamples and re-synthesizes to the requested size.
Requires Databricks connectivity? No for generation; yes for UC export.
When to use it: the real table exists in Unity Catalog and you want a synthetic twin of it.
What it does: browse catalog → schema → table (the dropdowns show exactly what the app's service principal has been granted), pick columns, and preview live rows. Then configure each column — retain / mask / synthesize — set a row count, and generate. Reads go through your SQL warehouse using Apache Arrow for speed; large generations run as background jobs with progress polling. This mode also unlocks saved configurations, so a table's masking recipe can be stored and reapplied.
Requires Databricks connectivity? Yes — a SQL warehouse for reads, and UC grants for browsing.
- Open the app and choose Basic / Manual schema from the source options.
- Enter an Entity Name — e.g.
Customers. (This becomes the audit-log label and the default export table name.) - Click + Add New Column.
- In the column panel: give it a name (
customer_id), pick Data Type →ID, chooseUUIDorSequential, and Save. - Repeat for the rest of your schema, for example:
full_name→ String → Full Nameemail→ String → Email, with domainsexample.com, example.orgage→ Number, min18, max90signup_date→ Datetime, range2020-01-01→ todayis_active→ Boolean, by percentage, 80% truetier→ String → String with Percentage, distributionGold:20, Silver:30, Bronze:50
- Set the Row Count.
- Click Generate Data. A table appears with a 100-row display sample (the full dataset is held server-side).
- Finish with Download CSV (full dataset) or Upload to Databricks UC (see table export).
- Choose Upload CSV and select your file.
- SDG lists the detected columns. Select the columns you want in the output (search box provided; select-all supported).
- Click Preview — you'll see sample rows, and SDG auto-fills suggested types per column (e.g. it spots emails, names, phones, dates).
- For each selected column choose a mode:
- Retain — keep original values as-is
- Mask — replace every value with a fixed mask string
- Synthetic — replace with generated values of a type you pick (see configuration reference)
- Set how many rows you want — it can be more than the file has (see how row count interacts with retain/mask/synthesize).
- Click Generate. Review the sample.
- Download CSV or Upload to Databricks UC.
- Choose Unity Catalog / Warehouse as the source.
- Pick a Catalog from the dropdown, then a Schema, then a Table. (If a catalog you expect is missing, the app's service principal lacks
USE CATALOGon it — see deployment grants.) - Select columns to include and click Preview to fetch sample rows.
- (Optional) If a teammate already saved a configuration for this table, pick it from the Saved Configurations dropdown — all column modes and rules are applied instantly.
- Configure each column: Retain, Mask, or Synthetic with type-specific rules.
- Set the final row count. It defaults to the source table's exact row count, and you can raise or lower it — see how row count interacts with retain/mask/synthesize below for exactly what each choice means.
- Click Generate Final Data. Large requests run asynchronously — the UI polls the job and shows progress.
- When done you'll see the sample plus row counts. Download CSV streams the full dataset; Upload to Databricks UC writes a Delta/Iceberg table.
- (Optional) Click Save Configuration and give the recipe a name so it can be reapplied later.
(Applies to Warehouse mode and CSV Upload mode — anywhere there is source data.)
The row-count field defaults to the source's exact row count, and what happens to each column depends on whether you keep that default, lower it, or raise it:
| Requested rows | Retained columns | Masked columns | Synthetic columns |
|---|---|---|---|
| = source count (default) | Exact copy of the source — every value, in order, unchanged | Mask string in every row | Freshly generated values for every row |
| < source count | The first N source rows, unchanged | Mask string in every row | Freshly generated values for every row |
| > source count | All original rows plus extra rows resampled from the source (random rows repeated to fill the gap) | Mask string in every row | Freshly generated values for every row — no repetition from resampling |
Two points worth being deliberate about:
- Retained columns stay faithful at the default count. If your goal is "same table, but with the sensitive columns synthesized," keep the default row count — retained columns will match the source exactly, row for row.
- Growing the dataset duplicates retained values, not invents them. When you request more rows than the source has, SDG fills the extra rows by sampling existing source rows with replacement. Synthetic columns are then regenerated across the entire output, so they stay unique and varied — but retained columns in the extra rows are repeats of real values. If a retained column must stay unique (e.g. a primary key), either switch it to Synthetic → ID, or don't grow past the source count.
Every synthetic column has a type plus type-specific options. The same types are available in all three modes (Basic mode exposes them when adding a column; CSV/Warehouse modes expose them when you set a column to Synthetic).
General rule: options are optional. Leave them empty and SDG generates realistic values via the Faker library. Fill them in to constrain the output.
| Type | Options | Example configuration | Notes / Don'ts |
|---|---|---|---|
| Full Name | Custom value pool (comma-separated) | John Smith, Sarah Connor, Priya Patel |
Pool = values are sampled from your list only. Empty = realistic random names. |
| First Name / Last Name | Custom value pool | John, Sarah, David |
Use when the target schema splits names into separate columns. |
| Domain suffixes (comma-separated) | example.com, test.org |
Generates name123@domain. Don't include @ unless you want it verbatim — both example.com and @example.com work. |
|
| Phone Number | Country code + local pattern (# = digit) |
code +44, pattern ####-###### |
Pattern is honored exactly — count your #s. |
| Company Name / Job Title | Custom value pool | Acme Corp, Globex |
Empty = realistic generated values. |
| Address | Mode (faker/custom), locale, include postcode / secondary / country, single- or multi-line |
locale en_US, single-line, postcode on |
In custom mode you must supply at least one address in the pool. |
| City / Country | Custom value pool | Dublin, Chennai, Austin |
Empty = realistic values. |
| String (plain) | Custom value pool | red, green, blue |
Empty = random dictionary words — fine for filler, not meaningful categories. Use a pool or String with Percentage for categorical columns. |
| String with Percentage | value:percent pairs (comma-separated) |
Gold:20, Silver:30, Bronze:50 |
Percentages must total exactly 100 or generation fails with a clear error. Counts are honored exactly, not probabilistically. |
| Number | Min, Max (integers) | min 1000, max 9999 |
Inverted ranges are auto-swapped. For decimals/currency, generate integers (e.g. cents) and divide downstream, or add a custom generator. |
| Datetime | Min, Max (ISO format), timezone (UTC / local / specific) | 2023-01-01T00:00:00 → 2024-12-31T23:59:59 |
Invalid date strings silently fall back to "last 10 years" — double-check your format if results look off. Output format: YYYY-MM-DD HH:MM:SS. |
| Boolean | Distribution: random / 50-50 / all true / all false / by percentage | by percentage, 25 % true |
"By percentage" yields an exact split (25% of rows true), not a coin flip per row. |
| ID | UUID or Sequential |
sequential | Sequential IDs are globally continuous (1, 2, 3, …) even though generation is parallelized. UUIDs are guaranteed unique; sequential restarts at 1 each generation. |
| Masked | Mask string | ***-**-XXXX |
Every row gets the identical mask string. Use Synthetic instead if you need fake-but-varied values. |
- ✅ Do use String with Percentage for any categorical column where the distribution matters (status, tier, region) — exact proportions make downstream aggregates believable.
- ✅ Do use custom value pools to keep generated data consistent with reference data in other tables (e.g. the same five product names everywhere).
- ✅ Do set realistic min/max ranges for numbers and dates — defaults are intentionally wide.
- ❌ Don't rely on Masked for analytical columns — every value is identical, which breaks group-bys and joins. Synthesize instead.
- ❌ Don't expect referential integrity across columns or tables (a
cityandcountryin the same row are independently random). Generate consistent pairs by using pools that encode the pair (Dublin|Ireland) or extend the generator. - ❌ Don't retain columns you intend to share externally without checking your data-handling policy — retain means real data flows through untouched.
Available in Warehouse mode; requires the optional Lakebase resource.
Configuring 40 columns of a wide table takes a few minutes; you shouldn't have to repeat it. A saved configuration stores the complete recipe — selected columns, each column's retain/mask/synthesize mode, and all type rules — under a name, keyed to the source table.
To save: configure your columns → click Save Configuration → enter a name (e.g. pii-masked-v1) → Save. The combination of catalog + schema + table + config name is unique; saving the same name again updates it.
To reuse: select the same source table → the Saved Configurations dropdown lists every recipe stored for that table → pick one → all settings apply instantly → adjust if needed → Generate.
Where it lives: a table_configurations table in the app's Lakebase schema, written by the app's service principal. Configurations are shared by all users of the app — they're team assets, not personal ones.
Why this matters in practice: the masking recipe for a table effectively becomes reviewable, reusable policy. New team members don't have to know which columns are sensitive — they pick pii-masked-v1 and get the approved treatment.
On first startup with Lakebase configured, the app creates its own schema — <PGAPPNAME>_schema, so sdg_app_schema by default — and three tables inside it. You never create these by hand; the bootstrap is idempotent and also applies small migrations on upgrade.
One row per generation, written automatically from every mode:
| Column | Type | Contents |
|---|---|---|
id |
SERIAL | Auto-incrementing primary key |
user_id |
TEXT | Signed-in user's email (from the Databricks Apps auth proxy) |
workspace_id |
TEXT | Workspace the generation ran in (auto-detected) |
table_name |
TEXT | Source: catalog.schema.table, the uploaded file name, or the Basic-mode entity name |
row_count |
INT | Rows generated |
columns |
TEXT | JSON list of the output columns |
timestamp |
TIMESTAMP | When the generation ran |
processing_time |
DOUBLE | Seconds taken (3-decimal precision) |
Query it with plain SQL from the Lakebase SQL editor or any Postgres client, e.g. who generated the most rows this month:
SELECT user_id, SUM(row_count) AS total_rows
FROM sdg_app_schema.audit_logs
WHERE timestamp >= date_trunc('month', now())
GROUP BY user_id ORDER BY total_rows DESC;Backs the saved configurations feature. Columns: config_name, catalog_name, schema_name, table_name, column_configs (the full recipe as JSONB), created_at, updated_at. A unique constraint on (catalog, schema, table, config name) makes saving an existing name an update rather than a duplicate. Managed entirely through the UI — no manual SQL needed.
This is the one table you do populate by hand, and it's a quiet productivity feature. It holds workspace-curated value pools that surface in the UI as one-click fill buttons when configuring synthetic columns in Warehouse mode — e.g. a "Use Sample names" or "Use Common Domains" button appears next to the custom-values field and fills it with your curated list, so users don't retype your organization's standard test values every time.
| Column | Type | Contents |
|---|---|---|
reference_type |
TEXT | Which generator the values belong to — the UI looks for name, company_name, email_suffix, and address |
reference_value |
TEXT | The values as a single comma-separated string |
Seed it once from the Lakebase SQL editor:
INSERT INTO sdg_app_schema.reference_data_table (reference_type, reference_value) VALUES
('name', 'Avery Johnson, Jordan Lee, Sam Patel, Riya Sharma, Chris Murphy'),
('company_name', 'Acme Corp, Globex, Initech, Umbrella Logistics, Stark Industries'),
('email_suffix', 'example.com, example.org, testmail.dev'),
('address', '12 Main St Springfield IL 62701, 401 Oak Ave Austin TX 73301');Notes for curators:
- One row per
reference_type; values live in a single comma-separated string (so values themselves must not contain commas). - Changes take effect on the next page load — the UI fetches
/api/reference_datawhen the Warehouse page opens; no app restart needed. - An empty table is perfectly fine: the fill buttons simply don't appear, and users type values manually or rely on Faker defaults.
Alongside these, remember the operational behavior: if Lakebase isn't configured at all, all three features (audit, saved configs, reference buttons) disable themselves gracefully — the rest of the app is unaffected.
Every mode ends with the same export dialog — Upload to Databricks UC — which persists the full generated dataset (not just the preview sample) as a managed Unity Catalog table.
The dialog, field by field:
| Field | Choices | Guidance |
|---|---|---|
| Table format | Delta / Iceberg | Delta is the Databricks-native default. Pick Iceberg if downstream consumers read via Iceberg clients. |
| Write mode | Overwrite / Append | Overwrite creates the table or replaces it entirely (CREATE OR REPLACE). Append creates the table if missing, then inserts — use it to accumulate batches. |
| Catalog | live dropdown | Lists the catalogs the app's service principal can see; you can also type a name. |
| Schema / Table | free text | Letters, digits and underscores only. The table name is pre-filled from your source (entity name, file name, or <source>_synthetic). |
What happens under the hood when you click Save:
- The app writes the dataset as a Parquet file to a staging path inside the configured UC Volume (
SDG_VOLUME_PATH/_staging/<uuid>.parquet) using the Databricks Files API. - The SQL warehouse executes
CREATE OR REPLACE TABLE <target> USING DELTA AS SELECT * FROM read_files('<staging path>')(or the Iceberg/append equivalents). The warehouse does the distributed write — the app container's memory never limits exported table size. - The staging file is deleted, success or failure.
Why a volume is involved at all: the warehouse cannot read the app container's memory, and apps cannot write tables directly. The volume is the governed hand-off point between the two. You never manage these files — staging is automatic and self-cleaning.
Required grants for the destination (one-time, per target schema):
GRANT USE CATALOG ON CATALOG <catalog> TO `<app-sp-client-id>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA <catalog>.<schema> TO `<app-sp-client-id>`;Timing note: the export waits for the dataset to exist server-side. Generated datasets are kept for 30 minutes (see limits); if you wait longer, the dialog will tell you to regenerate first.
- A Databricks workspace with Databricks Apps and Unity Catalog enabled
- A SQL warehouse (serverless works great) and its HTTP path (
/sql/1.0/warehouses/<id>— find it under SQL Warehouses → your warehouse → Connection details) - Permission to create an app, a UC volume, and grant privileges on a catalog/schema
Run in a SQL editor attached to a SQL warehouse (these are Unity Catalog statements — they will not run against Lakebase):
CREATE SCHEMA IF NOT EXISTS <catalog>.sdg;
CREATE VOLUME IF NOT EXISTS <catalog>.sdg.sdg_exports
COMMENT 'SDG app: staging area for table exports';(Optional, enables audit + saved configs) Create a Lakebase project:
- In the workspace sidebar: Compute → Lakebase Postgres → Create project.
- Open the project → Branches → production (default branch) → note the read-write endpoint's resource path, which looks like
projects/<project>/branches/<branch>/endpoints/<endpoint>.
Edit the environment section to match your workspace:
- name: LAKEBASE_ENDPOINT # optional — remove this block to disable audit/saved configs
value: "projects/<project>/branches/<branch>/endpoints/<endpoint>"
- name: SDG_VOLUME_PATH
value: "/Volumes/<catalog>/sdg/sdg_exports"
- name: SDG_CLOUD_FETCH # "true" on open networks = faster large reads;
value: "false" # "false" if your network blocks egress to cloud storage
PIP_INDEX_URL— most users should leave this commented out. Databricks Apps installsrequirements.txtfrom the public PyPI by default, so no configuration is needed. Only if your organization blocks pypi.org and routes installs through an internal mirror (Artifactory, Nexus, …) do you need it: store the mirror URL in a secret, attach it to the app as a resource with keypip-index-url, and uncomment thePIP_INDEX_URLblock inapp.yaml. If you uncomment it without attaching the resource, the deployment will fail with a missing-resource error.
Option A — Git folder (recommended):
- Push this repository to your Git provider.
- In the workspace: Workspace → Create → Git folder, paste the repo URL, pick your branch.
- Future updates = open the Git folder → kebab menu → Git… → Pull.
Option B — Databricks CLI sync:
bash scripts/build.sh # builds frontend/dist
databricks sync . /Workspace/Users/<you>/sdg-app --full- Compute → Apps → Create app → Custom. Give it a name (e.g.
sdg). - Open the app → Edit → Configuration → App resources, and add:
| Resource | Permission | Resource key | Provides |
|---|---|---|---|
| SQL warehouse | Can use | — | query + export execution |
| Secret (warehouse HTTP path) | Can read | http_path |
the HTTP_PATH env var |
| Database (your Lakebase project) | Can connect and create | — | audit + saved configs (optional) |
The secret holds the warehouse HTTP path (
/sql/1.0/warehouses/<id>). Create it once withdatabricks secrets create-scope <scope>anddatabricks secrets put-secret <scope> http_path, or skip the secret entirely and setHTTP_PATHas a plainvalue:inapp.yaml— it's a path, not a credential.
- Note the app's service principal client ID on the app's Authorization tab — you need it for the grants below.
Run in a SQL editor (replace placeholders):
-- Browsing + reading source tables (repeat per catalog/schema you want visible in the app)
GRANT USE CATALOG ON CATALOG <catalog> TO `<app-sp-client-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<app-sp-client-id>`;
GRANT SELECT ON SCHEMA <catalog>.<schema> TO `<app-sp-client-id>`;
-- Writing exported tables + using the staging volume
GRANT CREATE TABLE ON SCHEMA <catalog>.sdg TO `<app-sp-client-id>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <catalog>.sdg.sdg_exports TO `<app-sp-client-id>`;Grant only what you want the app to see — its catalog dropdowns will show exactly (and only) these.
- On the app page click Deploy and select the workspace folder from step 3.
- Wait for status Running, then open the app URL.
- Check the Logs tab: on startup the app prints a configuration report and explicitly warns about anything missing (no warehouse path, no Lakebase, etc.) instead of failing silently.
- Smoke test: Warehouse mode → your catalog appears in the dropdown → preview a table → generate 1,000 rows → Upload to Databricks UC → confirm the table in Catalog Explorer.
There are two ways to run the app locally, and understanding them avoids 90% of confusion:
-
Backend only —
python app.py→ everything at http://localhost:8000. Flask serves the API and the pre-built UI fromfrontend/dist. This is exactly what production does. You do not need npm or Node for this — only if you want to change the UI. -
Backend + frontend dev server —
python app.pyandnpm run dev→ UI at http://localhost:5173 with hot reload. The Vite dev server serves the UI source live (edits appear instantly) and proxies every/api/...call to the Flask backend on port 8000 (configured infrontend/vite.config.js). So yes — the dev UI is fully connected to the backend, as long as the backend is also running. If you start onlynpm run dev, the UI loads but every data operation fails with network errors, because there is nothing on port 8000 to answer.
| You want to… | Run |
|---|---|
| Use the app / test backend changes | python app.py → open :8000 |
| Edit UI code with instant feedback | python app.py and npm run dev → open :5173 |
| Ship UI changes | bash scripts/build.sh (rebuilds frontend/dist), then deploy |
| Tool | Version | Check with |
|---|---|---|
| Python | 3.11+ | python3 --version (macOS/Linux) / py --version (Windows) |
| Node.js + npm | Node 18+ (20 recommended) | node --version |
| Git | any recent | git --version |
Installers: python.org/downloads · nodejs.org · git-scm.com. On Windows, tick "Add python.exe to PATH" in the Python installer.
# 1. Clone and enter the repo
git clone <your-repo-url>
cd <repo-folder>
# 2. Create the environment file and fill in your values
cp .env.example .env
# Minimum for warehouse features: DATABRICKS_HOST, HTTP_PATH,
# DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET
# (Basic and CSV modes work with none of these set.)
# 3. Create and activate a Python virtual environment
python3 -m venv .venv
source .venv/bin/activate
# 4. Install backend dependencies
pip install -r requirements.txt
# 5. Start the backend (serves UI + API)
python app.py
# → open http://localhost:8000
# 6. (Only for UI development) in a SECOND terminal:
cd frontend
npm install
npm run dev
# → open http://localhost:5173 (hot reload; /api proxied to :8000)# 1. Clone and enter the repo
git clone <your-repo-url>
cd <repo-folder>
# 2. Create the environment file and fill in your values
Copy-Item .env.example .env
notepad .env
# 3. Create and activate a Python virtual environment
py -m venv .venv
.\.venv\Scripts\Activate.ps1
# If activation is blocked, run once:
# Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
# 4. Install backend dependencies
pip install -r requirements.txt
# 5. Start the backend (serves UI + API)
python app.py
# → open http://localhost:8000
# 6. (Only for UI development) in a SECOND terminal:
cd frontend
npm install
npm run dev
# → open http://localhost:5173 (hot reload; /api proxied to :8000)- Basic and CSV modes need no Databricks connection at all — ideal for UI or generator work.
- Warehouse browsing/exports locally use the service principal credentials from
.env. - Auto-reload + debugger for backend work:
FLASK_DEBUG=1 python app.py(macOS/Linux) or$env:FLASK_DEBUG="1"; python app.py(PowerShell). Never enable this in a shared deployment. - Saved configs / audit logs locally require
LAKEBASE_ENDPOINT(+PGUSERas your own user) in.env; without them those features quietly disable themselves. - Before deploying UI changes, always rebuild the bundle:
bash scripts/build.sh(orcd frontend && npx vite build). The deployed app servesfrontend/dist, not your source.
Every setting is an environment variable, declared in one place — backend/config.py — and documented in .env.example.
| Variable | Required | Default | Purpose |
|---|---|---|---|
DATABRICKS_HOST |
yes | (injected by Apps) | Workspace URL |
DATABRICKS_CLIENT_ID / _SECRET |
yes | (injected by Apps) | App service principal OAuth credentials |
HTTP_PATH |
yes | — | SQL warehouse path /sql/1.0/warehouses/<id> |
LAKEBASE_ENDPOINT |
no | — | Lakebase endpoint resource path; enables audit + saved configs |
PGDATABASE |
no | databricks_postgres |
Lakebase database name |
PGAPPNAME |
no | sdg_app |
Drives the Lakebase schema name (<PGAPPNAME>_schema) |
PGHOST / PGUSER |
no | — | Local-dev overrides for direct Postgres access |
SDG_VOLUME_PATH |
no | /Volumes/main/sdg/exports |
UC Volume used to stage exports |
SDG_CLOUD_FETCH |
no | true |
false = route large results through the Databricks endpoint (restricted networks) |
SDG_MAX_ROWS |
no | 1000000 |
Upper bound on rows per generation request |
SDG_CORS_ORIGINS |
no | (same-origin) | Extra origins for split frontend/backend development |
SDG_WORKSPACE_ID |
no | (auto-detected) | Override the workspace ID recorded in audit logs |
FLASK_DEBUG |
no | off | 1 = auto-reload + debugger (local dev only) |
FLASK_HOST |
no | 127.0.0.1 |
Local dev bind address; set 0.0.0.0 to expose on your LAN |
PORT |
no | 8000 |
Local dev port for python app.py |
Column types live in a registry. Adding one is a single function — no UI changes, no route changes:
# backend/services/data_generator.py (or your own module that imports it)
from backend.services.data_generator import generator, fake
@generator("ip_address", label="IP address")
def gen_ip(config, n, start_index):
return [fake.ipv4() for _ in range(n)]The new type is immediately served by GET /api/generator_types and usable in column configurations. The contract: receive the column's config dict, the number of values to produce, and the global start index (for sequential IDs); return exactly n values. To surface it in the UI dropdowns, add the type to frontend/src/types/dataTypes.ts and rebuild the bundle.
- Generated datasets are transient by design. The working copy lives in app memory for 30 minutes (job store TTL). The durable output is the exported UC table or downloaded CSV. If the app restarts mid-session, regenerate — nothing of record is lost.
- Single worker, many threads. The app intentionally runs as one gunicorn worker (jobs live in process memory) with threaded concurrency. To scale out, reimplement the small
JobStoreinterface on Redis or Lakebase and raise the worker count. - Memory is the budget for generation size.
SDG_MAX_ROWSis your dial; exported table size is effectively unbounded since the warehouse performs the write. - Synthetic ≠ anonymized. Synthesized columns contain no real values, but columns you choose to retain are real data. Review your column choices against your data-handling policies before sharing exports.
| Symptom | Likely cause | Fix |
|---|---|---|
| A catalog is missing from the dropdowns | App SP lacks USE CATALOG |
Grant it (see grants) |
Connection reset by peer on large generations |
Network blocks egress to cloud storage (CloudFetch) | Set SDG_CLOUD_FETCH=false in app.yaml, redeploy |
| Export fails with permission error | Missing CREATE TABLE on target schema or volume grants |
Run the grant statements for the destination |
| "Generated dataset is no longer available" | The 30-minute job TTL expired | Regenerate, then export |
| Saved configs / audit silently absent | LAKEBASE_ENDPOINT unset or Database resource not attached |
Attach Lakebase, set the endpoint, redeploy; check startup logs |
| Local UI on :5173 loads but every action fails | Backend not running | Start python app.py in another terminal — the dev server only proxies to it |
| UI changes don't appear after deploy | Stale frontend/dist or browser cache |
bash scripts/build.sh, redeploy, hard-refresh (Ctrl/Cmd+Shift+R) |
| Startup log warns about config | Missing env var | The warning names the variable — set it in app.yaml or .env |
pip install pytest ruff
pytest # unit tests
ruff check backend tests # lintCI (GitHub Actions) runs lint, tests, and a frontend build on every push and pull request.
Contributions are welcome — bug fixes, new generator types, documentation, UI improvements. Please read CONTRIBUTING.md and our Code of Conduct first. Security issues should be reported privately — see SECURITY.md.
This project is licensed under the Apache License 2.0 — see LICENSE for details.