JITSU-227: Reverse ETL console setup, model editor and run management - #1525
Merged
Merged
Conversation
![jitsu-code-review[bot]](https://avatars.githubusercontent.com/in/3391383?s=60&v=4){kind=small}
absorbb
added a commit
that referenced
this pull request
Sep 23, 2026
## Stack / merge order Stacked on **#1525** (`feat/jitsu-227-console-ui`). Merge #1525 first, then retarget this PR to `newjitsu` before merging. This branch now contains both the console UI and object-backed runner for combined local testing; this PR's diff includes persistence/deployment, delivery observability, and the opt-in Google full-audience replacement feature described below. Restacked on the recovered #1525 tip `d55901592`, including its 14 previously unpushed local commits. Preserves editor/preview, consent, scheduling, navigation, ClickHouse spooling and snapshot-progress fixes. Conflict resolution retains the object-backed backend, both safe failure paths, controller overrides and the existing MicroTime fix; the obsolete SQL snapshot writer is not restored. Includes safe startup diagnostics and the dev syncctl tunnel fix. Local ignored configuration and dev-only Helm edits remain separate. No deployment was performed as part of this recovery/restack. ## Summary `JITSU-227` Replaces Reverse ETL per-row PostgreSQL persistence with local SQLite indexing and GCS/S3 immutable snapshots, batches and receipts. PostgreSQL retains only control, audience ownership, checkpoints and task metadata. - **Removes six payload tables:** `reverse_sync_batch`, `reverse_sync_operation`, `reverse_sync_generation`, `reverse_sync_source_key`, `reverse_sync_desired`, `reverse_sync_membership`. - Removes five obsolete control accounting columns, the unused action enum, the legacy backend/fallback and its SQL retention sweeper. - **Object storage is required.** No per-source-row/member/operation PostgreSQL writes. - One small `artifact_head` pointer commits the immutable manifest together with lifecycle/store/checkpoint transitions. No object I/O inside SQL transactions. - New control rows atomically receive a previously uploaded empty manifest pointer. Legacy null-head rows fail closed even in phase `new`; interrupted new startup remains recoverable. - Exact batches are durable before provider calls. Recovery reconstructs SQLite from baseline + receipts, preserving partial success and sequence tombstones. Unknown provider outcomes still require reconciliation, not blind replay. - Bounded, checksummed, scope-bound artifacts; conditional immutable GCS/S3 writes; byte-bounded snapshot/baseline files; cancellation and operation-ID guards. - Destination interfaces and syncctl recovery flow stay intact. Lease renewal covers restoration and compaction. - Existing lifecycle/mirror tests now run against the sole artifact backend. Parity fixes cover omitted checkpoint cursors, interrupted finish resolution, cancelled staging and competing phase transitions. - **Helm deployment wiring included:** opt-in `reverseEtl` values for runner image, existing runtime Secret, dedicated runner service account and cloud identity annotations; namespace-scoped lease-only RBAC, separate from connector jobs. - Configurable CPU/memory/ephemeral-storage requests and limits plus disk-backed `/tmp` size limit for scheduled, manual and recovery runs. syncctl validates settings on startup; required store/bucket Secret keys prevent missing-storage starts. - Helm value validation, offline render tests (Helm 3/4), and syncctl unit tests in a dedicated CI workflow. Cached generated Zod output is cleared before regeneration. ## Google full audience replacement and delivery observability - New-sync setup offers **Mirror · full replacement** alongside existing snapshot-diff mirroring. Supported for managed and existing owned audiences, with explicit exclusive management/takeover confirmation. Existing sync configurations are not converted. - Capture and durably save Google's cutoff before uploads; upload every unique desired member, including unchanged members; wait for all uploads to be accepted before submitting `audienceMembers:removeAll` with that original `removeAsOfTime`. - Persist the cleanup receipt and poll it on status-check jobs without re-extracting or re-uploading. Missing/ambiguous cleanup receipts block replay. Promote and prune the durable baseline only after confirmed cleanup. - Replacement is asynchronous, not atomic. A successful empty snapshot clears the audience; no other tools/users may upload to an exclusively managed target. Failed extraction or rejected uploads never authorize cleanup. - No additional schema changes for replacement. Google does not expose a removed-member count for cleanup; logs explicitly distinguish uploaded/accepted records from cleanup status and matched/targetable audience size. - Adds source/deduplication/comparison counts, short delivery-stage updates, one cumulative delivery summary per attempt, and actionable redacted failures. Pending Google work is presented as a status refresh rather than recovery. ### Latest validation - Runner: 265 passed, one opt-in scale test skipped; typecheck and bundled build pass. - Destination functions: 209 tests passed; Google replacement coverage uses mocked HTTP. - Console: 33 focused Reverse ETL integration tests and 136 unit tests passed; typecheck passes. - Covers all-upload gating, empty replacement, lost cleanup receipts, status-only retries, pruning/restoration, configuration confirmation and unchanged default behavior. - No live audience changes or deployment performed for this feature; deployment remains manual. ## Rollout — destructive schema cutover - **Manual deployment.** No `deploy:console` label. Runner deploys separately. - **Pause schedules and drain old Reverse ETL workers before applying Prisma.** Back up the database; reconcile pending provider requests with the old runner/schema; explicitly retire/reset test syncs and audiences first. - Applying the schema drops the six old tables and five columns. Prisma may require explicit data-loss confirmation. Do not bypass that in unattended deployment. - **Explicit test-sync reset, no migration or automatic deletion**, as requested. Prefer a fresh disabled test sync with a new empty Jitsu-managed audience. Deleting old state alone loses the removal baseline and is not a safe reset. - Configure `RETL_OBJECT_STORE` / `RETL_OBJECT_BUCKET`, then deploy the new runner and verify storage before enabling new syncs. Node >=22.13; Node24 recommended. - Old workers cannot run after the schema cutover. Rollback requires coordinated backup restoration and provider reconciliation, not just unsetting storage. - No automatic object GC yet. Do not use age-only bucket deletion; referenced baselines and unresolved batches must survive. Monitor orphan/bucket growth. - Existing checkouts may retain six ignored generated Zod files for removed models; cleanup instructions are in `services/retl-runner/src/artifacts/README.md`. - This restack does not deploy or change live state. Earlier explicitly authorized local diagnostics rebuilt the runner and reset the paused test sync; they are not an automatic migration/deployment step provided by this PR. See `services/retl-runner/src/artifacts/README.md` for detailed cutover, permissions, storage configuration and retention precautions. `helm/REVERSE_ETL.md` documents GCS/EKS/Minikube authentication, runtime Secret keys, resources, and the pre-upgrade schema hook hazard: pause/drain **before** invoking Helm, not as part of its upgrade. ## Verification - **Recovery/restack validation:** console and runner TypeScript checks pass; runner bundle builds; 229 runner tests pass (1 opt-in scale test skipped), 90 console integration tests pass with the reduced schema, and all 136 console unit tests pass. Warehouse-query: 170 tests pass. The 9 committed Helm tests and strict lint pass. Scoped runtime/Helm reviews, verification of the corrected storage quickstart, and fresh final review cover the recovered changes and conflict resolutions. - **222 runner tests passed**, including the opt-in million-identity SQLite test. - Runner TypeScript check and bundled build pass. - Console TypeScript check passes after Prisma regeneration and removal of obsolete ignored generated model files. - Tests use disposable PostgreSQL with the reduced Prisma schema, local SQLite, immutable test objects, the real S3 SDK against a local HTTP service, and GCS doubles. - Coverage includes missing legacy tables/counters, legacy reset rejection in every phase, lost startup/commit responses, restart, partial failure, staged finish continuation, phase conflicts, async mirrors/removals, corruption, duplicate operation IDs and snapshots larger than one artifact. - Incremental local runtime/schema reviews and a fresh final pre-push review completed clean for `6061f083..18ce2f9`. - Current deployment follow-up: full syncctl unit suite passes (`go test -vet=off ./... -count=1`; pre-existing `fmt.Errorf` vet error is not changed). - Strict Helm lint and all 7 offline chart contract tests pass on both Helm 3.18.4 and Helm 4.2.2. Coverage includes disabled defaults, required configuration, GKE/EKS annotations, external service accounts, least-privilege RBAC, resource propagation, duplicate/conflicting env rejection and no automatic data-loss flag. - Previous revision also passed the Node24 bundled-SQLite smoke test. No live GCS/S3 bucket or advertising API was called during validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Reverse ETL console workflow for
JITSU-227using Jitsu's existing visual style and components.Safety/design
Validation
console:devbrowser walkthrough against a separate synthetic database: model preview, list/detail/run screens, existing-audience wizard/mapping, lost-response retry with identical request, Google destination card, explicit unchecked managed-audience confirmations. No JS errors; no page overflow at 1024px/390px.Deployment
Manual deployment, per request. Do not add
deploy:console. Console and runner deployments remain separate.