fix(cdc): emit fallback notification for oversized payloads + visible errors#1410
Conversation
Schema constraint shape reportdata-shape report errored (exit 0): node:internal/modules/runmain:107 triggerUncaughtException( ^ Error ERRMODULENOTFOUND: Cannot find package 'postgres' imported from /home/runner/work/Backend-Service/Backend-Service/scripts/schema-shape-report.mjs Did you mean to import "postgres/cjs/src/index.js"? at Object.getPackageJ; manual check required |
Code reviewFound 1 issue:
|
… errors (#1120) Postgres `pg_notify` enforces an 8000-byte payload cap. The previous `cdc_notify()` trigger function wrapped the call in a broad `EXCEPTION WHEN OTHERS ... RAISE WARNING ... RETURN NULL`, so when a row's `to_jsonb(NEW)` overflowed the cap (e.g. an enrichment-worker `flowsheet` UPDATE with `artist_bio` + seven streaming URLs) the notification was silently dropped — the originating mutation committed, but no consumer ever saw the event. The only signal was a buried PG-log WARNING line. Migration 0094 replaces the trigger function so it: - Detects the oversized case up-front (`octet_length(payload_text) > 7800`) and emits a minimal `cdc_oversized` fallback notification carrying `(table, schema, action, primary_key, payload_bytes, reason='payload_too_large')` so downstream consumers can refetch the row from the source of truth. - Routes any other unexpected exception through a dedicated `cdc_error` channel (`reason='trigger_exception'`) in addition to the existing `RAISE WARNING`, restoring listener visibility into trigger failures. - Returns `NEW`/`OLD` instead of bare `NULL` so the trigger contract is unambiguous if a future maintainer rewires it as `BEFORE`. Tests in `tests/integration/cdc-oversized-fallback.spec.js` verify both paths against real PG: normal INSERTs still fire `cdc`, oversized INSERTs fire `cdc_oversized` (and NOT `cdc`), and the row commits regardless. Also pins the function body so an accidental rollback of 0094 fails CI before redeploy. The migration header comment documents the new notification channels for listener-side consumers.
…for #1120 AC #3 Migration 0094 emits to two new pg_notify channels (cdc_oversized, cdc_error) when the primary cdc payload would have been silently dropped. Without consumers, those notifications go to /dev/null and AC Wire the consumer side end-to-end: - shared/database/src/cdc-listener.ts: define CdcOversizedEvent and CdcErrorEvent shapes matching the SQL payloads; add onCdcOversizedEvent and onCdcErrorEvent registrations; LISTEN on both new channels alongside the existing cdc subscription in startCdcListener; clear the new callback arrays in stopCdcListener. - apps/backend/services/cdc/dispatcher.ts: wire the dispatcher's fallback sinks to Sentry.captureMessage with stable fingerprints ('cdc-oversized-payload' / 'cdc-trigger-exception'). Module-level latch keeps the registration idempotent across stray startCdcDispatcher calls; shutdown drops the latch. - apps/enrichment-worker/worker.ts: mirror the Sentry wiring in the worker process so its independent LISTEN connection also surfaces the fallback channels (consumer='enrichment-worker' tag for disambiguation). Tests: - BS#1120 describe block in cdc-listener.test.ts pins channel subscription, callback dispatch, multi-callback fan-out, callback-error isolation, malformed-payload tolerance, and stopCdcListener teardown of the new callback arrays. - BS#1120 describe block in cdc-websocket.test.ts (alongside the existing dispatcher tests) pins the dispatcher's Sentry wiring, captureMessage tag/extra/fingerprint shape, double-start idempotency, and shutdown-resets-latch behavior. - Updated the BS#1014 enableLivenessProbe channel-order assertion to cover the new cdc_oversized + cdc_error subscriptions.
93a824e to
d4f8ca8
Compare
Closes #1120
Summary
Migration 0094 replaces the
cdc_notify()trigger function so it stops silently dropping CDC events when a row's JSON payload exceeds Postgres's 8000-bytepg_notifycap, and stops swallowing other trigger exceptions into the PG log only.octet_length(payload::text) > 7800) and emits a minimalcdc_oversizedfallback notification carrying(table, schema, action, primary_key, payload_bytes, reason='payload_too_large')so downstream consumers can refetch the row from the source of truth instead of going dark.cdc_errorchannel (reason='trigger_exception') in addition to the pre-existingRAISE WARNING, restoring listener-side visibility into trigger failures.NEW/OLDinstead of bareNULLso the trigger contract is unambiguous if a future maintainer rewires it asBEFORE.Tests in
tests/integration/cdc-oversized-fallback.spec.jsexercise both paths against real PG: normal INSERTs still firecdc, oversized INSERTs firecdc_oversized(and NOTcdc), and the row commits regardless (visibility-only failure mode). A function-body pin guards against an accidental rollback of 0094.Test plan
npm run test:unit— 3151/3151)npm run lint— 0 errors)npm run format:check)npm run typecheck)npm run lint:migrations)scripts/check-precondition-guards.sh)scripts/check-bulk-update-analyze.mjs)integration-testsjob (touchesshared/database/src/migrations/**+tests/**)