Skip to content

fix: pgrst_ddl_watch triggering PostgREST notifications unnecessarily - #2464

Draft
laurenceisla wants to merge 2 commits into
developfrom
laurence/pgrst-ddl-watch-fix
Draft

laurenceisla wants to merge 2 commits into
developfrom
laurence/pgrst-ddl-watch-fix

Conversation

@laurenceisla

Copy link
Copy Markdown
Contributor

It should not NOTIFY:

  • When a trigger is created on a temporary table
  • When objects are created inside Supabase schemas: auth, realtime, storage

What kind of change does this PR introduce?

  • Updates the pgrst_ddl_watch event trigger

Additional context

We've had reports where the event trigger notifies PostgREST unnecessarily, generating frequent queries to the Schema Cache, which can result expensive.

Action Items

  • New extension releases were Checked for any breaking changes
  • Extensions compatibility Checked
  • Backup and Restore Checked
    • Proceed to backup testing while extensions are enabled
      • After every restore, re-run the tests specified at point 3.1

Extensions compatibility testing

  1. Enable every extension
    1. Check Postgres’ log output for any error messages while doing so
      1. This might unearth incompatibilities due to unsupported internal functions, missing libraries, or missing permissions
  2. Disable every extension
    1. Check Postgres’ log output for any cleanup-related error messages
  3. Re-enable each extension
    1. Run basic tests against the features they offer, e.g.:
      1. pg_net - execute HTTP requests
      2. pg_graphql - execute queries and mutations
      3. …to be filled in

Backup Testing

Follow the testing steps steps for all the following cases:

  • Pause on new Postgres version, restore on new Postgres version
  • Pause on older Postgres version, restore on new Postgres version
  • Run a single-file backup backup, restore the backup

Testing steps

  1. Generate dummy data
    • the ‘Countries’ or ‘Slack clone’ SQL editor snippets are decent datasets to work with, albeit limited
  2. Save a db stats snapshot file
    • Do this by running supa db-stats gather -p <project_ref>
  3. Backup the database, through pausing the project, or otherwise
  4. Restore the backup, through unpausing the project or cli
  5. Check the data has been recovered successfully
    1. Visual checks/navigating through the tables works
    2. Run supa db-stats verify against the project and the previously saved file

- When a trigger is created on a temporary table
- When objects are created or dropped inside Supabase schemas: auth, realtime, storage
@laurenceisla
laurenceisla force-pushed the laurence/pgrst-ddl-watch-fix branch from 3cd5e45 to 38501ca Compare September 23, 2026 02:41
@team-gigang

Copy link
Copy Markdown

We're hitting this too. Really hoping this lands soon 🙏

wnsals1346 added a commit to Gigang-ST/gigang-client that referenced this pull request Sep 23, 2026
Realtime 클라이언트가 접속하면 Supabase가 `realtime.messages` 일자 파티션을
정비하면서 `ALTER TABLE ... OWNER TO`를 소유자가 이미 맞는데도 매번 실행한다.
그게 `pgrst_ddl_watch`를 울려 PostgREST가 스키마 캐시를 전면 재적재한다.

prd 실측:
- `ALTER TABLE realtime.messages_*`  272회/일
- 스키마 캐시 재적재                 약 170회/일 (1회 940ms, 그중 77%가 타임존 조회)
- DB 실행시간의 6.4%

2026-09-23 84분 장애가 이 경로였다 — `PGRST002` 336건이 한 시간에 몰렸고
(나머지 23시간은 0건) Postgres 재시작으로 끝났다. 재적재가 `authenticator`의
statement_timeout(8초)을 넘기면 PostgREST가 캐시를 못 만들고 무한 재시도에 빠진다.

우리 코드 버그가 아니라 Supabase 쪽 버그이고 수정이 진행 중이다
(supabase/postgres#2464 — `realtime` 스키마 DDL은 NOTIFY하지 않게).

이 레이어가 방아쇠인 이유는 파티션 정비가 **클라이언트 접속에 반응**하기 때문이다.
루트 레이아웃에 붙어 비로그인 포함 모든 방문자가 채널을 열게 하므로 깨우는 횟수가
최대가 된다. 실측 대조로 prd 272회/일 vs dev 약 20회/일.

차단은 관문 한 곳에서 한다 — 채널을 여는 `PresenceLayer`가 안 뜨면 연결 자체가
없다. 전광판의 `지금 보는 중 N명`은 인원이 0이라 스스로 안 그려지므로 따로
건드릴 게 없다(`presence-count.tsx`가 `count === 0`이면 null).

되살리려면 `PRESENCE_ENABLED`만 true로. 아래 코드는 한 줄도 안 건드렸다.
조건은 셋 중 하나 — ① supabase/postgres#2464 반영 ② PostgREST 16+ (#5100로
재적재 비용 80% 감소) ③ RAM 상향.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owltus added a commit to Owltus/back_office that referenced this pull request Sep 24, 2026
…r au lieu de 7 980, zéro erreur en régime normal

L'utilisateur a relevé 354 erreurs PostgREST et 354 avertissements API Gateway
sur son tableau de bord. Même chiffre : ce sont mes pings de préchauffage —
six requêtes en refus de permission par minute × 59 minutes. Je noyais le
dashboard sous mon propre bruit.

Recherche faite avant de toucher au code (« regarde comment les autres font »).
Trois constats, sources en tête du handler `sonder` :

1. La pratique documentée pour une instance gratuite est UN ping tous les
   3 jours, pour éviter la pause à 7 jours d'inactivité. Personne ne fait sept
   par minute, et cette app n'en a même pas besoin : usage quotidien + import
   nocturne.
2. Un refus de permission est tranché AVANT tout accès aux données. Ces pings
   revenaient en 300 ms pendant que la base ne servait plus rien : ils ne
   prouvaient rien de ce qu'ils prétendaient surveiller.
3. LE MÉCANISME DE LA PANNE EST UN BUG SUPABASE CONNU : supabase/supabase#50043.
   La maintenance quotidienne des partitions realtime.messages exécute ~90
   `ALTER TABLE … OWNER TO` par jour ; chacun déclenche pgrst_ddl_watch →
   NOTIFY pgrst → PostgREST reconstruit tout son cache et répond 503 pendant
   ce temps (371 s d'indisponibilité en 28 h chez le rapporteur). Correctif
   supabase/postgres#2464 encore en BROUILLON ; pgrst_ddl_watch appartient à
   supabase_admin, aucun contournement côté projet. Nos logs le montraient
   exactement : cinq NOTIFY la même seconde, rechargements à 1,2 / 2,0 / 6,4 s,
   puis la boucle de redémarrage.

Ce que le Worker peut faire : ne pas aggraver, et DIRE quand ça casse.

`sonder()` : UNE requête `/auth/v1/health` toutes les 10 min (cron
`*/10 4-22 * * *`). 200 = aucune ligne de log d'erreur. Échec = console.error,
donc visible comme ERREUR dans le dashboard Cloudflare — là où l'ancien
dispositif affichait « 0 Errors » en pleine panne parce qu'il avalait tout.
Cette URL a répondu 504 pendant toute la panne du 24/09 : la sonde l'aurait vue.

Limite assumée : /auth/v1/health prouve GoTrue vivant, pas Postgres servant.
Une sonde qui LIT exigerait public.ping() exécutable par anon, ce qui
contredit verif_advisor.sql n° 2 (red team). Décision de sécurité : à
l'utilisateur.

Déployé : `wrangler deploy` + `wrangler triggers deploy`, les deux calendriers
annoncés. Déclenchement réel à constater dans les logs.

CLAUDE.md : la règle « rafale de trois pings » est marquée FAUSSE et remplacée ;
le bug #50043 et la recommandation Supabase Nano → Micro sont consignés.
Post-mortem : section 9, cause mécanique et classement révisé des hypothèses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011aYzWQ6FKXaVEhM4ix8i17
@laurenceisla
laurenceisla force-pushed the laurence/pgrst-ddl-watch-fix branch from 38501ca to 151cf1d Compare September 25, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants