Umami answers 500 for about 80 seconds after each weekly database-credential rotation

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

Observed on the 2026-09-23 weekly database-credential rotation (details on #2915): the rotation invalidates Umami's old database password immediately, and an Umami server pod still holding it answers HTTP 500 (Prisma P1000, Postgres 28P01) until the Reloader-driven rollout replaces it. The window was about 80 seconds, and it recurs every week at the same time.

Audience and problem

Site visitors and dashboard users of every tenant tracked by Umami. Any request that needs a new database connection on the old pod during that window fails, so tracking events can be dropped and dashboard loads can error. #2915's fix makes the tenant reconcile job retry through the gap; it does not close the gap itself.

Hypothesis

If the old and new credentials overlap during rotation (the old password stays valid until the rollout onto the new one has finished), the weekly rotation stops producing any 5xx. The same rotation pattern is shared by other apps (fleetdm), so the mechanism should be reusable.

Success signal

Across the next two Wednesday rotations: zero 28P01 authentication failures from Umami pods in the Postgres logs, and zero 5xx from Umami in the rotation window. Guardrail: the rotation still happens weekly, and no credential outlives its overlap window.

Acceptance criteria

• A design choice recorded here: overlapping credentials at the secret-store role, or an ordering that restarts consumers before the old password is revoked.
• Implemented for Umami, with the mechanism usable by the other rotated-credential apps.
• Verified live on a real rotation against the success signal above.

Rough size: medium (design plus one rollout).