BUZZ-175: Add application-owned writer lock foundations

[TheSentinel454]

Summary

Buzz is moving database invariants from triggers and foreign keys into application-owned transaction protocols. This PR adds the first transition foundations while preserving all current database backstops.

• Add a shared community-write lock for single-community writer transactions, and make deletion lifecycle transitions take the matching exclusive lock. A supported serving transaction never spans communities.
• Add shared event-writer and exclusive replica-probe lock ordering around the existing heartbeat publication model.
• Document the supported writer contract, reviewed backfill path, and transition gates.
• Keep one replica-floor publication concept and remove an unrelated reconciliation-script edit found during review. This narrows the change rather than adding parallel protocols.

The blast radius is limited to new opt-in writer helpers plus lock acquisition in deletion and replica probing. Existing trigger and GUC enforcement remains authoritative, so callers can migrate in later pull requests.

Related issue

BUZZ-175. This is the first implementation slice from the OSC migration plan.

Testing

No manual testing. CI covers the Rust checks and database tests added here.

Generated with Codex

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is ea143ea64beb79abacf423920cd70bbf23698b64...12a45379251adc9d4d57300cd2854aac621f10b7.
A new review must complete for this exact range. When manual authorization
is required, a user with write access must comment exactly
@buzz-security-review 12a45379251adc9d4d57300cd2854aac621f10b7 to authorize a new review.
Any previous review applies only to its recorded range.

[wesbillman]

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Review clear: no actionable introduced defect found at d9e0b3aea49a54c2caa57bf1b5a539838eecc54c against 4ab4f786085a23fe6126529861840eff6048ceee. Two nonblocking test-assertion improvements are noted inline.

Reviewed community admission/deletion ordering, probe lock → sample → activity scan → heartbeat commit → ring publication, rollback/error paths, and mixed-version compatibility. This head adds single-community, opt-in helpers; production deletion behavior and existing trigger/GUC backstops remain unchanged. The PR description should reflect that narrower scope.

Existing PostgreSQL CI, verified against this head/base pair, passed 428/428 tests including all five added witnesses. Review was source-only; no additional test, mutation, or deployment execution. No blocking changes requested.

[wesbillman]

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Changes requested: one P1 deletion-fence regression at 9e178ab098b44f12e55bdd1d624b11acbf4176db, against base 4ab4f786085a23fe6126529861840eff6048ceee. The new writer coexistence and exact-denial assertions address the prior suggestions, but the accompanying production change removes lifecycle exclusivity. Restore that barrier and cover the production transition without an incidental foreign-key row lock before merge.

Reviewed writer admission through quiescing/fencing/abort, trigger and serving-lease backstops, and replica probe transaction/ring publication. Source-only on the pinned Blox; no checkout, build, tests, or PR-code execution. Existing exact-head PostgreSQL CI passed; Desktop Smoke E2E (4) failed and was not diagnosed here.

Re-review

[wesbillman]

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Review clear: the prior P1 lifecycle-lock regression is repaired at a6a3032e446e6e66e8e41a229ef655ea79f36202, against base 8f8c4dfadecb2298a9140af5d323487aaad844ab. No new blocking defect found in this bounded corrective follow-up.

lock_community_deletion is exclusive again; quiescing, fencing, and abort use it while admitted writers retain the shared counterpart. The new regression exercises real begin_quiescing against an open update of an existing allowlist row’s non-key note, avoiding the earlier incidental parent-FK lock. The 50 ms unfinished-task assertion is timing-based, not deterministic mutation proof; that remains a nonblocking test limitation.

Reviewed the repair and necessary seven-path merge integration. The APIs remain opt-in, existing trigger/GUC backstops remain authoritative, and replica proof publication still follows heartbeat commit. Unrelated merged upstream changes were excluded.

Existing PostgreSQL CI passed 486/486 tests, including the new regression. Its checkout was synthetic merge 2a5f42225a11f4169c3fa6753fb5a2cdb4562b2e, containing this head merged into 676e8c43825fa478850a3a320b1367d3288cce1a, not the exact pinned base pair. Source-only review on Blox; no checkout, tests, mutation run, or PR-code execution performed by this review. No blocking changes requested; this is not an approval.

[wpfleger96]

🤖 No blocking defects at a6a3032e4. This is a clear review, not an approval.

This agrees with the earlier automated review at this head. The lifecycle helper is exclusive again, so begin_quiescing, fence and abort all wait for admitted writers holding the shared key. Abort now locks in the order advisory lock, community row, request row, which matches the other lifecycle transitions. sample_writer takes the exclusive floor lock before it samples S, and it commits the heartbeat before probe_once records the ring entry. So a failed or dropped probe never publishes an uncommitted token. The PostgreSQL lane passed 486/486, including all six new tests and the updated abort/quiesce test.

A few small things I think are worth fixing in this PR:

1. Two lock-mode witnesses can't detect a missing lock. runtime/tests.rs:3125-3154 and store/deletion.rs:4146-4177 keep shared_contender open while they try the exclusive lock. That contender blocks the exclusive attempt on its own, so both tests would still pass if the helper took no lock at all. They catch shared-vs-exclusive but not shared-vs-absent. The probe-wait and begin_quiescing_waits_for_open_admitted_writer_note_update tests do cover a missing lock, so this is about test precision, not a coverage hole. Rolling back shared_contender before the exclusive attempt fixes both.
2. The PR body overstates the scope. It says this adds "stable ordering for multi-community writes". The code adds a single-community wrapper, and the new ARCHITECTURE.md section says a supported serving transaction never spans communities. That also answers your multi-community question on runtime/mod.rs. I'd narrow the summary bullet to match.
3. Something to design for when callers migrate: a compliant writer holding the shared floor lock for a long time stalls the probe, and an exclusive waiter queued in Postgres also blocks new shared requesters behind it. So one slow writer can hold up every other compliant channel-event writer until the writer's lock_timeout (5s by default) fails the probe. Nothing calls begin_replica_floor_locked_event_write_transaction yet, so nothing is affected today. When the first caller lands, it's worth bounding how long writers hold the shared lock, or giving the probe a short SET LOCAL lock_timeout so a missed sample doesn't block writers.
4. Nits:
   • The only change left in scripts/reconcile-schema-after-pgschema.sql deletes a blank line. The body says that unrelated edit was removed, so I'd drop the file from the diff entirely.
   • A blank line now separates the doc comment from its test at replica_fence.rs:1101-1102.
   • let mut sampling = sampling; (replica_fence.rs:1065) and let mut probing = probing; (runtime/tests.rs:3194) can be folded into the let mut on the spawn.

The 50/100ms "still waiting" assertions remain timing-based, as the earlier review noted. That's fine for this slice.

[TheSentinel454]

@buzz-security-review 12a4537

[wpfleger96]

🤖 No blocking defects at 12a453792. This is a clear review, not an approval.

The three commits since my last review at a6a3032e4 address its notes:

• Both lock-mode witnesses now roll back shared_contender before trying the exclusive lock (runtime/tests.rs:3135-3138, store/deletion.rs:4158-4161). The writer is now the only lock holder when the exclusive attempt runs, so both tests fail if the helper takes no lock.
• scripts/reconcile-schema-after-pgschema.sql is out of the diff, the stray blank line before the masked-activity test is gone, and both let mut rebinds are folded into the spawns.
• The PR body now describes a single-community write lock, which matches the code and ARCHITECTURE.md.

The branch is based on main from 09-24, and main has since added owner deletion admission and automatic preparation in deletion.rs. It still merges cleanly. The reordered abort (advisory lock, then community row, then request row) uses the same order as main's new complete_owner_preparation (shared advisory lock, community row, owners, request row), so the new path doesn't bring back the deadlock this PR fixes.

When this was published, the PostgreSQL lane that runs the changed #[ignore] tests was still queued. The failed Codex Security Review job was cancelled during its own output validation, so it says nothing about this diff.

[TheSentinel454]

Re: review at a6a3032 @wpfleger96

Thanks. Items 1, 2 and 4 are fixed in 12a4537, as your re-review confirms. On item 3: agreed. It only matters once something calls begin_replica_floor_locked_event_write_transaction, and nothing calls it in this PR or #7828. I'll add the bound with that first caller: a short SET LOCAL lock_timeout on the probe's exclusive floor lock, plus a limit on how long writers hold the shared lock.