End a refusal run at a claim that was sent, not only at our own late row

[zaoxing]

A small fix to the publisher lease's 2 x TTL latch in CaptureStorageService: a claim whose INSERT was sent now ends a run of refusals, so the service no longer latches over a catalog that nobody held for part of the run.

The defect

ensure_publisher_lease latches the service ("held by another publisher for over 2 x TTL") once lease refusals have lasted two TTLs. #159 made a refusal by the service's own late claim row restart that clock. The comment there gives the reason: a claim sends its INSERT only once its head read found no live holder, so any rival's earlier refusals ended there.

The same reasoning applies to a claim whose INSERT was sent and then timed out, but that path (the std::exception branch) never reset the clock. So refusals by two different rivals, separated by a stretch where nobody held the lease, added up:

1. A rival holds the lease and refuses the service, which starts the clock.
2. The rival stops and releases with a tombstone, well inside 2 x TTL.
3. The service's next claim finds the lease free and sends its INSERT. The request times out (the 1 s claim bound at a 3 s TTL), and the writer quarantines for a TTL.
4. A second publisher takes the free lease.
5. The quarantine ends, the second publisher refuses the service's claims, and the clock — still running from step 1 — passes 2 x TTL. The service latches for good.

The fix

In that branch, writer_.quarantined() is exactly the case where the INSERT may have been sent: CatalogWriter::acquire_lease quarantines a non-renewing claim only when claim_insert_sent(). The branch already used it to tell a claim that wrote nothing from one that may have. When it is set, the service now resets held_elsewhere_since_ns_, as it does for a refusal by its own row. A real takeover still latches: the clock restarts at the second publisher's first refusal, and a holder that keeps refusing for 2 x TTL from there is latched as before.

native/csrc/catalog/storage_service.cpp only; 10 lines.

Tests

• New: tests/test_native_capture_storage_live.py::test_a_claim_sent_while_nobody_held_the_lease_restarts_the_refusal_clock reproduces steps 1–5 against a real ClickHouse. A _Switch holds the claim INSERT open and never forwards it, so no late row of the service's own is involved. It fails on main with indexing stopped: publisher lease held by another publisher for over 2 x TTL ... held by 'second-publisher', and passes with the fix (3 of 3 runs).
• Related suites, against ClickHouse 25.12 on the dev box:
  • test_native_capture_storage_live.py -k 'lease or latch or rival or claim or start': 30 passed. This includes the existing latch tests: a rival that takes over still latches, a rival that stops within two TTLs does not, and our own late claim does not.
  • test_native_lease_request_bound.py, test_native_catalog_lease_live.py and test_native_capture_storage_wiring.py: all passed.

How it was found

The LeaseLifecycle TLA+ model on the specs/formal-models-archive branch, re-checked against main @ 5b3b632. Config LeaseLifecycle_O3_unkrival violates NoFalsePositiveLatch with this trace. No spec files are part of this PR.

[Copilot]

Copilot review overview

🟡 Changes recommended

Quarantine expiry can bypass the reset, and the regression test does not reliably establish the initial rival refusal.

Review effort: Balanced
Findings: 2

Open (2)

• Use claim outcome to reset refusal timing after quarantine expiry · New
• Wait for confirmed rival ownership before recording refusal · New

What changed in this PR

Updates the publisher lease latch so a timed-out claim INSERT ends the preceding run of rival refusals.

Changes:

• Resets the refusal clock when a failed claim quarantines the writer.
• Adds a live regression test covering two rivals separated by a free lease.

File	Description
tests/​test_native_capture_storage_live.py	Adds the timed-out claim handover regression test.
native/​csrc/​catalog/​storage_service.cpp	Resets refusal tracking after a quarantined claim failure.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.