0.22.0: concurrency and runtime ownership hardening

Author: llblab

AGENTS.md

@@ -109,6 +109,7 @@ export type TelegramDeliveryFailureReason =
   | "target-unauthorized"
   | "stale-handle"
   | "invalid-view"
+  | "commit-unknown"
   | "transport-failed";
 
 export type TelegramDeliveryResult<T> =
@@ -121,7 +122,7 @@ export type TelegramDeliveryResult<T> =
     };
 ```
 
-Expected availability, authorization, and lifecycle outcomes return failures rather than throwing. Programmer errors may still throw for malformed objects that cannot satisfy the TypeScript contract. Transport failures are redacted before reaching callers and are also recorded in bridge runtime diagnostics.
+Expected availability, authorization, lifecycle, and ambiguous mutation outcomes return failures rather than throwing. `commit-unknown` means a non-idempotent Bot API mutation may have committed before its response was lost; callers must not blindly replay it. When earlier chunks remain known, `partial` still carries their recoverable logical handle. Programmer errors may still throw for malformed objects that cannot satisfy the TypeScript contract. Transport failures are redacted before reaching callers and are also recorded in bridge runtime diagnostics.
 
 When a multi-chunk `send` or growing `edit` fails after materializing part of the logical view, `partial` contains a valid handle for every message still visible from that operation. Callers may pass it to `editTelegramView` to reconcile the view or to `deleteTelegramView` for cleanup. A failure before any message exists omits `partial`; callers must never infer message ids or construct a handle.
 

CHANGELOG.md

@@ -129,13 +129,17 @@ Delete `~/.pi/agent/locks.json` to reset singleton runtime ownership for all par
 
 ## Atomicity
 
-Current baseline is read-modify-write JSON. This is enough for interactive pi singleton starts.
+Every ownership mutation runs inside a short cross-process transaction acquired through exclusive file creation. The transaction covers the complete registry read/check/write sequence, so concurrent processes cannot both win an ordinary acquisition or stale-overwrite unrelated extension/profile keys. The registry payload itself still commits through same-directory temp-file replacement.
 
-If multiple instances may start concurrently, use an atomic helper later:
+A guard is written completely to a private staged file and published with one exclusive hard-link operation, so no live empty/partially written generation becomes recoverable by age. A crashed transaction owner leaves a complete guard that a later process recovers only when the recorded owner PID is no longer alive. Recovery contenders serialize through a second atomically published guard so one process cannot rename a newer live transaction generation. Cleanup compares the exact random guard generation before unlinking. Malformed guards, unsupported atomic publication, and unverifiable recovery fail closed after bounded contention rather than risking concurrent registry mutation.
 
-- Lock file around `locks.json`, or
-- Temp file + rename with conflict checks, or
-- OS-level exclusive open for a short critical section
+Transactional reads treat only a missing registry as empty. Read, parse, and shape failures abort without replacing the existing file or erasing unrelated ownership keys.
+
+Each runtime retains the profile key plus exact owner identity it acquired. New leaders receive a collision-resistant epoch independent from heartbeat timestamps and a monotonic same-process runtime generation. Refresh, release, and thread-state snapshot publication commit only while the profile key, PID, cwd, instance id, epoch, and runtime generation still match. Snapshot rename runs inside the same lock-registry transaction guard as its final owner check. Forced replacement additionally requires the exact previously observed owner; same-process lifecycle handoff permits only a newer runtime generation to replace an older one. A delayed old runtime cannot reverse that transition, publish prepared thread state, or reinterpret its retained owner token under another profile key.
+
+Direct Bot API authority follows this exact ownership check. Accepted local queue work may continue through Pi after ownership moves, but stale preview/final/menu/file mutations fail closed instead of bypassing the replacement transport owner. Follower recovery retries registration while the exact external owner remains live and promotes only after an election transaction confirms that the observed owner still remains stale or that no owner appeared after an inactive observation; IPC unreachability alone never authorizes takeover. Simultaneous election losers re-register with the winner using their carried exact thread target.
+
+The ownership heartbeat starts immediately after acquisition, before binding handoff or slow leader startup. It remains active through provisioning/server/polling startup; startup errors release the exact acquired lock after cleanup, and ownership loss during startup stops the partial runtime and returns failure rather than announcing leadership. Topic provisioning also stamps the acquired epoch and rechecks it before and after every persistence and Bot API boundary, including `epoch → undefined` loss.
 
 ## Migration
 

docs/architecture.md

@@ -155,15 +155,15 @@ Leader election is heartbeat-gated and lock-backed:
 4. If the leader heartbeat is stale, attempt an atomic leadership takeover; ordinary `/telegram-connect` on a follower is not a leadership move while the leader is live.
 5. If several followers detect stale leadership, atomic compare/write lock acquisition ensures only one becomes leader.
 
-Followers first try to re-register after leader reload or unknown-heartbeat responses, carrying their last known target, slot, and thread name so the new leader can reuse the same binding, then promote only after the grace window expires. If the exact live target is absent from persisted bindings, the leader recovers it instead of creating another Telegram thread and keeps its carried slot only when that slot is free. Every successful reuse refreshes the binding timestamp. The leader does not restore persisted followers into the live registry speculatively: after a bounded re-registration grace window it removes older follower records that still lack a live bus owner from current `state.json`, while preserving a recently refreshed binding across the brief registry gap caused by follower session replacement and never deleting Telegram tabs. This preserves real thread bindings through transient reload gaps without allowing historical records or competing pollers to masquerade as live state.
+Followers first try to re-register after leader reload or unknown-heartbeat responses, carrying their last known target, slot, and thread name so the new leader can reuse the same binding. After the grace window they promote only when the exact observed leader lease has become stale or inactive; an unavailable IPC endpoint never authorizes replacing a still-live owner. If the exact live target is absent from persisted bindings, the leader recovers it instead of creating another Telegram thread and keeps its carried slot only when that slot is free. Every successful reuse refreshes the binding timestamp. The leader does not restore persisted followers into the live registry speculatively: after a bounded re-registration grace window it removes older follower records that still lack a live bus owner from current `state.json`, while preserving a recently refreshed binding across the brief registry gap caused by follower session replacement and never deleting Telegram tabs. This preserves real thread bindings through transient reload gaps without allowing historical records or competing pollers to masquerade as live state.
 
 ## Leader/Follower Communication
 
 Implemented transport:
 
 ### Local IPC endpoint under agent temp dir
 
-Leader opens a local Node `net` endpoint: a Unix-domain socket under the agent temp directory on Unix-like platforms, or a deterministic Windows named pipe (`\\.\pipe\pi-telegram-...`) on native Windows. Followers register, heartbeat, and exchange routed events. The transport boundary owns endpoint derivation, socket-vs-pipe detection, bounded operation-aware retry policy, timeout/transient IPC error classification, endpoint reachability probes, and request-scoped transport events. Follower registration uses a longer registration-specific response timeout than ordinary heartbeat/forwarding calls because the leader may need to provision a Telegram thread before it can return the assigned target; timing out that handshake leaves a visible tab with no follower heartbeat. Keep this handshake to the true critical path: create/reuse the target, persist the live binding, and return it. Connected notices and replaced-thread reconciliation cleanup are non-critical and should run after registration so a follower becomes routable before Telegram client/server UI convergence work finishes.
+Leader opens a local Node `net` endpoint: a Unix-domain socket under the agent temp directory on Unix-like platforms, or a deterministic Windows named pipe (`\\.\pipe\pi-telegram-...`) on native Windows. On Unix, each server listens on a private generation socket and atomically publishes the stable profile path as a relative symlink; delayed shutdown closes only its private path and cannot remove a replacement generation's link. Followers register, heartbeat, and exchange routed events. The transport boundary owns endpoint derivation, socket-vs-pipe detection, bounded operation-aware retry policy, timeout/transient IPC error classification, endpoint reachability probes, and request-scoped transport events. Follower registration uses a longer registration-specific response timeout than ordinary heartbeat/forwarding calls because the leader may need to provision a Telegram thread before it can return the assigned target; timing out that handshake leaves a visible tab with no follower heartbeat. Keep this handshake to the true critical path: create/reuse the target, persist the live binding, and return it. Connected notices and replaced-thread reconciliation cleanup are non-critical and should run after registration so a follower becomes routable before Telegram client/server UI convergence work finishes.
 
 Pros:
 
@@ -347,7 +347,7 @@ Current state under the agent dir:
 
 - `locks.json`: current bus leader identity, capability secret, heartbeat, and cleanup fencing epoch. The local bus endpoint is derived from the agent directory by default; legacy `busSocketPath` entries are tolerated but are not required.
 - `tmp/telegram/state.json`: volatile extension+bot observable/debug snapshot, not routing authority. It writes `source: "snapshot"` and `writtenAtMs` so consumers do not confuse it with an authoritative database. Every process on one Telegram profile reads this shared path, but only the active transport lock owner may persist it; followers become writers only after promotion. Status-only persistence refreshes disk-backed bindings before serialization so an already-loaded stale view cannot erase newer leader records. It mirrors `/telegram-status`-style projections: top-level `bot` stores bot-wide capability state such as `threadMode: "unknown" | "enabled" | "disabled"`, `runtime` identifies leader/follower role and process status, `liveRoster` mirrors followers/current targets/reservations, `diagnostics` mirrors status/debug signals, `threads` stores current routeable bindings, `bot.lastSlot` stores the compact slot cursor used when all current threads are gone, and `reservations` records short-lived slot collision guards.
-- Local bus endpoints: Unix-like platforms use `tmp/telegram/bus.sock` and `tmp/telegram/followers/*`; native Windows uses deterministic named pipes under `\\.\pipe\pi-telegram-...`. These are transient IPC endpoints, not durable routing state.
+- Local bus endpoints: Unix-like platforms expose stable `tmp/telegram/bus.sock` and `tmp/telegram/followers/*` symlinks backed by private generation sockets; native Windows uses deterministic named pipes under `\\.\pipe\pi-telegram-...`. These are transient IPC endpoints, not durable routing state.
 
 The bridge must not keep a durable `telegram-targets.json` history. Stale/offline/failed thread entries are reconciliation observations, not reusable source-of-truth state; persisting them increases collision risk. `sync` is event-driven assumption reconciliation, not full Telegram bot-state mirroring, because Bot API does not expose a complete topic/thread listing surface. `state.json` therefore exists for extension+bot observability, diagnostics, startup hints, and explaining reconciliation decisions; live bus/runtime state remains authoritative for routing and provisioning. Non-current routeable thread bindings are pruned during load/persist; old session records must not be retained just to compute the next slot because `bot.lastSlot` is the only durable cursor. Previous-process leader bindings are treated as occupied TTL-bounded reservations until Telegram confirms deletion: reload/startup may close/delete/probe the old thread, known reservations are retried proactively on leader startup, and if Telegram still accepts the old thread id, the new leader should provision the next free slot (`B`, `C`, …) rather than creating a duplicate same-letter tab or blocking startup on Telegram UI convergence. Routing must use live current threads/follower registry, never reservations. The bus leader provisions its own thread during bus startup/connect and provisions follower threads on `follower.register`; registered followers also live in the leader's in-memory registry and communicate over the local bus socket. The live follower registry can resolve a follower by exact `{ chatId, threadId? }`; the leader uses that target ownership to forward message and edited-message updates to followers, and the follower receiver accepts those updates in addition to callbacks and reactions. Terminal status and `[telegram|thread:name]` resolve the matching current-instance identity through the same target-aware path, preferring registered local metadata over stale shared bindings. Media album grouping and split-text coalescing keys include the thread target, queue reaction mutations can scope by chat/thread to avoid cross-target message-id collisions, active-turn target is exposed for lifecycle cleanup and local direct-tool defaults, transport reply dedup is chat/thread-scoped, stored menu state is keyed by chat/message so callback state lookup cannot collide across chats, and generated button turns plus section prompt/open actions preserve the callback thread target. `telegram_message` and immediate `telegram_attach` delivery can also carry an explicit `thread_id` with `chat_id`; when a follower is registered, their default direct-tool target is the assigned thread target and the bus-aware API runtime routes the send through the leader instead of calling Bot API transport locally.
 
@@ -372,7 +372,7 @@ All files containing routing, chat ids, thread ids, or process details use priva
 
 - Leader prunes the follower from the live registry after missed heartbeats, but heartbeat pruning is only immediate liveness bookkeeping.
 - A missed heartbeat does not delete, close, mark offline, or send a disconnected notice for the follower's Telegram thread binding because the common cause may be leader reload, IPC handoff, or transient reconnect rather than a dead follower.
-- Followers treat rejected/missing heartbeat acknowledgements as registration loss: retain the last known target locally, clear registered truth, try to re-register with the current leader, wait a short leader-reload grace window, retry, and then promote themselves if the leader still cannot route them.
+- Followers treat rejected/missing heartbeat acknowledgements as registration loss: retain the last known target locally, clear registered truth, try to re-register with the current leader, wait a short leader-reload grace window, and retry. They promote only after the exact leader lease becomes stale or inactive; a live owner with an unreachable endpoint leaves the follower disconnected/retrying rather than creating a competing poller.
 - After leader reload grace expires, persisted manual-follower records without a matching live registry owner leave current `state.json`; this compaction is non-destructive and does not close/delete the Telegram thread.
 - Every successful follower registration/re-registration sends a compact connected notice in the assigned thread so recovery and reconnection are visible during live testing without confusing heartbeat suspicion with real disconnect.
 - Successful forwarded updates and follower-originated API calls refresh liveness, so active followers are not pruned only because the interval heartbeat tick lagged.
@@ -426,7 +426,7 @@ Live client and native Windows evidence gates are tracked in `BACKLOG.md`; this
 - Local IPC is the default internal bus. Registered followers receive normalized inbound updates and send allowlisted, target-scoped Bot API calls through the leader.
 - Thread targets are current-state bindings, not durable historical delivery addresses. Stale/offline/failed entries are reconciliation evidence only.
 - Failover promotes a remaining follower after dead or clean-disconnected leaders without creating competing pollers; follower heartbeat recovery owns re-register → grace → promotion while preserving thread bindings across transient leader reload gaps.
-- Thread cleanup is centralized in `thread-reconciler`, fenced by leader epoch, and requires confirmed delete/stale evidence before state is marked deleted.
+- Thread cleanup is centralized in `thread-reconciler`, fails closed without a leader epoch while leadership exists, revalidates that epoch immediately before every close/delete call and local cleanup-state mutation, and requires confirmed delete/stale evidence before state is marked deleted.
 - Stable docs/UI now describe classic mode, opt-in Threaded Mode, manual follower registration, status/diagnostics, unbound-thread reroute/restore UX, and operator recovery boundaries.
 
 ## Evidence Gates