fix(control-plane): recover ambiguous canonical write responses safely

[huangruiteng]

Why / 背景

A promoted File authority can exceed the default five-second RPC budget while verifying retained history or waiting behind the per-Goal writer fence. A lost response is ambiguous: the operation may already be committed. The old bridge removed the runtime locator and retried, potentially creating concurrent servers. A retiring server could also remove its replacement locator.

A second failure appears after an exact Monitor provider write: an unrelated append to the run index can leave its durable transaction at provider_pending forever, because retry compares the current whole-index digest to the digest frozen before provider writeback.

Change / 改动

• Post-send timeout, disconnect, or unverifiable response is explicit ambiguous state; never blindly replay it or invalidate a live locator. Pre-send failure rechecks the locator without deleting it. A directly spawned exited child is reaped for genuine crash recovery. Server publication and cleanup are token-fenced.
• Terminal and Monitor mutations get explicit 45-second RPC budgets; canonical writer contention gets a bounded 30-second wait. Hard leases and provider CAS remain authoritative.
• Read-only todo receipt --operation-id retrieves an exact historical receipt from the selected canonical provider. It grants neither a lease nor retry authority.
• Pending Monitor settlement may resume after unrelated append-only run-index growth only when the original byte prefix and digest are intact, its effect is absent from the current index, its provider receipt matches the frozen plan, and the new prepared WAL fences the current index. Rewritten/truncated history and duplicate effects still fail closed.

User paths / 用户入口

CLI gains exact receipt readback and accurate mutation failure semantics. Managed Turn and Lark use the shared Effect/coordination runtime; there is no new frontend setting or separate source of truth. The active Goal was inspected read-only; no unmerged code was used to mutate it.

Verification / 验证

• Python runtime/authority/CLI suite: 84 passed before the extra pre-send case; full runtime integration suite: 46 passed including that case.
• TypeScript writer/receipt tests: 17 passed; provider failure matrix: 35 passed; Monitor transaction suite: 19 passed.
• Control-plane typecheck, Ruff, and git diff check passed.

Remaining / 后续

This bounded reliability fix does not solve File-v0 whole-history capacity growth or the distinct auxiliary Monitor selected-Todo identity conflict. Historical turns require exact provider receipt plus original Turn/lease/quota reconciliation after deployment. No quota spend is fabricated here.

[huangruiteng]

Validation note / 验证说明：直接运行 npm run test:control-plane 时，部分跨语言测试调用了系统 Python 3.9，因 dataclass(slots=...) 不受支持而失败；我中止了该无效环境下的长跑，没有声称全量通过。使用仓库 uv 测试环境隔离重跑后，先前 4 个 File provider 读回失败用例 4/4 通过，todo continuation 跨语言用例 36/36 通过。受影响的 Monitor transaction 套件 19/19、provider failure matrix 35/35、writer/receipt 套件 17/17、typecheck/Ruff 均通过。

[huangruiteng]

CI follow-up / CI 修复：旧 head 的 test-shard (1) 失败有两处测试预算漂移：terminal test spy 未转发新增 timeout 参数；checkpoint probe 和测试外层的 30/15 秒 deadline 短于新的 30 秒 canonical writer 锁等待。9862bd411 仅更新测试桩与探针预算，不改变生产写入语义。使用 uv 测试环境重跑两份完整测试文件：41 passed；受影响的 6 个参数化用例全部通过；Ruff 与 git diff --check 通过。新 head CI 已重新触发，仍待评审。

[huangruiteng]

CI follow-up 2 / 第二轮 CI 修复：旧 head 的 test-shard (4) 只报架构清单漂移。新增 Todo CLI registry codec read 后，项目 registry I/O census 未更新。d846e640f 使用仓库生成器刷新 manifest，新增站点仍为既有 codec_api 分类；架构相关测试 9/9、manifest --check、git diff --check 通过。生产行为未变，新 head CI 已重新触发，仍待评审。

[huangruiteng]

CI follow-up 3 / 第三轮 CI 修复：node-minimum-compatibility 在 Node 22.18.0 上运行了 4 条 SQLite authority 用例，但该 Node 内置 SQLite 3.50.2 不满足仓库明确的 WAL-reset/statement-finalization 准入；File authority 才支持公开 Node minimum。f8489b184 沿用既有 sqliteRuntimeIdentity 资格检查，在未合格 runtime 跳过 SQLite 用例，并保留失败时的具体 provider 回执。复验：Node 22.18.0 File 4 pass / SQLite 4 skip；合格 Node 22.22.3 与 Node 24 均 8/8 pass；control-plane typecheck、diff --check 通过。生产行为未变，新 head CI 已重新触发。

[huangruiteng]

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机

Canonical Effect 写入可能超过原有 RPC 等待时间；请求已发送而响应丢失时，客户端不能判定写入是否提交。旧的运行时桥接会移除 locator 并重试，可能同时启动两个服务。另一个独立但相邻的问题是 Monitor 的 provider_pending 结算把其他正常追加的索引记录当成冲突。本 PR 的方向正确：把不确定结果显式化，通过原 provider 的精确 receipt 只读核对，并只在原索引前缀未变时恢复结算。

改动思路

请求发送后发生 EOF、超时或响应形状错误时，Python 桥接抛出 EffectRuntimeResponseAmbiguous，保留当前服务 locator，禁止盲目重放。TypeScript 服务端用 token 与文件锁约束 locator 发布/清理；todo receipt --operation-id 读取所选 canonical provider 的历史 receipt，不授予 lease 或重试权限。Monitor 在 WAL 恢复时验证冻结的索引字节前缀、provider receipt 及 effect 缺席，再把当前索引纳入 fence。写入锁与相关 RPC 等待预算分别延长到 30 秒和 45 秒。

具体改动

关键调用链：_request_with_info 负责区分发送前失败与发送后不确定结果；effect_runtime_request 对后者不再删除 locator 或重试；readLocalCoordinationOperationReceipt 委托现有 AuthorityStore.readReceipt；pendingIndexHistoryIntact 只允许经验证的追加增长。审查了这些生产入口、直接调用 _request_with_info 的 restart_effect_runtime、CLI doctor 消费者、测试和 registry manifest。新 receipt 没有复制第二套写入权威。

独立 exact-head 验证：相关 Python 测试 131 passed，TypeScript 测试 79 passed（本机 Node 24），架构检查 9 passed；控制面 TypeScript typecheck、所改 Python 文件 Ruff、公共边界扫描（0 命中）和 diff check 均通过。另有一个针对真实生产函数、仅注入响应异常的负例复现了下述失败。未运行真实 PostgreSQL 服务集成测试；本轮按队列策略未查询远端 CI，因此这些不能算已通过。

对主干的风险

• [P1，阻塞] loopx/control_plane/effect_runtime.py:580 新增的 EffectRuntimeResponseAmbiguous 会从 restart_effect_runtime 的直接 _request_with_info("runtime.shutdown") 调用逸出。该函数在 506–517 行只捕获 EffectRuntimeRejected、EffectRuntimeRemoteError 和 OSError。因此 doctor --restart-runtime 在 shutdown 已发送、响应超时或 EOF 时，跳过原有有界 token/pid 观察，既不返回 stopped，也不返回 shutdown_pending；服务可能已经在关闭。exact-head 注入复现输出 UNCAUGHT runtime_response_ambiguous，而现有 131 个 Python 测试仍通过，说明这个消费端缺少负例覆盖。请在现有 restart 包装层接住新异常并继续原观察逻辑，补充响应超时/EOF 与 replacement locator 的回归，并验证 doctor 的结构化 JSON 回读。

其余边界：todo receipt 查不到记录不等于可以安全重试仍在途的操作；File-v0 的全历史容量问题不在本 PR 解决范围。PG 真服务与远端 CI 尚未验证，不能以模拟 provider 测试替代。

我的整体评价

REQUEST_CHANGES。主要可靠性修复与现有 provider/Monitor 权威边界一致，改动规模也有测试支撑；但新异常引入后遗漏了直接 restart 消费者，破坏了现有运维入口。这个小而相关的兼容性修复应在当前 PR 完成，不需要另起协议或扩展范围。完成修复和负例验证后，请对新的 exact head 重新审查。相邻的未来向整理已评估：复用现有 restart 观察分支即可，无需新增抽象。

English verdict: REQUEST_CHANGES - The transport fix is directionally sound, but an ambiguous shutdown response escapes the existing restart wrapper and suppresses its structured status; handle the new exception and cover timeout/EOF before merge.

[huangruiteng]

Follow-up runtime-path audit for exact head f8489b184906a5185c6cec7b190d4ad0e4e0a13e (not a review verdict):

• This change gives coordination.local_authority.monitor_poll a 45-second RPC budget and treats a post-send missing response as ambiguous rather than a reason to replace the server or blindly retry. That is the relevant path for a newly observed 5-second monitor-poll timeout.
• coordination.local_authority.todo_update in provider_update.py and quota.spend.commit in spend_commit.py still use the generic 5-second budget. Therefore this PR must not be described as resolving those separate timeout reports. Please keep them as a bounded follow-up with operation-receipt readback, measured lock/execution time, same-effect replay, and no-double-spend coverage.
• A client's appended=false after a transport timeout is not durable proof of non-commit. The exact operation/effect receipt should be read before any retry.

Local validation at this head: 71 focused Python tests and 36 focused TypeScript tests passed; control-plane typecheck and standard premerge canary passed. This observation does not grant merge authority or change the repository's maintainer-review requirement.

[huangruiteng]

Latency refinement on exact head 0b9190f: File-v0 now reuses one process-wide, fully verified journal across Effect requests only when the raw SHA-256 digest, path and store identity match. The cache is bounded to one document with a 128 MiB raw-file admission limit; changed bytes or identity force full validation, and ambiguous writes clear it. On the active ~75 MB File-v0 journal, read-only new-handle loadAuthority measured ~1.97-1.99 s before, then ~2.18 s cold / ~40-43 ms warm after. This is a warm-path fix, not a claim that File-v0 whole-history writes or cold starts are constant-time; qualified SQLite promotion remains the durable scaling path. Validation: 279/279 File authority tests, 8/8 deferred File/SQLite lifecycle tests, 11/11 Effect runtime request-scope tests, TS typecheck, and diff check. Main conflict was reconciled; PR is mergeable, with CI and maintainer review still pending. No Goal store, private logs or credentials were changed or included.

[huangruiteng]

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

评审 head：d2132d6db2d32262867c632a9b70cddfd44796c9。这是对该 head 的完整复审；先前针对 f8489b1 的阻塞结论不自动沿用。

动机

Canonical File authority 的历史校验或 Goal 写入锁等待可能超过旧的 5 秒 RPC 预算。请求发送后丢失响应并不等于操作未提交；删除运行时 locator 后重试会有重复执行和双服务风险。Monitor 的 provider_pending 又会把其他正常追加的 run-index 记录误判为冲突。本 PR 修复这些相邻的可靠性边界，并提供所选 provider 的精确历史 receipt 只读回查。File-v0 全历史容量增长及其他 5 秒通用调用并不在本次完成范围。

改动思路

Python Effect 桥接把发送后的 EOF、超时和不可验证响应归为 EffectRuntimeResponseAmbiguous，保留 locator，禁止盲目重放；真正发送前失败才重新读取服务状态。TypeScript 服务端在文件锁内发布和按 token/PID/fingerprint 清理 locator。File provider 对同一路径、store identity 与原始 SHA-256 摘要完全匹配的文档复用一次已验证结果，缓存最多一份且只接纳不超过 128 MiB 的文档；字节或身份变化仍需完整验证，模糊写入清空缓存。Receipt CLI 继续委托现有 provider readReceipt，不新造写入权威，也不授予 lease 或重试权限。Monitor 仅在冻结的旧索引前缀未变、effect 不重复且 provider receipt 匹配时继续结算，并在锁内以当前索引建立新的 WAL fence。

具体改动

检查了 effect_runtime_request / _request_with_info、直接调用它的 restart_effect_runtime、todo receipt --operation-id 的参数/事件/处理路径、local_authority_read、File provider、服务端 locator 生命周期、Monitor WAL、terminal/Monitor 的 45 秒预算与 canonical writer 的 30 秒等待、registry I/O manifest 以及相关 Python/TypeScript 回归。上轮 [P1] 所指的 doctor --restart-runtime 消费端现已在原有异常分支捕获 EffectRuntimeResponseAmbiguous，然后继续有界观察 token/PID：原 token 仍服务时返回 shutdown_pending，替代 token 出现时返回 stopped，不额外发送 shutdown。

独立验证：相关 Python 测试 56 passed，包括真实本地 socket 的 EOF/超时和 doctor JSON 回读；File-provider/receipt/Monitor/writer-boundary TypeScript 套件 315 passed（子进程显式使用仓库测试 Python，避免系统 Python 3.9 的无效环境失败）；control-plane typecheck、所改 Python 文件 Ruff、公共边界扫描及 diff check 通过。Standard canary 的 18 项行为检查通过；期间 main 前移使首次质量回执失效，按新 base 重审同一 27 文件 diff 后，质量回执 cqr_2dbc98089483b688c796 校验有效，当前 base 的 quick premerge 门禁通过。此轮未在本地运行真实 PostgreSQL 集成；远端 CI 不作为本轮技术评审的通过依据。

对主干的风险

此前 shutdown 已发送而响应丢失导致 doctor 无结构化结果的阻塞风险已修复并以 EOF、超时、替代/仍服务 token 负例覆盖。File 缓存按原始字节摘要和 store identity 命中，返回给外部的 head、receipt 和 scan 结果保持克隆边界；它改善 warm read，不解决冷启动或整个 File-v0 历史写入的容量复杂度。Monitor 对截断、改写旧前缀及重复 effect 仍拒绝。真实 PostgreSQL 服务器未在本地验证，且作者自己的 COMMENTED 技术评审不能替代独立 maintainer 对运行时/控制面 PR 的合并授权。未来向整理已检查：继续复用现有 restart 观察、provider receipt 和 Monitor WAL 比新增第二套状态更合适，本 PR 无需扩大重构。

我的整体评价

APPROVE（技术结论，非合并授权）。当前 head 对原阻塞项完成了最小兼容修复，负例和读回足以证明相关路径，其他改动也保持在既有权威边界内。请由独立 maintainer 核验当前 head、仓库所需检查及合并策略后决定是否合并；本评审不使用 admin bypass 自合并该运行时变更。

English verdict: APPROVE - At d2132d6, the ambiguous shutdown response now preserves structured restart observation without replay; focused Python/TypeScript tests and current-base quality gate pass. Separate maintainer merge authority remains required.