Flag turns that go silent as stalled instead of spinning forever

[oldsj]

Stacked on #272 (restart). It targets that branch, so this diff shows only the stall watch. GitHub moves the base to main when #272 merges.

On 2026-10-18 the main thread showed "Searching the web…" for over two hours, and four tasks were stuck the same way. Each turn's task had gone silent inside kagent, and nothing in Mainloop flagged it. This PR marks such turns as stalled, shows them with stop and restart, and raises attention and orchestration-health signals. The next task event clears the mark. Nothing is ever replayed.

What changes

Tracking (migration 00270, adds deliveries.activity_at, activity_kind and stalled_at):

• The delivery loop records when it last saw a live event of a delivered turn's task, and what that event left the turn doing:
  • event: thinking or writing
  • tool: a tool call started with no result yet
  • parked: waiting for input or auth
• Live events are the send stream, plus every event of a subscription after the first one. The first event of a subscription is the task as it stands, so re-subscribing to a silent task does not reset its clock. A snapshot counts only when its status timestamp is newer than the stored activity.
• Writes are throttled and fenced on the claim. They never touch updated_at, state or outcome_version.

Marking stalled:

• The mark-stalled-turns sweep runs every 30 s. It marks a turn quiet for longer than SESSION_STALL_SECONDS (1500), or SESSION_STALL_TOOL_SECONDS (3600) while a tool runs.
• Parked turns never stall.
• The sweep publishes session:message so an open page refreshes.
• The next live event clears the mark in the same write, and publishes again.

Where it shows (contract first):

• NativeSessionInfo.stalled_turn (a TurnStall, or null) on /main-thread and /sessions/{id}/native.
• HealthView.stalled_turns: oldest first, narrowed by the task, session and attempt filters. The time window does not apply.
• An attention item (turn_stalled, reason stalled, links to the main thread or the session) for the main thread and standalone sessions. It is opened and resolved by the existing attention sweep. Task-owned sessions are skipped, because task liveness already blocks those tasks as stalled and opens a card.

Web:

• The turn recovery panel now also shows a stalled turn: "No activity for 25m (while a tool runs). It may still finish. Stop or restart?". It has stop, plus restart session when restart applies.
• The Health page gains a "Stalled turns" section. Entries it cannot read are counted and named, not hidden.
• Inbox rows for these items link to the thread.

Specs updated: docs/specs/orchestration-health.md (new "Stalled turns" section), docs/specs/chat.md, docs/specs/sessions.md.

Verification

Checks run locally (macOS, using a disposable PostgreSQL 16 and a build overlay for the Linux-only pushgate):

• go test -race ./... passes for every package except pushgate, workspaces git-push tests (both need Linux prlimit) and one devloop unicode filename test (macOS normalizes names). These are environment-only failures in packages this PR does not touch; CI runs them on Linux.
• Fake-clock tests in internal/sessions/stall_test.go:
  • a turn stalls past its allowance, and the thread, attention and the health view all show it;
  • the next event clears it, attention resolves, and nothing is resent;
  • a running tool does not stall within its longer allowance, and stalls past it;
  • a parked turn never stalls.
• golangci-lint (v2.14.0, GOOS=linux): 0 issues.
• go generate ./...: no diff.
• redocly: valid (same 6 warnings as before).
• make lint: no issues.
• web: tsc -b and eslint . are clean. vitest passes for features/chat, features/sessions, features/health and features/inbox. vite build passes.

Not run: anything against a live kagent. This is fixture and fake proof only.

For the owner to check

• Thresholds are 25 min, or 60 min while a tool is running. For the incident (two WebSearch calls started, no result), the main thread would have read as stalled about an hour after 16:13.
• A stall is an observation: no delivery state changes and there is no automatic stop or restart.
• The live bubble's spinner is unchanged. The panel under it carries the stall message and the actions.

Closes #265. Part of #262 (the turn-silence watchdog asked for in its comments).

🤖 Generated with Claude Code