Steer fallback (cancel+merge) requeues the same event indefinitely: 620 bootstrap turns in 37 minutes, each re-sending the full base prompt

[dziduchm]

Summary

With BUZZ_ACP_SUBSCRIBE=all, a single inbound event that arrives while a turn is in flight (or repeatedly fails mid-turn) can be re-dispatched indefinitely. Each redispatch creates a fresh ACP session and sends the full ~24K-character base prompt to the model provider. In our hive this loop ran at exactly ~20 turns/minute for 37 minutes (620 sessions), burning roughly 4-5M tokens of input against a shared provider plan until the plan's rate limit tripped and the agent went mute.

The loop survives because cancel+merge batches are intentionally preserved and never dead-lettered, so neither the per-scope retry budget nor the dead-letter paths ever bound redelivery of a steer-released event.

Environment

• buzz-acp from desktop 0.5.23 (also reproduced logically against current origin/main source)
• Agent config: BUZZ_ACP_SUBSCRIBE=all (agent answers a channel without mention)
• Model backend: MiniMax-M3 via OpenAI-compatible endpoint
• 10-slot lazy Hermes pool (BUZZ_ACP_LAZY_POOL=true, children spawn per turn)

Timeline (from relay DB + profile state.db + agent log)

1. 15:00:30 UTC: relay workflow posts a @scout prompt (kind:9) to a channel our agent subscribes to in all mode.
2. 15:00-15:08: normal turns; the agent replies. Its own kind:9 reply is, of course, also an event in a subscribed channel and comes back as an inbound event.
3. 15:00:38 onwards: the agent log fills with pairs of:
   • steer: no active_run_id ... falling back to cancel+merge (release_native_steer puts the event back at the queue front)
   • Turn ended: reason=interrupted_during_api_call (the provider had begun rate-limiting)
4. Each cycle: redispatch -> fresh session -> full [Base] prompt sent -> 429 -> interrupted -> event released to queue front again.
5. Cadence locks to ~3.0 seconds per cycle (one turn attempt), i.e. 20/min, sustained 37 minutes.
6. 15:37:25 UTC: last redispatch. The provider plan is exhausted; hermes gives up after 3 retries ("API call failed after 3 retries. HTTP 429: Token Plan usage limit reached"), pool tears down to lazy. Agent stays mute until the plan resets.

Key evidence

• 620 sessions in the profile DB in the window whose first user message is the bare bootstrap prompt (<base> + event text), vs 9 real work sessions.
• All 620 prompts reference only 7 unique event ids (from the ["e","<id>"] tags embedded in the prompt). Two events account for 157 of the redeliveries:
  • the agent's OWN kind:9 reply (92 redispatches),
  • the relay workflow prompt (65 redispatches).
• Zero dead-letter log lines in or after the window. Zero reconnect storms in the window (reconnect storms on other days were checked and ruled out).
• Relay ingested only ~54 events in the window: the loop is client-side (buzz-acp re-queuing), not relay re-delivery.

Root cause chain

1. Inbound event during an in-flight turn is delivered as a steer attempt (try_native_steer, crates/buzz-acp/src/lib.rs, ~line 4288 in current main).
2. If there is no active run (or the run is being torn down), the fallback path does cancel+merge and release_native_steer (crates/buzz-acp/src/queue.rs ~line 928) puts the event back at the queue FRONT for normal dispatch.
3. Redispatch starts a brand-new session: the full base prompt is paid again even though the turn will fail.
4. The turn fails mid-API-call (rate limit). An interrupted_during_api_call cancel+merge batch is preserved, "never dead-lettered like a real failure" (see the cancel+merge preservation test around lib.rs ~line 10579). So the event is released to the queue front again.
5. Goto 1. Nothing increments any counter: MAX_RETRIES/retry_counts dead-lettering in queue.rs (~line 538) and the hard-cap/idle-timeout dead-letter paths in lib.rs (~lines 4613-4749) do not cover this path.

Two design details amplify it:

• Self-echo: in all subscribe mode the agent's own published kind:9 replies return as inbound events, so an agent can steer itself with its own output.
• Fresh session per redispatch: the cost of one loop iteration is a full base-prompt input, not a cheap retry.

Impact

• Token burn: ~620 x (base prompt ~6-8K tokens) in under 40 minutes from a handful of events.
• Blast radius beyond the looping agent: the whole hive shares one provider plan, so the loop exhausted the plan and other agents started receiving 429s in the same hour.
• Availability: the looping agent ends up mute until the plan resets; the queued event is still sitting at the queue front and can re-seed the loop on the next wake.

Proposed fixes (any one would have stopped this)

1. Redelivery bound on cancel+merge requeues: track a per-event redispatch counter in the queue; after N failed redispatches of the same event (e.g. 3-5), dead-letter with a log line. This keeps the "never lose a cancel+merge batch" guarantee for the normal case but bounds pathological loops.
2. Self-echo filter: ignore inbound events whose author pubkey equals the agent's own pubkey (optionally scoped to BUZZ_ACP_SUBSCRIBE=all mode). Removes the most common loop seed; does not cover the relay-workflow-echo case, so fix 1 is still needed.
3. Backoff on rate-limit-class failures: if the last turn failed with a provider rate-limit error, delay the redispatch of the released event (exponential backoff) instead of dispatching immediately. This turns a 20/min storm into a handful of spaced retries.

Fix 1 is the most self-contained seam (queue.rs already has retry-count machinery for batches; the cancel+merge path just bypasses it). Happy to attempt a PR if the maintainers agree on the desired semantics.

Reproduction sketch

1. Run an agent with BUZZ_ACP_SUBSCRIBE=all against any provider.
2. Have the provider start returning 429s (or squeeze the plan).
3. Post any message to a channel the agent subscribes to while a turn is in flight.
4. Observe the steer fallback + redispatch loop at one iteration per turn cycle until the queue drains or the provider dies.