buzz-acp: failure notices wake sibling agents, so several broken agents in one channel loop on each other indefinitely

[jiwonschol]

Summary

When two or more agents with the same owner are broken at the same time in a shared channel, the ⚠️ I couldn't process the last request… failure notice one agent posts wakes the others, and they keep waking each other with no human involved. On my self-hosted setup this ran for 4.5 days.

What happened

Three codex agents (codex-acp, --respond-to owner-only, same owner) sit in one channel. Two lost their ChatGPT login (refresh token was revoked), and the third hit its usage limit. After that:

1. Agent A retries one event 10 times (~20 min), dead-letters it, and posts the ⚠️ notice to the channel.
2. That notice is an ordinary kind:9 message from a sibling, so it passes is_owner_or_sibling on agents B and C. They each try to handle it, fail 10 times, and post their own ⚠️.
3. Agent A then picks up B's notice, and the cycle repeats.

A new batch started roughly every 25–45 minutes, including 3–5 AM. Over 4.5 days I saw:

• ~3,100 failed turns (1,474 + 1,506 + 132)
• ~280 ⚠️ notices in the channel
• ~7,300 kind:7 👀/💬 reactions and matching kind:5 deletions from retry attempts

No tokens were billed, because every turn failed before reaching the model. The main costs are channel noise, a much heavier channel history, and my instruction to those agents to stand down having no effect.

Why it loops (buzz-acp, origin/main @ ef2aa1a)

• post_failure_notice (pool.rs) builds the notice with build_message and no tag that marks it as a system notice.
• RespondTo::OwnerOnly resolves through is_owner_or_sibling (lib.rs), which accepts any sibling identity by design. A sibling's failure notice is therefore treated like a normal instruction.
• The fast path that should stop the retries doesn't fire for codex. is_auth_error only matches Re-authenticate / API Error: 401 (Claude's wording). The codex revoked-token failure arrives as -32603 Internal error (see [Bug] buzz-acp drops JSON-RPC error.data, hiding actionable failures — CLI/adapter version mismatch surfaces only as "-32603 Internal error" retry loops #2422), so it goes through all 10 retries instead of being dead-lettered immediately.

Suggested fix

• Tag failure notices (for example ["buzz", "failure-notice"]) and have the inbound gate skip them, at least when they come from siblings. A failure notice is meant for the human and should never trigger another agent's turn.
• Separately, extend is_auth_error to cover codex's revoked or expired token errors once [Bug] buzz-acp drops JSON-RPC error.data, hiding actionable failures — CLI/adapter version mismatch surfaces only as "-32603 Internal error" retry loops #2422 lets the error detail through.

Related

#2422 (error detail dropped → -32603 retry loops), #3831 / #5196 (auth failures not surfaced), #5918 / #7317 (usage-limit dead-lettering), #5450 (sibling gate).

Local workaround

A small watchdog follows journalctl -u 'buzz-agent@*' and runs systemctl disable --now on any unit that logs dead-lettering batch. That breaks the cycle after one notice per broken agent.

Environment: buzz-acp 0.5.23 (Linux, systemd), codex-acp, self-hosted relay.