🎃 Send an alert webhook when a connector keeps failing #1006

Part of the October Challenge #997.

Why

Today a broken connector (expired credentials, changed upstream API) is only noticed when someone opens the Audit Log or a user complains that Claude "can't reach the ERP". Admins want a push: "connector SAP failed 5 times in 10 minutes", in Slack or in their own on-call tooling. Zapier, n8n and most API gateways ship failure notifications; we have nothing yet. This issue is the first, reviewable step: one webhook per organization, one alert rule, signed payloads.

Self-hosted vs Cloud

• Self-hosted: enabled in Community and Business. The webhook URL goes through the existing SSRF guard with the instance's normal policy, so SSRF_ALLOWED_HOSTS / the admin allowlist can permit an internal Mattermost or Alertmanager.
• Cloud (DEPLOYMENT_MODE=cloud): enabled, but stricter, because requests leave our infrastructure on behalf of a tenant: https: only, private/loopback/link-local (incl. 169.254.169.254) always blocked, and the instance allowlist is ignored (on Cloud it contains internal hosts such as motis, see docker-compose.cloud.yml:138). No redirects followed.
• Who can use it: org ADMIN only (configure, test, delete). Settings are per organization; an alert only ever contains data from the organization the failing invocation belongs to.

What to build

1. Settings storage in OrgSettings (no Prisma migration needed), key alert_webhook: { url, format: 'json' | 'slack', threshold: number (default 5), windowMinutes: number (default 10), cooldownMinutes: number (default 60), enabled: boolean, secret }. Encrypt secret with encrypt(secret, ENCRYPTION_KEY, aad = organizationId).
2. Admin API next to the SMTP endpoints, ADMIN only: GET/PUT/DELETE /api/admin/settings/alert-webhook, POST /api/admin/settings/alert-webhook/test. The secret is generated server-side (32 random bytes, hex), returned once on create/rotate (PUT ... { rotateSecret: true }), never by GET. Validate the URL with the SSRF guard on save, not only on send.
3. Detection: in AuditService.logInvocation, after the row is written and when status !== 'SUCCESS' and organizationId and connectorId are set, call a new AlertsService.recordFailure(...) without awaiting it (fire-and-forget, errors logged, never thrown — the MCP tool call must not get slower or fail because of alerting). Count failures per (org, connector) in a sliding window: Redis incr+expire when connected, an in-memory map otherwise (Redis is optional, see redis.service.ts:18).
4. Dedupe/throttle: after an alert fires for (org, connector), suppress further alerts for cooldownMinutes.
5. Dispatch: POST with Content-Type: application/json, 5 s timeout, maxRedirects: 0, one retry. Headers: X-AnythingMCP-Event: connector.failing, X-AnythingMCP-Timestamp: <unix>, X-AnythingMCP-Signature: sha256=<hex HMAC-SHA256(secret, "<timestamp>.<raw body>")>.
   • json body: { event, organizationId, connector: { id, name }, failures, windowMinutes, lastError: { tool, message }, firstFailureAt, lastFailureAt, logsUrl }.
   • slack body: { text } (works with Slack/Mattermost/Discord-compatible incoming webhooks).
   • lastError.message: truncate to 300 chars. It can contain upstream response bodies; do not include tool input or output.
6. UI: an "Alerts" card on /settings/admin (admin-only page) with URL, format, threshold/window, enable toggle, "Send test", and a copy-once secret display.

Where to look

• packages/backend/src/audit/audit.service.ts:21 — logInvocation, the single place every tool call is recorded; :44 status union (TIMEOUT exists in the enum too).
• packages/backend/src/mcp-server/dynamic-mcp-tools.ts:486 — the ERROR call site (shows which fields are available: organizationId, connectorId, error).
• packages/backend/prisma/schema.prisma:78 (OrgSettings), :678 (ToolInvocation).
• packages/backend/src/settings/org-settings.service.ts:6 and packages/backend/src/settings/site-settings.controller.ts:113–180 — the SMTP settings endpoints: the pattern to copy (ADMIN guard, secret never returned, test endpoint).
• packages/backend/src/common/crypto/encryption.util.ts:17 — encrypt/decrypt with AAD.
• packages/backend/src/common/ssrf.util.ts:147 (assertSafeOutboundUrl(url, env)), :43 (readPolicy — env-driven, so on Cloud pass an env with SSRF_ALLOWED_HOSTS='', SSRF_ALLOW_PRIVATE/SSRF_ALLOW_LOCALHOST unset), :369 (createSsrfGuardedAgents(env) — connect-time check against DNS rebinding). Note: the DB allowlist provider (:76) is global; bypass it explicitly for webhooks on Cloud.
• packages/backend/src/common/guarded-http.util.ts:39 — how axios is wired with guarded agents.
• packages/backend/src/common/redis.service.ts:86 — incr/expire.
• packages/backend/src/common/deployment.service.ts:6 — isCloud().
• packages/frontend/src/app/settings/admin/page.tsx and packages/frontend/src/lib/api.ts:986 (adminSettings) — UI and client to extend.

Acceptance criteria

• Unit specs next to the new service (*.spec.ts, run with npm test --workspace=packages/backend): threshold reached → one dispatch; cooldown suppresses the second; success calls never count; other orgs' failures never count; dispatch errors never reject logInvocation.
• Signature spec: a receiver can verify sha256= with the documented algorithm; secret never appears in GET responses or logs.
• SSRF spec (with SSRF_GUARD=enabled, see ssrf.util.spec.ts): on Cloud http://, http://169.254.169.254, https://10.0.0.1 and an allowlisted host (motis) are rejected at save and at send.
• Non-admin gets 403 on all endpoints (roles.guard.ts:11).
• Docs: new section in docs/operations/observability.md (payload, headers, verification snippet in Node).

Out of scope

• Other events (license expiry, OAuth token refresh failure, KG jobs) — follow-up issues, reuse the same dispatcher.
• Multiple webhooks per org, per-connector rules, email alerts, PagerDuty/Opsgenie native formats.
• A delivery log / retry queue UI.
• Anything under ee/ (do not add code there).

Size

L (several days, may be split: backend detection+dispatch first, UI second).

How to claim

Comment "I'd like to work on this" and we'll assign you. Rules in #997.