From eb463e3ffd5c59706bccd90dd9d5dbae7016e109 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 00:40:12 +0530 Subject: [PATCH 01/42] docs(aiops): design for sub-agent orchestration in the chat agent Design doc for reducing ~10min AI Ops chat runs via two complementary layers: intra-turn parallel tool calls (Layer A) and a read-only dispatch_agent sub-agent tool with context isolation (Layer B). Documents three prerequisites found during exploration: - createSessionProfile read-modify-write race (silent profile loss) - STS DurationSeconds=900 vs 10-minute runs - CloudFront originReadTimeout=60 requires progress heartbeats Rejects the LangGraph Send map-reduce supervisor: it would require a state-reducer rewrite to enable parallel mutations, which no current workload needs. Co-Authored-By: Claude Opus 5 (1M context) --- ...-26-aiops-subagent-orchestration-design.md | 405 ++++++++++++++++++ 1 file changed, 405 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md diff --git a/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md b/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md new file mode 100644 index 00000000..9cc47f0e --- /dev/null +++ b/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md @@ -0,0 +1,405 @@ +# AIOps Sub-Agent Orchestration — Design + +**Date:** 2026-07-26 +**Status:** Approved for planning +**Surface:** Interactive chat (`planning-agent.ts`) first; Agent Ops reuses the runtime later. + +## Problem + +AI Ops chat runs take ~10 minutes. `planning-agent.ts` is a strictly serial state machine: + +``` +memory_recall → planner → (generate → guard → tools → generate)* → reflect → revise* → final → memory_save +``` + +Every lap costs one streaming LLM call plus one tool execution, and the message history grows +monotonically, so late laps are markedly slower than early ones. With `MAX_ITERATIONS = 30` a +seven-step plan realistically burns 15–25 laps. + +Four workload shapes contribute, confirmed with the user: + +1. Wide fan-out (many accounts × regions × services) +2. Deep sequential investigation +3. Slow individual AWS calls +4. Long final-report generation + +No single lever fixes all four. This design ships two complementary layers and explicitly defers a +third. + +## Root causes identified in the current code + +### Serialization is prompt-enforced, not structural + +`planning-agent.ts:404` instructs the executor: + +> "Execute exactly the current step — do not skip ahead or bundle future steps into a single call." + +LangGraph's `ToolNode` already executes multiple `tool_calls` from one AI message concurrently. The +prompt suppresses that free parallelism, converting an N-way independent sweep into N serial laps +whose only purpose is emitting the next single command. `planning-agent.ts:292` compounds this by +telling the planner to merge related read-only queries into one *step*, while the executor prompt +then forbids merging them into one *turn*. + +### Context bloat + +Raw tool outputs accumulate in the orchestrator's message list. `prepareContext` compacts, but +compaction itself costs an extra LLM call and loses fidelity. + +### Concurrency is currently unsafe (blocker) + +`session-manager.ts:159-172` performs an unlocked read-modify-write on a shared per-tenant +credentials file: + +``` +Agent 1: read → {profileA} +Agent 2: read → {profileA} +Agent 1: write → {profileA, profileB} +Agent 2: write → {profileA, profileC} ← profileB silently lost +``` + +The agent holding the lost profile then fails with "The config profile could not be found". This is +latent today because nothing runs concurrently; it becomes a routine failure the moment +`get_aws_credentials` is parallelized (which every multi-account fan-out does). + +Additionally `assumeRoleForAccount` (`aws-credentials-tool.ts:44`) uses `DurationSeconds: 900`. +Credentials acquired at minute 0 expire at minute 15 — long runs already risk this, and fan-out +makes long runs more likely. + +### CloudFront caps silent periods at 60s + +`infra/compute/index.ts:962` sets `originReadTimeout: 60` while the ALB is at `idleTimeout: 1200` +(`infra/compute/index.ts:788`). A 10-minute run survives today only because streamed tokens +continuously reset the 60s timer. A fan-out phase where sub-agents think silently for 90s will drop +the connection. Progress heartbeats are therefore a correctness requirement, not a UX nicety. + +## Approaches considered + +### A. Intra-turn tool parallelism (no sub-agents) — **adopted** + +Remove the anti-batching prompt rules; let one AI turn emit N `tool_calls`; add a bounded semaphore +around `ToolNode`. + +- Cost is *negative* — fewer LLM laps means cheaper and faster. +- HIL survives untouched: `pendingToolCallsOf` (`guard.ts:16`), the guard's batched risk assessment + (`guard.ts:105`), and `approval-batch-card.tsx` already handle multi-call turns. +- Does not address context bloat, deep investigation, or report generation. + +### B. Sub-agent as a tool — the Claude Code `Task` pattern — **adopted** + +A `dispatch_agent` tool. The orchestrator emits several in one turn; `ToolNode` runs them +concurrently; each spawns a lean read-only ReAct loop with its own context, returning one compressed +findings report instead of dozens of raw tool outputs. + +- The primary win is **context isolation**, not only parallelism. Keeping the orchestrator's history + small is what also helps the deep-sequential and report-generation cases. +- Minimal graph surgery — it is a tool, not a topology change. +- Cost: 3–8× tokens on fan-out tasks, which the budget governor bounds. + +### C. LangGraph `Send` map-reduce supervisor — **rejected** + +Planner marks steps parallelizable; a fan-out node emits `Send()` to N executor branches; a reduce +node merges. + +- Only approach in which sub-agents could safely mutate, since each branch is independently + interruptible. +- But `ReflectionState`/`graphState` has no reducers for concurrent writes to `messages`, `plan`, or + `toolResults`; streaming attribution via `langgraph_node` breaks with N identical branches; and + `iterationCount` becomes meaningless. This is a rewrite of the state layer. +- Rejected because every real fan-out workload in the product (audits, inventories, right-sizing + sweeps, security reviews) is read-only. Parallel mutation is a problem we do not have. + +**Decision: ship A and B together.** A alone leaves context bloat; B alone still serializes inside +each sub-agent. + +## Architecture + +``` +Orchestrator (planning-agent — topology unchanged) + memory_recall → planner → generate ⇄ guard ⇄ tools → reflect ⇄ revise → final + │ + ├── execute_command (parallel — Layer A) + ├── execute_command (parallel — Layer A) + └── dispatch_agent ──────────────┐ (Layer B) + │ + ┌──────────────────────────────────────┘ + ▼ (ephemeral, non-checkpointed, read-only) + Sub-agent ReAct loop + generate → tools → generate (≤ 8 laps) + ▼ + compressed findings report (~1500 tokens) +``` + +### Why one sub-agent per tool call + +`dispatch_agent` takes a single sub-agent spec, not an array. The orchestrator emits N calls in one +turn and `ToolNode`'s existing parallelism *is* the fan-out. This avoids a second concurrency +mechanism and gives each sub-agent its own tool card in the UI for free. + +### Why a plain loop, not a compiled sub-graph + +Sub-agents are ephemeral and non-resumable. A checkpointer would add a Postgres write per lap for no +benefit. The sub-agent is a plain `while` loop over `model.invoke` + tool execution. + +## Components + +### 1. `session-manager` concurrency fix (prerequisite) + +**File:** `apps/web-ui/lib/agent/session-manager.ts` + +- Per-tenant async mutex serializing the read-modify-write in `createSessionProfile`. +- Atomic write: write to a temp file in the same directory, then `fs.rename`. +- Mutex map keyed by resolved credentials-file path, so the legacy shared-file path is covered too. +- Lazy credential refresh: `get_aws_credentials` re-assumes the role when the cached profile is + within 120 s of its `expiresAt` rather than handing back a profile that will expire mid-command. + Pre-existing weakness (`DurationSeconds: 900` vs. 10-minute runs), but fan-out makes long runs more + likely, so it lands here. + +### 2. Layer A — parallel tool calls + +**Files:** `apps/web-ui/lib/agent/planning-agent.ts`, `apps/web-ui/lib/agent/fast-agent.ts` + +- Replace the executor's "do not bundle" rule with explicit guidance to batch **independent + read-only** calls into one turn, while keeping dependent calls sequential and keeping mutations one + at a time. +- Reviser prompt gets the same treatment. +- Wrap `ToolNode` invocation in a bounded semaphore (`TOOL_CONCURRENCY`, default 6) so a 45-call turn + does not open 45 simultaneous AWS CLI subprocesses. + +The guard, approval routing, and the batch approval card need no changes. + +### 3. Layer B — `dispatch_agent` + +**New file:** `apps/web-ui/lib/agent/subagent.ts` + +Tool contract: + +```ts +dispatch_agent({ + role: string, // "EC2 inventory auditor for account 1234" + task: string, // full standalone brief — the sub-agent sees NO parent history + expectedOutput: string // "instance IDs with CPU <5% over 14d, with evidence" +}) → string // structured findings report, hard-capped (see SUBAGENT_REPORT_MAX_CHARS) +``` + +The report cap is enforced in characters, not tokens, so it is deterministic and testable: +`SUBAGENT_REPORT_MAX_CHARS = 6000` (≈1500 tokens). Over-length reports are truncated with an +explicit `[TRUNCATED — report exceeded cap]` marker, reusing `truncateOutput` from `agent-shared.ts`. + +The tool description explicitly requires `task` to be self-contained. Under-specified sub-agent +briefs are the dominant failure mode reported in Anthropic's multi-agent research: sub-agents +duplicate each other's work or return unusable fragments. + +Sub-agent internals: + +- Reuses the orchestrator's `createAgentModels(modelConfig)` instances — they are stateless per + invoke, so no new model construction per sub-agent. +- Own message list seeded only with its system prompt and `task`. +- Read-only tool subset (see §4). +- `SUBAGENT_MAX_ITERATIONS = 8`, `SUBAGENT_TIMEOUT_MS = 180_000`. +- Returns a structured report: findings, evidence (resource IDs, metric values), and an explicit + "could not determine" section, truncated per `SUBAGENT_REPORT_MAX_CHARS`. + +### 4. The read-only jail + +Two independent layers: + +1. **Filtered tool list.** Sub-agents receive read-only tools only. Excluded: `dispatch_agent` + (depth cap = 1), `ask_user` (no human reachable from inside a tool call), `write_file`, + `edit_file`, `write_file_to_s3`. +2. **Runtime interception.** Every sub-agent tool call is re-checked through `classifyTool` + (`tool-classifier.ts`). A mutative or unknown-named tool is refused with: *"You cannot mutate + state. Report the recommended change in your findings; the orchestrator will execute it under + human approval."* + +Layer 2 matters because MCP tools with unrecognized names hit `classifyTool`'s fail-closed default. +In the orchestrator that default means "ask the human"; inside a sub-agent, where no human is +reachable, it must mean "block". + +**Consequence: the existing HIL story is unchanged.** Mutations only ever occur on the single +guarded orchestrator path, so `guard.ts`, `gate-routing.ts`, and `interruptBefore: +["approval_gate"]` need no modification. + +### 5. Budget governor + +**New file:** `apps/web-ui/lib/agent/subagent-budget.ts` + +Run-scoped, in-memory, keyed by `threadId`. A run executes on a single ECS replica, so in-process +state is sufficient; no distributed coordination. + +| Control | Default | Env override | +|---|---|---| +| Max concurrent sub-agents | 3 | `SUBAGENT_MAX_CONCURRENCY` | +| Max sub-agents per run | 8 | `SUBAGENT_MAX_PER_RUN` | +| Max sub-agent tokens per run | 400 000 | `SUBAGENT_MAX_TOKENS_PER_RUN` | +| Max iterations per sub-agent | 8 | `SUBAGENT_MAX_ITERATIONS` | +| Sub-agent timeout | 180 s | `SUBAGENT_TIMEOUT_MS` | + +- Tokens metered from `usage_metadata` on each sub-agent model response. +- **On exhaustion `dispatch_agent` does not throw.** It returns *"Sub-agent budget exhausted; perform + this work yourself, serially."* The run degrades to current behaviour rather than failing. +- Bedrock `ThrottlingException` → exponential backoff retry inside the sub-agent loop. The low + concurrency cap is the primary defence. +- Budget entries are removed when the run ends, and are bounded by a max-entries LRU so an abandoned + run cannot leak memory. + +The whole of Layer B is gated by `SUBAGENTS_ENABLED` (default off at first deploy), matching the +existing `WORKING_MEMORY_ENABLED` / `EPISODIC_MEMORY_ENABLED` convention. Per-tenant configurability +is deliberately out of scope — bounded defaults were chosen over configurability. + +### 6. Streaming, UI, and heartbeat + +Sub-agent tokens are **not** streamed into the transcript. Three concurrent token streams interleaved +into one linear chat column are unreadable, and the sub-agent's narration was never the deliverable — +it is the same step-ceremony prose the executor prompt already suppresses. + +**New stream part** (`apps/web-ui/app/api/chat/stream-parts.ts`): + +```ts +{ + type: 'data-subagent', + id: `subagent-${subagentId}`, + data: { id, role, task, status: 'running' | 'done' | 'failed', + toolCount, tokensIn, tokensOut, summary?: string } +} +``` + +Plumbing: sub-agent model invocations are tagged via `.withConfig({ metadata: { subagent_id } })`. +`route.ts` already discriminates on `event.metadata?.langgraph_node`; it gains a branch routing any +event carrying `subagent_id` to a `data-subagent` part instead of the transcript. No side channel is +required. + +**Heartbeat:** a `data-subagent` tick is emitted every 15 s while any sub-agent is in flight. +Tool-completion events alone are insufficient — a single sub-agent thinking for 90 s would exceed +CloudFront's 60 s `originReadTimeout`. + +**Client:** `run-state.ts` gains a `case 'data-subagent'` and a `subagents: Map` +field. New `apps/web-ui/components/agent/chat/subagent-card.tsx` renders collapsed cards: + +``` + ⏳ Auditing account 1111 (Prod) 6 tools · 12.4k tokens + ✅ Auditing account 2222 (Staging) 4 tools · 8.1k tokens + ⏳ Auditing account 3333 (Dev) 9 tools · 15.2k tokens +``` + +### 7. Expandable cards and transcript persistence + +Cards expand to show that sub-agent's full transcript after it completes. + +Stream data parts are **not** persisted today — chat history is rebuilt from the LangGraph +checkpointer, and `build-chat-transcript.ts` handles only text and tool-invocation parts. Live +in-session expansion therefore works from client state, but expansion after a page reload requires +storage. + +The full transcript must **not** be placed in the `dispatch_agent` `ToolMessage` — that message +enters the orchestrator's context and would defeat the entire point of context isolation. + +**New Prisma model** (`libs/prisma/schema.prisma`), following the `AgentOpsEvent` TTL pattern: + +```prisma +model AgentSubagentRun { + id String @id @default(cuid()) + tenantId String + threadId String + subagentId String + role String + task String + status String // running|done|failed + toolCount Int @default(0) + tokensIn Int @default(0) + tokensOut Int @default(0) + summary String? + transcript Json? // full message list — fetched on demand, never sent to the orchestrator + createdAt DateTime @default(now()) + expiresAt DateTime // 30-day TTL, matching ChatMessage + + @@index([tenantId, threadId, createdAt]) + @@index([expiresAt]) + @@map("agent_subagent_runs") +} +``` + +Written through the repository pattern (`apps/web-ui/lib/db/repositories/agent/`) using +`getTenantClient(tenantId)`. Fetched on expand via `GET /api/chat/subagents/[threadId]`, guarded by +`authorize('read', 'AIOps')` and scoped to the caller's tenant. + +> **Migration note:** per `prisma-migrate-dev-drops-raw-sql-indexes`, audit the generated migration +> SQL for silent `DROP INDEX` statements against the existing HNSW/GIN indexes before applying. + +## Error handling + +Follows the existing never-crash-the-run convention throughout `planning-agent.ts`: + +| Failure | Behaviour | +|---|---| +| Sub-agent model/provider error | Returns an error report string; run continues | +| Sub-agent hits iteration cap | Returns partial findings marked incomplete | +| Sub-agent exceeds timeout | Aborted; returns partial findings marked incomplete | +| One sub-agent fails | Siblings unaffected — each is an independent tool call | +| All sub-agents fail | Orchestrator sees N error reports and may retry serially | +| Budget exhausted | Graceful degradation message, not an error | +| Transcript persistence fails | Logged and swallowed — never blocks the run | + +## Testing + +**Unit (Vitest, `apps/web-ui`):** + +- Budget governor: semaphore bounds concurrency; exhaustion returns the degradation message rather + than throwing; per-run counters reset. +- Read-only jail: mutative tool rejected; unknown-named tool rejected (fail-closed); read-only tool + permitted. +- Report truncation at the token cap, with marker. +- Sub-agent timeout returns partial findings. +- `stream-parts`: `data-subagent` part construction. +- `run-state`: `data-subagent` reducer builds and updates the map correctly. + +**Concurrency:** + +- `createSessionProfile` under `Promise.all` of 10 concurrent writes — all 10 profiles must survive. + *This test fails against the current implementation and is the regression test for the prerequisite + fix.* + +**Integration:** + +- Mocked model emitting three `dispatch_agent` calls in one turn; assert concurrent execution, + correct merge of the three reports, and that no raw sub-agent tool output entered the orchestrator's + message list. + +**Not covered:** E2E. A multi-minute agent run is too slow and flaky for the Playwright suite. + +## Measurement + +`llmAuditLog` (`agent-shared.ts:368`) already records per-node latency and token counts but is never +aggregated. A per-run summary line is added — total wall time split by node, LLM time vs tool time, +sub-agent count and token spend — logged at run end. + +A baseline is captured **before** Layer A ships, so the contribution of each layer is attributable +rather than assumed. + +## Sequencing + +| # | Work | Gate | +|---|---|---| +| 0 | `session-manager` mutex + atomic write + concurrency test | Blocker for 2 and 3 | +| 1 | Per-run timing summary from `llmAuditLog` | Baseline before changes | +| 2 | Layer A — parallel tool calls | Measure against baseline | +| 3 | Layer B — `dispatch_agent`, jail, budget governor | Behind `SUBAGENTS_ENABLED` | +| 4 | `data-subagent` streaming, heartbeat, collapsed cards | With 3 | +| 5 | `AgentSubagentRun` persistence + expand-after-reload | With 4 | + +## Explicitly out of scope + +- **Prompt caching.** Deferred to future work by user decision. Noted here because the executor + system prompt is large and byte-identical across every lap, and `createAgentModels` sets no cache + configuration — it is likely the single largest remaining win for the deep-sequential case, where + parallelism is structurally impossible. +- **Parallel mutations** (approach C). Requires a state-reducer rewrite; no current workload needs it. +- **Recursive sub-agents.** Depth capped at 1, matching Claude Code. Depth 2+ makes cost and latency + unpredictable and debugging impractical. +- **Agent debate / critic panels.** The reflector already fills the critique role. +- **Shared scratchpad files between sub-agents.** Reintroduces exactly the race condition being fixed + in step 0. +- **Per-tenant budget configuration UI.** Bounded defaults were chosen over configurability. +- **Agent Ops adoption.** The runtime is written to be surface-agnostic, but wiring it into + `executor-graphs.ts` is separate work. +- **Raising `DurationSeconds` beyond 900 s.** The refresh-on-near-expiry fix in step 0 covers the + failure mode without widening the credential lifetime. From 6df725a669758462ee48be0a3c4f4c273433dc28 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 00:45:15 +0530 Subject: [PATCH 02/42] docs(aiops): make sub-agent budget controls tenant-configurable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds §5a: budget controls move from hardcoded defaults to per-tenant config under the `aiops-subagents` TenantConfigService key, exposed via GET/PUT /api/settings/aiops and an aiops-settings.tsx panel. Env vars become platform ceilings rather than plain overrides — web-ui is a shared ECS task, so an unbounded tenant concurrency value would saturate the event loop and Bedrock quota for co-tenants. Resolution is clamp(tenantConfig ?? default, MIN, envCeiling), applied on read so lowering a ceiling retroactively binds existing rows. SUBAGENTS_ENABLED stays an absolute kill-switch. Co-Authored-By: Claude Opus 5 (1M context) --- ...-26-aiops-subagent-orchestration-design.md | 65 +++++++++++++++---- 1 file changed, 51 insertions(+), 14 deletions(-) diff --git a/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md b/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md index 9cc47f0e..3f818626 100644 --- a/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md +++ b/docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md @@ -226,14 +226,6 @@ guarded orchestrator path, so `guard.ts`, `gate-routing.ts`, and `interruptBefor Run-scoped, in-memory, keyed by `threadId`. A run executes on a single ECS replica, so in-process state is sufficient; no distributed coordination. -| Control | Default | Env override | -|---|---|---| -| Max concurrent sub-agents | 3 | `SUBAGENT_MAX_CONCURRENCY` | -| Max sub-agents per run | 8 | `SUBAGENT_MAX_PER_RUN` | -| Max sub-agent tokens per run | 400 000 | `SUBAGENT_MAX_TOKENS_PER_RUN` | -| Max iterations per sub-agent | 8 | `SUBAGENT_MAX_ITERATIONS` | -| Sub-agent timeout | 180 s | `SUBAGENT_TIMEOUT_MS` | - - Tokens metered from `usage_metadata` on each sub-agent model response. - **On exhaustion `dispatch_agent` does not throw.** It returns *"Sub-agent budget exhausted; perform this work yourself, serially."* The run degrades to current behaviour rather than failing. @@ -242,9 +234,52 @@ state is sufficient; no distributed coordination. - Budget entries are removed when the run ends, and are bounded by a max-entries LRU so an abandoned run cannot leak memory. -The whole of Layer B is gated by `SUBAGENTS_ENABLED` (default off at first deploy), matching the -existing `WORKING_MEMORY_ENABLED` / `EPISODIC_MEMORY_ENABLED` convention. Per-tenant configurability -is deliberately out of scope — bounded defaults were chosen over configurability. +### 5a. Per-tenant configuration + +The budget controls are tenant-configurable from AI Ops settings. **They are not purely +tenant-controlled**, because web-ui runs as a shared ECS task: a tenant setting concurrency to 50 +would saturate the container's event loop for every other tenant on that replica and flood their +Bedrock account with `ThrottlingException`. + +**Resolution order** — `effective = clamp(tenantConfig ?? default, MIN, envCeiling)`: + +| Control | Default | Min | Env ceiling | +|---|---|---|---| +| `enabled` | `false` | — | `SUBAGENTS_ENABLED` (absolute kill-switch) | +| `maxConcurrentSubagents` | 3 | 1 | `SUBAGENT_MAX_CONCURRENCY` (default 6) | +| `maxSubagentsPerRun` | 8 | 1 | `SUBAGENT_MAX_PER_RUN` (default 16) | +| `maxSubagentTokensPerRun` | 400 000 | 50 000 | `SUBAGENT_MAX_TOKENS_PER_RUN` (default 1 000 000) | +| `subagentMaxIterations` | 8 | 2 | `SUBAGENT_MAX_ITERATIONS` (default 16) | +| `subagentTimeoutMs` | 180 000 | 30 000 | `SUBAGENT_TIMEOUT_MS` (default 300 000) | + +`SUBAGENTS_ENABLED=false` is absolute: tenant config cannot re-enable the feature. The separate +`enabled` field exists so `maxSubagentsPerRun: 0` is never the accidental off-switch; the minimum of +1 makes that explicit. + +Clamping happens **server-side on read**, not only on write, so a config row written before a ceiling +was lowered cannot exceed the new ceiling. + +**Storage:** `TenantConfigService` under config key `aiops-subagents`, matching the existing +`discovery-cron` / `mcp-servers` convention. No new table. + +**API:** `GET`/`PUT /api/settings/aiops`, modelled directly on +`apps/web-ui/app/api/settings/discovery/route.ts`: + +- `authorize('read' | 'update', 'Agent')` — `Agent` already maps to the `AIOps` module + (`rbac/types.ts:33`), so no RBAC changes are needed. +- Zod 4 schema validating and clamping the payload; `GET` returns effective values plus the + ceilings, so the UI can show what a slider is bounded by and why. +- `AuditService.logUserAction` on save, matching the discovery route. + +**UI:** `apps/web-ui/components/settings/aiops-settings.tsx`, following `discovery-settings.tsx` and +`scheduler-settings.tsx` — React Hook Form + Zod resolver, a typed TanStack Query hook in +`lib/queries/settings.ts` keyed via `query-keys.ts`, sonner toast on save. When the platform +kill-switch is off, the panel renders read-only with an explanatory note rather than silently +accepting values that will not take effect. + +**Wiring:** resolved once during `createReflectionGraph` (alongside the existing `getSkillContent` +lookup) and passed into the sub-agent runtime via `GraphConfig` — not re-fetched per +`dispatch_agent` call, which would add a DB round-trip to every dispatch. ### 6. Streaming, UI, and heartbeat @@ -349,6 +384,8 @@ Follows the existing never-crash-the-run convention throughout `planning-agent.t permitted. - Report truncation at the token cap, with marker. - Sub-agent timeout returns partial findings. +- Budget resolution: tenant value below MIN clamps up; above the env ceiling clamps down; absent + tenant config falls back to defaults; `SUBAGENTS_ENABLED=false` overrides a tenant `enabled: true`. - `stream-parts`: `data-subagent` part construction. - `run-state`: `data-subagent` reducer builds and updates the map correctly. @@ -383,8 +420,9 @@ rather than assumed. | 1 | Per-run timing summary from `llmAuditLog` | Baseline before changes | | 2 | Layer A — parallel tool calls | Measure against baseline | | 3 | Layer B — `dispatch_agent`, jail, budget governor | Behind `SUBAGENTS_ENABLED` | -| 4 | `data-subagent` streaming, heartbeat, collapsed cards | With 3 | -| 5 | `AgentSubagentRun` persistence + expand-after-reload | With 4 | +| 4 | AI Ops settings: `/api/settings/aiops` + `aiops-settings.tsx` | With 3 | +| 5 | `data-subagent` streaming, heartbeat, collapsed cards | With 3 | +| 6 | `AgentSubagentRun` persistence + expand-after-reload | With 5 | ## Explicitly out of scope @@ -398,7 +436,6 @@ rather than assumed. - **Agent debate / critic panels.** The reflector already fills the critique role. - **Shared scratchpad files between sub-agents.** Reintroduces exactly the race condition being fixed in step 0. -- **Per-tenant budget configuration UI.** Bounded defaults were chosen over configurability. - **Agent Ops adoption.** The runtime is written to be surface-agnostic, but wiring it into `executor-graphs.ts` is separate work. - **Raising `DurationSeconds` beyond 900 s.** The refresh-on-near-expiry fix in step 0 covers the From b287c95711b944e595b2f426a8d469aca390428d Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 00:59:35 +0530 Subject: [PATCH 03/42] docs(aiops): implementation plan for sub-agent orchestration 11 TDD tasks covering the session-manager concurrency prerequisite, timing baseline, Layer A parallel tool calls, tenant-configurable budgets with platform ceilings, the read-only sub-agent runtime and dispatch_agent tool, data-subagent streaming with a CloudFront heartbeat, collapsed cards, and transcript persistence. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-07-26-aiops-subagent-orchestration.md | 3523 +++++++++++++++++ 1 file changed, 3523 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md diff --git a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md new file mode 100644 index 00000000..a7d0d8dd --- /dev/null +++ b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md @@ -0,0 +1,3523 @@ +# AIOps Sub-Agent Orchestration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Cut AI Ops chat run time from ~10 minutes by adding intra-turn parallel tool calls and a read-only `dispatch_agent` sub-agent tool with isolated context, bounded by tenant-configurable budgets. + +**Architecture:** The orchestrator graph (`planning-agent.ts`) keeps its exact topology. Layer A removes the prompt rules that forbid batching independent tool calls, so `ToolNode`'s existing concurrency is used. Layer B adds `dispatch_agent` as an ordinary tool — the orchestrator emits several in one turn and `ToolNode` fans them out. Each sub-agent is an ephemeral, non-checkpointed ReAct loop restricted to read-only tools that returns one compressed report, so raw tool output never enters the orchestrator's context. + +**Tech Stack:** TypeScript 5, Next.js 15.5.15 App Router, LangChain/LangGraph, Prisma 5 (web-ui), Vitest 4, TanStack Query 5, React Hook Form + Zod 4, sonner, Tailwind + Radix (shadcn/ui). + +**Spec:** `docs/superpowers/specs/2026-07-26-aiops-subagent-orchestration-design.md` + +## Global Constraints + +- **Working directory for all commands:** `apps/web-ui` unless stated otherwise. Tests run with `bun run test` (this is `vitest run` — single pass, not watch). +- **Path alias:** `@/` maps to `apps/web-ui/`. Use it for all cross-directory imports; relative imports only within the same directory. +- **Indentation:** 4 spaces in `lib/` and `app/api/` files; 2 spaces in `components/`. Match the file you are editing. +- **Multi-tenant safety:** every DB query is scoped via `getTenantClient(tenantId)`. Data access goes through the repository factory (`@/lib/db/repository-factory`), never Prisma directly from a route or service. +- **API responses:** `NextResponse.json({ success: true, data }, { status })` or `{ success: false, error: string }`. +- **RBAC:** every mutating route calls `authorize(action, Subject)` from `@/lib/rbac/authorize`. The `Agent` subject already maps to the `AIOps` module (`lib/rbac/types.ts:33`) — do not add a new subject. +- **Toasts:** import `toast` from `"sonner"` directly in new code. +- **Forms:** React Hook Form + `@hookform/resolvers/zod` + Zod 4. Do not build forms with manual `useState`. +- **Zod 4:** use `err.issues` (not `.errors`); `z.record()` requires the two-arg key+value form. +- **UI primitives:** consume `components/ui/*`; never modify them. +- **Never claim a step passes without running the command and reading its output.** + +--- + +### Task 1: Make AWS session-profile writes concurrency-safe + +Parallel tool calls and parallel sub-agents both call `get_aws_credentials` concurrently. `createSessionProfile` currently does an unlocked read-modify-write of one shared file, so concurrent callers silently lose each other's profiles and the agent then fails with "The config profile could not be found". This task is a hard prerequisite for Tasks 3 and 7. + +**Files:** +- Modify: `apps/web-ui/lib/agent/session-manager.ts` +- Modify: `apps/web-ui/lib/agent/aws-credentials-tool.ts:100-121` +- Test: `apps/web-ui/lib/agent/session-manager.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `createSessionProfile(accountId: string, credentials: AwsCredentials, tenantId?: string): Promise` — unchanged signature, now serialized. + - `getOrCreateSessionProfile(accountId: string, tenantId: string, assume: () => Promise): Promise` — returns a cached profile when it has more than `PROFILE_REFRESH_MARGIN_MS` of life left, otherwise assumes fresh credentials via the `assume` callback. + - `__resetProfileCacheForTests(): void` + +- [ ] **Step 1: Write the failing concurrency test** + +Create `apps/web-ui/lib/agent/session-manager.test.ts`: + +```typescript +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs/promises'; +import * as os from 'os'; +import * as path from 'path'; + +import { + createSessionProfile, + getOrCreateSessionProfile, + getTenantCredentialsFilePath, + __resetProfileCacheForTests, +} from './session-manager'; + +const TENANT = 'tenant-concurrency-test'; + +function creds(region = 'us-east-1') { + return { + accessKeyId: 'AKIAEXAMPLE', + secretAccessKey: 'secret', + sessionToken: 'token', + region, + }; +} + +beforeEach(() => { + __resetProfileCacheForTests(); +}); + +afterEach(async () => { + await fs.rm(path.dirname(getTenantCredentialsFilePath(TENANT)), { recursive: true, force: true }); +}); + +describe('createSessionProfile concurrency', () => { + it('keeps every profile when 10 callers write at the same time', async () => { + const accountIds = Array.from({ length: 10 }, (_, i) => `10000000000${i}`); + + const profiles = await Promise.all( + accountIds.map(id => createSessionProfile(id, creds(), TENANT)), + ); + + const contents = await fs.readFile(getTenantCredentialsFilePath(TENANT), 'utf-8'); + for (const profile of profiles) { + expect(contents).toContain(`[${profile.profileName}]`); + } + }); + + it('writes the credentials file atomically (no partial content observed)', async () => { + await createSessionProfile('222222222222', creds(), TENANT); + const filePath = getTenantCredentialsFilePath(TENANT); + const stat = await fs.stat(filePath); + expect(stat.mode & 0o777).toBe(0o600); + + const dir = path.dirname(filePath); + const leftovers = (await fs.readdir(dir)).filter(f => f.endsWith('.tmp')); + expect(leftovers).toEqual([]); + }); +}); + +describe('getOrCreateSessionProfile caching', () => { + it('reuses a fresh profile instead of assuming again', async () => { + const assume = vi.fn().mockResolvedValue(creds()); + + const first = await getOrCreateSessionProfile('333333333333', TENANT, assume); + const second = await getOrCreateSessionProfile('333333333333', TENANT, assume); + + expect(assume).toHaveBeenCalledTimes(1); + expect(second.profileName).toBe(first.profileName); + }); + + it('re-assumes when the cached profile is near expiry', async () => { + const assume = vi.fn().mockResolvedValue(creds()); + + const first = await getOrCreateSessionProfile('444444444444', TENANT, assume); + // Force the cached entry to be 30s from expiry — inside the 120s refresh margin. + first.expiresAt = new Date(Date.now() + 30_000); + + const second = await getOrCreateSessionProfile('444444444444', TENANT, assume); + + expect(assume).toHaveBeenCalledTimes(2); + expect(second.profileName).not.toBe(first.profileName); + }); + + it('scopes the cache by tenant', async () => { + const assume = vi.fn().mockResolvedValue(creds()); + + await getOrCreateSessionProfile('555555555555', TENANT, assume); + await getOrCreateSessionProfile('555555555555', `${TENANT}-other`, assume); + + expect(assume).toHaveBeenCalledTimes(2); + await fs.rm(path.dirname(getTenantCredentialsFilePath(`${TENANT}-other`)), { recursive: true, force: true }); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/session-manager.test.ts` + +Expected: FAIL. `getOrCreateSessionProfile` and `__resetProfileCacheForTests` are not exported, so the import fails to resolve. + +- [ ] **Step 3: Add the per-file lock and atomic write** + +In `apps/web-ui/lib/agent/session-manager.ts`, add `randomUUID` to the imports: + +```typescript +import { randomUUID } from 'crypto'; +``` + +Then add the lock helper immediately after the `PROFILE_PREFIX` constant (around line 53): + +```typescript +/** + * Serializes read-modify-write cycles per credentials file. + * + * Node is single-threaded, but `await` between the read and the write yields the + * event loop: two concurrent callers both read the old content and the second + * write clobbers the first caller's profile. The victim then runs + * `aws --profile ` against a profile that is not in the file and gets + * "The config profile could not be found". Chaining every mutation onto a + * per-path promise removes the interleaving window. + * + * The map is keyed by resolved file path, so the legacy shared-file path is + * covered too. Key count is bounded by tenant count, so it is not a leak. + */ +const fileLocks = new Map>(); + +function withFileLock(filePath: string, fn: () => Promise): Promise { + const previous = fileLocks.get(filePath) ?? Promise.resolve(); + // Run `fn` whether the predecessor resolved or rejected — one caller's + // failure must not deadlock every later caller on the same file. + const next = previous.then(fn, fn); + fileLocks.set(filePath, next.catch(() => undefined)); + return next; +} +``` + +Replace the body of `writeCredentialsFile` (line 90) with an atomic write: + +```typescript +async function writeCredentialsFile(filePath: string, content: string): Promise { + const dir = path.dirname(filePath); + await fs.mkdir(dir, { recursive: true }); + // Write-then-rename: a concurrent reader (the AWS CLI) never observes a + // half-written credentials file. The temp file is created in the same + // directory so the rename stays on one filesystem and is therefore atomic. + const tmpPath = path.join(dir, `.credentials.${process.pid}.${randomUUID()}.tmp`); + try { + await fs.writeFile(tmpPath, content, { mode: 0o600 }); + await fs.rename(tmpPath, filePath); + } catch (error) { + await fs.rm(tmpPath, { force: true }).catch(() => undefined); + throw error; + } +} +``` + +Wrap the read-modify-write inside `createSessionProfile`. Replace lines 159-172 (from `// Read current credentials file` through the `writeCredentialsFile` call) with: + +```typescript + await withFileLock(credentialsFile, async () => { + const content = await readCredentialsFile(credentialsFile); + const profiles = parseCredentialsFile(content); + + const profileCreds = new Map(); + profileCreds.set('aws_access_key_id', credentials.accessKeyId); + profileCreds.set('aws_secret_access_key', credentials.secretAccessKey); + profileCreds.set('aws_session_token', credentials.sessionToken); + profileCreds.set('region', credentials.region); + profiles.set(profileName, profileCreds); + + await writeCredentialsFile(credentialsFile, serializeCredentialsFile(profiles)); + }); +``` + +- [ ] **Step 4: Add the profile cache with refresh-on-near-expiry** + +Append to `apps/web-ui/lib/agent/session-manager.ts`, before the module-load cleanup call on line 233: + +```typescript +/** + * Re-assume when a cached profile has less than this much life left. STS + * credentials here last 900s (see assumeRoleForAccount); handing back a profile + * with 10s remaining guarantees the next AWS CLI call fails mid-flight. + */ +const PROFILE_REFRESH_MARGIN_MS = 120_000; + +const profileCache = new Map(); + +function profileCacheKey(tenantId: string, accountId: string): string { + return `${tenantId}:${accountId}`; +} + +/** + * Return a usable session profile for (tenant, account), assuming fresh + * credentials only when there is no cached profile or the cached one is within + * PROFILE_REFRESH_MARGIN_MS of expiry. + * + * Fan-out makes this matter twice over: N sub-agents auditing the same account + * previously triggered N AssumeRole calls, and a long run could hand out a + * profile that expired before it was used. + */ +export async function getOrCreateSessionProfile( + accountId: string, + tenantId: string, + assume: () => Promise, +): Promise { + const key = profileCacheKey(tenantId, accountId); + const cached = profileCache.get(key); + if (cached && cached.expiresAt.getTime() - Date.now() > PROFILE_REFRESH_MARGIN_MS) { + console.log(`[SessionManager] Reusing cached profile ${cached.profileName} for ${key}`); + return cached; + } + + const credentials = await assume(); + const profile = await createSessionProfile(accountId, credentials, tenantId); + profileCache.set(key, profile); + return profile; +} + +/** Test seam — clears the in-process profile cache between test cases. */ +export function __resetProfileCacheForTests(): void { + profileCache.clear(); + fileLocks.clear(); +} +``` + +- [ ] **Step 5: Run the tests to verify they pass** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/session-manager.test.ts` + +Expected: PASS, 5 tests. + +- [ ] **Step 6: Route `get_aws_credentials` through the cache** + +In `apps/web-ui/lib/agent/aws-credentials-tool.ts`, change the import on line 5: + +```typescript +import { getOrCreateSessionProfile } from './session-manager'; +``` + +Replace lines 100-121 (the assume-then-create block, from `// 2. Assume the role` through the two `console.log` lines that follow profile creation) with: + +```typescript + // 2/3/4. Resolve region, then get a cached-or-fresh session profile. + // getOrCreateSessionProfile only calls STS when there is no live + // profile left for this (tenant, account) pair. + const region = account.regions?.[0] || env.AWS_REGION || env.NEXT_PUBLIC_AWS_REGION || 'us-east-1'; + + const profile = await getOrCreateSessionProfile(accountId, tenantId, async () => { + const { credentials } = await assumeRoleForAccount( + account.roleArn!, + account.externalId, + ); + return { + accessKeyId: credentials.AccessKeyId!, + secretAccessKey: credentials.SecretAccessKey!, + sessionToken: credentials.SessionToken!, + region, + }; + }); + + console.log(`[Tool] Profile: ${profile.profileName} for account: ${accountId}`); + console.log(`[Tool] Profile expires at: ${profile.expiresAt.toISOString()}`); +``` + +The `region` const previously declared at line 107 is now declared above — delete the old declaration so there is no duplicate binding. + +- [ ] **Step 7: Typecheck and run the full agent test suite** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no errors in `lib/agent/session-manager.ts` or `lib/agent/aws-credentials-tool.ts`. + +Run: `cd apps/web-ui && bunx vitest run lib/agent/` +Expected: PASS. Note the pre-existing failing mock-harness tests recorded in project memory — compare against the count on `git stash`ed baseline if anything looks new. + +- [ ] **Step 8: Commit** + +```bash +git add apps/web-ui/lib/agent/session-manager.ts apps/web-ui/lib/agent/aws-credentials-tool.ts apps/web-ui/lib/agent/session-manager.test.ts +git commit -m "fix(agent): make AWS session-profile writes concurrency-safe + +Unlocked read-modify-write on the shared per-tenant credentials file let +concurrent get_aws_credentials calls silently drop each other's profiles, +surfacing as 'The config profile could not be found'. Serialize per file, +write atomically via rename, and cache profiles with refresh-on-near-expiry +so fan-out does not multiply AssumeRole calls." +``` + +--- + +### Task 2: Per-run timing summary + +Establishes the measurement baseline before any behaviour changes, so Layer A's and Layer B's contributions are attributable rather than assumed. `llmAuditLog` already computes per-call latency and token counts but discards them unless `LLM_AUDIT` is on. + +**Files:** +- Create: `apps/web-ui/lib/agent/run-timings.ts` +- Create: `apps/web-ui/lib/agent/run-timings.test.ts` +- Modify: `apps/web-ui/lib/agent/planning-agent.ts` (5 call sites) +- Modify: `apps/web-ui/app/api/chat/route.ts` (stream teardown) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + - `recordNodeTiming(threadId: string | undefined, node: string, latencyMs: number, tokensIn: number, tokensOut: number): void` + - `summarizeRun(threadId: string): RunTimingSummary | null` + - `logRunSummary(threadId: string): void` — logs and then discards the run's entry. + - `interface RunTimingSummary { totalLlmMs: number; totalTokensIn: number; totalTokensOut: number; byNode: Record }` + +- [ ] **Step 1: Write the failing test** + +Create `apps/web-ui/lib/agent/run-timings.test.ts`: + +```typescript +import { describe, it, expect, beforeEach } from 'vitest'; +import { recordNodeTiming, summarizeRun, logRunSummary, __resetRunTimingsForTests } from './run-timings'; + +beforeEach(() => __resetRunTimingsForTests()); + +describe('run timings', () => { + it('aggregates calls per node', () => { + recordNodeTiming('t1', 'EXECUTOR', 1000, 500, 100); + recordNodeTiming('t1', 'EXECUTOR', 2000, 600, 150); + recordNodeTiming('t1', 'REFLECTOR', 500, 200, 50); + + const summary = summarizeRun('t1')!; + + expect(summary.totalLlmMs).toBe(3500); + expect(summary.totalTokensIn).toBe(1300); + expect(summary.totalTokensOut).toBe(300); + expect(summary.byNode.EXECUTOR).toEqual({ calls: 2, ms: 3000, tokensIn: 1100, tokensOut: 250 }); + expect(summary.byNode.REFLECTOR.calls).toBe(1); + }); + + it('keeps runs isolated by threadId', () => { + recordNodeTiming('t1', 'EXECUTOR', 1000, 10, 10); + recordNodeTiming('t2', 'EXECUTOR', 5000, 20, 20); + + expect(summarizeRun('t1')!.totalLlmMs).toBe(1000); + expect(summarizeRun('t2')!.totalLlmMs).toBe(5000); + }); + + it('ignores calls with no threadId rather than throwing', () => { + expect(() => recordNodeTiming(undefined, 'EXECUTOR', 100, 1, 1)).not.toThrow(); + expect(summarizeRun('t1')).toBeNull(); + }); + + it('discards the run after logging its summary', () => { + recordNodeTiming('t1', 'EXECUTOR', 100, 1, 1); + logRunSummary('t1'); + expect(summarizeRun('t1')).toBeNull(); + }); + + it('returns null for an unknown thread', () => { + expect(summarizeRun('never-seen')).toBeNull(); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/run-timings.test.ts` +Expected: FAIL — cannot resolve `./run-timings`. + +- [ ] **Step 3: Implement the module** + +Create `apps/web-ui/lib/agent/run-timings.ts`: + +```typescript +/** + * Per-run LLM timing accumulator. + * + * llmAuditLog already measures latency and tokens for every model call, but + * throws the numbers away unless LLM_AUDIT is enabled. This module keeps a + * lightweight always-on aggregate so a run's wall time can be attributed to + * specific graph nodes — the baseline needed to tell whether parallelism + * actually helped. + * + * Keyed by threadId because runs are concurrent within one process. + */ + +export interface NodeTiming { + calls: number; + ms: number; + tokensIn: number; + tokensOut: number; +} + +export interface RunTimingSummary { + totalLlmMs: number; + totalTokensIn: number; + totalTokensOut: number; + byNode: Record; +} + +/** Backstop against unbounded growth if a run never reaches its teardown. */ +const MAX_TRACKED_RUNS = 200; + +const runs = new Map>(); + +export function recordNodeTiming( + threadId: string | undefined, + node: string, + latencyMs: number, + tokensIn: number, + tokensOut: number, +): void { + if (!threadId) return; + + let byNode = runs.get(threadId); + if (!byNode) { + if (runs.size >= MAX_TRACKED_RUNS) { + // Evict the oldest insertion — Map preserves insertion order. + const oldest = runs.keys().next().value; + if (oldest !== undefined) runs.delete(oldest); + } + byNode = {}; + runs.set(threadId, byNode); + } + + const entry = byNode[node] ?? { calls: 0, ms: 0, tokensIn: 0, tokensOut: 0 }; + entry.calls += 1; + entry.ms += latencyMs; + entry.tokensIn += tokensIn; + entry.tokensOut += tokensOut; + byNode[node] = entry; +} + +export function summarizeRun(threadId: string): RunTimingSummary | null { + const byNode = runs.get(threadId); + if (!byNode) return null; + + let totalLlmMs = 0; + let totalTokensIn = 0; + let totalTokensOut = 0; + for (const entry of Object.values(byNode)) { + totalLlmMs += entry.ms; + totalTokensIn += entry.tokensIn; + totalTokensOut += entry.tokensOut; + } + + return { totalLlmMs, totalTokensIn, totalTokensOut, byNode }; +} + +export function logRunSummary(threadId: string): void { + const summary = summarizeRun(threadId); + runs.delete(threadId); + if (!summary) return; + + const rows = Object.entries(summary.byNode) + .sort((a, b) => b[1].ms - a[1].ms) + .map(([node, t]) => + ` ${node.padEnd(12)} calls=${String(t.calls).padStart(3)} llm=${(t.ms / 1000).toFixed(1)}s in=${t.tokensIn} out=${t.tokensOut}`); + + console.log( + `\n📊 [RUN SUMMARY] thread=${threadId}\n` + + ` TOTAL llm=${(summary.totalLlmMs / 1000).toFixed(1)}s in=${summary.totalTokensIn} out=${summary.totalTokensOut}\n` + + rows.join('\n'), + ); +} + +/** Test seam. */ +export function __resetRunTimingsForTests(): void { + runs.clear(); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/run-timings.test.ts` +Expected: PASS, 5 tests. + +- [ ] **Step 5: Record timings from the planning-agent nodes** + +In `apps/web-ui/lib/agent/planning-agent.ts`, add to the imports from `./run-timings`: + +```typescript +import { recordNodeTiming } from "./run-timings"; +``` + +Three nodes do not currently receive the runtime config. Change their signatures so a threadId is available: + +- `planNode(state: ReflectionState)` → `planNode(state: ReflectionState, runtimeConfig?: any)` +- `reflectNode(state: ReflectionState)` → `reflectNode(state: ReflectionState, runtimeConfig?: any)` +- `finalNode(state: ReflectionState)` → `finalNode(state: ReflectionState, runtimeConfig?: any)` + +LangGraph already passes `(state, config)` to every node, so no `addNode` call changes. + +Then, immediately after each `llmAuditLog(...)` call, add the matching `recordNodeTiming` line. There are five call sites; each uses the `_auditStart_*` timestamp already in scope: + +```typescript +// after llmAuditLog('PLANNER', _auditInputs_plan, response, _auditStart_plan); +recordNodeTiming(runtimeConfig?.configurable?.thread_id, 'PLANNER', Date.now() - _auditStart_plan, + (response as any).usage_metadata?.input_tokens ?? 0, (response as any).usage_metadata?.output_tokens ?? 0); +``` + +Repeat with these node labels and variables: + +| Node function | Label | Start var | Response var | +|---|---|---|---| +| `planNode` | `'PLANNER'` | `_auditStart_plan` | `response` | +| `generateNode` | `'EXECUTOR'` | `_auditStart_exec` | `response` | +| `reflectNode` | `'REFLECTOR'` | `_auditStart_ref` | `response` | +| `reviseNode` | `'REVISER'` | `_auditStart_rev` | `response` | +| `finalNode` | `'FINAL'` | `_auditStart_fin` | `summaryResponse` | + +`generateNode` and `reviseNode` already receive `runtimeConfig`. + +- [ ] **Step 6: Log the summary when the stream ends** + +In `apps/web-ui/app/api/chat/route.ts`, add the import: + +```typescript +import { logRunSummary } from '@/lib/agent/run-timings'; +``` + +Find the `for await (const event of stream)` loop (line 826). Locate the enclosing `try`/`finally` (or the point where the stream completes and the controller is closed) and add to the teardown path: + +```typescript + // Emit the per-run timing breakdown regardless of how the run ended, + // so aborted and errored runs are measured too. + logRunSummary(threadId); +``` + +- [ ] **Step 7: Verify typecheck and tests** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +Run: `cd apps/web-ui && bunx vitest run lib/agent/` +Expected: PASS (same pre-existing failures as the Task 1 baseline, no new ones). + +- [ ] **Step 8: Capture the baseline manually** + +Start the app (`docker compose up -d postgres` at repo root, then `cd apps/web-ui && bun run dev`), run one representative slow AI Ops chat task, and copy the `📊 [RUN SUMMARY]` block from the server log into the commit message. This is the number every later task is measured against. + +- [ ] **Step 9: Commit** + +```bash +git add apps/web-ui/lib/agent/run-timings.ts apps/web-ui/lib/agent/run-timings.test.ts apps/web-ui/lib/agent/planning-agent.ts apps/web-ui/app/api/chat/route.ts +git commit -m "feat(agent): per-run LLM timing summary + +Aggregates the latency and token counts llmAuditLog already computes into a +per-node breakdown logged at run end, so the cost of each graph node is +attributable before parallelism changes land." +``` + +--- + +### Task 3: Layer A — allow batched parallel tool calls + +`ToolNode` already executes multiple `tool_calls` from one AI message concurrently. The executor prompt forbids using that, turning an N-way independent sweep into N serial laps. This task removes the prohibition and bounds the resulting shell-command concurrency. + +**Files:** +- Create: `apps/web-ui/lib/agent/concurrency.ts` +- Create: `apps/web-ui/lib/agent/concurrency.test.ts` +- Modify: `apps/web-ui/lib/agent/tools.ts:101-157` (`executeCommandTool`) +- Modify: `apps/web-ui/lib/agent/planning-agent.ts:402-407` (executor prompt), `:291-292` (planner prompt), `:750-759` (reviser prompt) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `class Semaphore { constructor(limit: number); run(fn: () => Promise): Promise }` + +- [ ] **Step 1: Write the failing semaphore test** + +Create `apps/web-ui/lib/agent/concurrency.test.ts`: + +```typescript +import { describe, it, expect } from 'vitest'; +import { Semaphore } from './concurrency'; + +const defer = () => { + let resolve!: () => void; + const promise = new Promise(r => { resolve = r; }); + return { promise, resolve }; +}; + +describe('Semaphore', () => { + it('never runs more than `limit` tasks at once', async () => { + const sem = new Semaphore(2); + let active = 0; + let peak = 0; + + const task = async () => sem.run(async () => { + active++; + peak = Math.max(peak, active); + await new Promise(r => setTimeout(r, 5)); + active--; + }); + + await Promise.all(Array.from({ length: 10 }, task)); + + expect(peak).toBe(2); + expect(active).toBe(0); + }); + + it('releases the slot when a task throws', async () => { + const sem = new Semaphore(1); + + await expect(sem.run(async () => { throw new Error('boom'); })).rejects.toThrow('boom'); + + // If the slot leaked, this would hang rather than resolve. + await expect(sem.run(async () => 'ok')).resolves.toBe('ok'); + }); + + it('queues callers beyond the limit and runs them as slots free', async () => { + const sem = new Semaphore(1); + const first = defer(); + const order: string[] = []; + + const a = sem.run(async () => { order.push('a-start'); await first.promise; order.push('a-end'); }); + const b = sem.run(async () => { order.push('b-start'); }); + + expect(order).toEqual(['a-start']); + first.resolve(); + await Promise.all([a, b]); + expect(order).toEqual(['a-start', 'a-end', 'b-start']); + }); + + it('treats a limit below 1 as 1', async () => { + const sem = new Semaphore(0); + await expect(sem.run(async () => 'ok')).resolves.toBe('ok'); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/concurrency.test.ts` +Expected: FAIL — cannot resolve `./concurrency`. + +- [ ] **Step 3: Implement the semaphore** + +Create `apps/web-ui/lib/agent/concurrency.ts`: + +```typescript +/** + * Minimal counting semaphore. + * + * Once the executor is allowed to emit many tool calls in one turn, ToolNode + * runs them all concurrently. That is the point — but execute_command spawns a + * real subprocess per call, so an unbounded 45-call turn would fork 45 shells + * inside the web-ui container. This bounds that specific blast radius. + */ +export class Semaphore { + private available: number; + private readonly waiters: Array<() => void> = []; + + constructor(limit: number) { + this.available = Math.max(1, Math.floor(limit)); + } + + async run(fn: () => Promise): Promise { + await this.acquire(); + try { + return await fn(); + } finally { + this.release(); + } + } + + private acquire(): Promise { + if (this.available > 0) { + this.available--; + return Promise.resolve(); + } + return new Promise(resolve => this.waiters.push(resolve)); + } + + private release(): void { + const next = this.waiters.shift(); + if (next) { + // Hand the slot straight to the next waiter — do not increment, or a + // third caller could take the slot we just promised away. + next(); + return; + } + this.available++; + } +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/concurrency.test.ts` +Expected: PASS, 4 tests. + +- [ ] **Step 5: Bound shell-command concurrency** + +In `apps/web-ui/lib/agent/tools.ts`, add the import near the top: + +```typescript +import { Semaphore } from './concurrency'; +``` + +Add below the `MAX_OUTPUT_LENGTH` constant (line 83): + +```typescript +/** + * Caps concurrent shell subprocesses across the whole process. Other tools + * (AWS SDK, Prisma, MCP) are network/DB calls that are already pooled by their + * own clients, so only execute_command needs an explicit bound. + */ +const TOOL_CONCURRENCY = Number(process.env.TOOL_CONCURRENCY) || 6; +const commandSemaphore = new Semaphore(TOOL_CONCURRENCY); +``` + +In `executeCommandTool`, wrap the `execAsync` call (line 131). Replace: + +```typescript + const { stdout, stderr } = await execAsync(command, { +``` + +with: + +```typescript + const { stdout, stderr } = await commandSemaphore.run(() => execAsync(command, { +``` + +and close the extra paren on the options object — the closing `});` on line 136 becomes `}));`. + +- [ ] **Step 6: Replace the anti-batching prompt rules** + +In `apps/web-ui/lib/agent/planning-agent.ts`, in `generateNode`'s `## Execution Discipline` block, replace this line (line 404): + +``` +- Execute exactly the current step — do not skip ahead or bundle future steps into a single call. +``` + +with: + +``` +- Execute the current step. When that step covers several INDEPENDENT read-only lookups (different accounts, regions, or services), issue them as MULTIPLE tool calls in this single turn — they run in parallel. One call per turn for independent reads wastes minutes. +- Calls that DEPEND on each other must stay sequential: if call B needs call A's output (an account id, a resource id, a profile name), make call A now and call B on the next turn. +- Never batch mutations. Issue at most ONE state-changing call per turn so each is reviewed and approved individually. +- Do not skip ahead to later plan steps. +``` + +In `planNode`'s `## Rules for Plan Steps`, replace line 292: + +``` +- Keep the plan SHORT: never more than 7 steps. Merge related read-only queries into a single step (e.g., one step for "inventory EC2 + RDS + EBS across regions", not one step per service per region). +``` + +with: + +``` +- Keep the plan SHORT: never more than 7 steps. Merge related read-only queries into a single step (e.g., one step for "inventory EC2 + RDS + EBS across regions", not one step per service per region). The executor issues the individual queries in that step as parallel tool calls, so a merged step costs no more time than a narrow one. +``` + +In `reviseNode`'s `## Revision Approach`, append a ninth item after item 8 (line 759): + +``` +9. When the fix requires several independent read-only re-checks, issue them as multiple tool calls in this single turn rather than one per turn. +``` + +- [ ] **Step 7: Verify typecheck and tests** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +Run: `cd apps/web-ui && bunx vitest run lib/agent/` +Expected: PASS with no new failures. + +- [ ] **Step 8: Measure against the Task 2 baseline** + +Re-run the same representative task used in Task 2 Step 8. Compare the `📊 [RUN SUMMARY]` `EXECUTOR calls=` count and total wall time. A large drop in executor call count is the signal this task worked. Record both numbers in the commit message. + +- [ ] **Step 9: Commit** + +```bash +git add apps/web-ui/lib/agent/concurrency.ts apps/web-ui/lib/agent/concurrency.test.ts apps/web-ui/lib/agent/tools.ts apps/web-ui/lib/agent/planning-agent.ts +git commit -m "perf(agent): allow batched parallel tool calls in one executor turn + +ToolNode already ran multiple tool_calls concurrently; the executor prompt +forbade emitting them, so an N-way independent sweep cost N serial LLM laps. +Independent read-only calls now batch into one turn while dependent calls and +mutations stay sequential. Shell subprocesses are bounded by TOOL_CONCURRENCY." +``` + +--- + +### Task 4: Sub-agent budget resolution + +Pure configuration logic with no I/O beyond the tenant-config read, so it is fully unit-testable and can land before anything consumes it. Mirrors `lib/agent-ops/agent-ops-defaults.ts`. + +**Files:** +- Create: `apps/web-ui/lib/agent/subagent-budget.ts` +- Create: `apps/web-ui/lib/agent/subagent-budget.test.ts` + +**Interfaces:** +- Consumes: `TenantConfigService.getConfig` from `@/lib/tenant-config-service`. +- Produces: + - `const SUBAGENT_CONFIG_KEY = 'aiops-subagents'` + - `interface SubagentBudgetConfig { enabled: boolean; maxConcurrentSubagents: number; maxSubagentsPerRun: number; maxSubagentTokensPerRun: number; subagentMaxIterations: number; subagentTimeoutMs: number }` + - `const BUDGET_BOUNDS: Record, { min: number; max: number; default: number }>` + - `platformSubagentsEnabled(): boolean` + - `clampBudget(input: Partial | null): SubagentBudgetConfig` + - `resolveSubagentBudget(tenantId: string): Promise` + - `validateBudgetInput(input: unknown): string | null` + +- [ ] **Step 1: Write the failing test** + +Create `apps/web-ui/lib/agent/subagent-budget.test.ts`: + +```typescript +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('@/lib/tenant-config-service', () => ({ + TenantConfigService: { getConfig: vi.fn() }, +})); + +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { + clampBudget, + resolveSubagentBudget, + validateBudgetInput, + platformSubagentsEnabled, + BUDGET_BOUNDS, +} from './subagent-budget'; + +beforeEach(() => { + vi.clearAllMocks(); + process.env.SUBAGENTS_ENABLED = 'true'; +}); +afterEach(() => { + delete process.env.SUBAGENTS_ENABLED; + delete process.env.SUBAGENT_MAX_CONCURRENCY; +}); + +describe('platformSubagentsEnabled', () => { + it('is false unless SUBAGENTS_ENABLED is exactly "true"', () => { + delete process.env.SUBAGENTS_ENABLED; + expect(platformSubagentsEnabled()).toBe(false); + process.env.SUBAGENTS_ENABLED = 'false'; + expect(platformSubagentsEnabled()).toBe(false); + process.env.SUBAGENTS_ENABLED = 'true'; + expect(platformSubagentsEnabled()).toBe(true); + }); +}); + +describe('clampBudget', () => { + it('returns defaults for null input', () => { + const budget = clampBudget(null); + expect(budget.maxConcurrentSubagents).toBe(BUDGET_BOUNDS.maxConcurrentSubagents.default); + expect(budget.enabled).toBe(false); + }); + + it('clamps a value above the ceiling down', () => { + expect(clampBudget({ maxConcurrentSubagents: 50 }).maxConcurrentSubagents) + .toBe(BUDGET_BOUNDS.maxConcurrentSubagents.max); + }); + + it('clamps a value below the floor up', () => { + expect(clampBudget({ maxSubagentsPerRun: 0 }).maxSubagentsPerRun) + .toBe(BUDGET_BOUNDS.maxSubagentsPerRun.min); + }); + + it('rounds non-integers', () => { + expect(clampBudget({ subagentMaxIterations: 7.6 }).subagentMaxIterations).toBe(8); + }); + + it('falls back to the default for non-numeric values', () => { + expect(clampBudget({ subagentTimeoutMs: 'abc' as unknown as number }).subagentTimeoutMs) + .toBe(BUDGET_BOUNDS.subagentTimeoutMs.default); + }); + + it('honours an env ceiling lower than the built-in max', () => { + process.env.SUBAGENT_MAX_CONCURRENCY = '2'; + expect(clampBudget({ maxConcurrentSubagents: 6 }).maxConcurrentSubagents).toBe(2); + }); +}); + +describe('resolveSubagentBudget', () => { + it('returns a disabled budget when the platform kill-switch is off', async () => { + process.env.SUBAGENTS_ENABLED = 'false'; + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ enabled: true } as never); + + expect((await resolveSubagentBudget('t1')).enabled).toBe(false); + }); + + it('returns tenant config clamped when the platform allows it', async () => { + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ + enabled: true, maxConcurrentSubagents: 99, + } as never); + + const budget = await resolveSubagentBudget('t1'); + expect(budget.enabled).toBe(true); + expect(budget.maxConcurrentSubagents).toBe(BUDGET_BOUNDS.maxConcurrentSubagents.max); + }); + + it('returns defaults (disabled) when the tenant has no config', async () => { + vi.mocked(TenantConfigService.getConfig).mockResolvedValue(null as never); + expect((await resolveSubagentBudget('t1')).enabled).toBe(false); + }); + + it('never throws when the config read fails', async () => { + vi.mocked(TenantConfigService.getConfig).mockRejectedValue(new Error('db down')); + await expect(resolveSubagentBudget('t1')).resolves.toMatchObject({ enabled: false }); + }); +}); + +describe('validateBudgetInput', () => { + it('accepts a well-formed payload', () => { + expect(validateBudgetInput({ + enabled: true, maxConcurrentSubagents: 3, maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400000, subagentMaxIterations: 8, subagentTimeoutMs: 180000, + })).toBeNull(); + }); + + it('rejects a non-object', () => { + expect(validateBudgetInput(null)).toMatch(/object/i); + }); + + it('rejects a non-boolean enabled', () => { + expect(validateBudgetInput({ enabled: 'yes' })).toMatch(/enabled/i); + }); + + it('rejects an out-of-range number with a message naming the field', () => { + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: 999 })) + .toMatch(/maxConcurrentSubagents/); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/subagent-budget.test.ts` +Expected: FAIL — cannot resolve `./subagent-budget`. + +- [ ] **Step 3: Implement the module** + +Create `apps/web-ui/lib/agent/subagent-budget.ts`: + +```typescript +/** + * Sub-agent budget configuration. + * + * web-ui runs as a SHARED ECS task, so these limits cannot be purely + * tenant-controlled: one tenant setting concurrency to 50 would saturate the + * container's event loop for every co-tenant and flood their Bedrock quota. + * Env vars are therefore CEILINGS, not overrides: + * + * effective = clamp(tenantConfig ?? default, MIN, envCeiling) + * + * Clamping runs on READ (not only on write) so lowering a ceiling retroactively + * binds config rows written while it was higher. + * + * SUBAGENTS_ENABLED is an absolute platform kill-switch: tenant config can + * never re-enable the feature when it is off. + */ +import { TenantConfigService } from '@/lib/tenant-config-service'; + +export const SUBAGENT_CONFIG_KEY = 'aiops-subagents'; + +export interface SubagentBudgetConfig { + enabled: boolean; + maxConcurrentSubagents: number; + maxSubagentsPerRun: number; + maxSubagentTokensPerRun: number; + subagentMaxIterations: number; + subagentTimeoutMs: number; +} + +type NumericKey = keyof Omit; + +interface Bound { min: number; max: number; default: number; envCeiling: string } + +/** + * `max` is the hard platform ceiling. `envCeiling` names an env var that may + * lower it further (never raise it) for a given deployment. + */ +const BOUNDS_SPEC: Record = { + maxConcurrentSubagents: { min: 1, max: 6, default: 3, envCeiling: 'SUBAGENT_MAX_CONCURRENCY' }, + maxSubagentsPerRun: { min: 1, max: 16, default: 8, envCeiling: 'SUBAGENT_MAX_PER_RUN' }, + maxSubagentTokensPerRun: { min: 50_000, max: 1_000_000, default: 400_000, envCeiling: 'SUBAGENT_MAX_TOKENS_PER_RUN' }, + subagentMaxIterations: { min: 2, max: 16, default: 8, envCeiling: 'SUBAGENT_MAX_ITERATIONS' }, + subagentTimeoutMs: { min: 30_000, max: 300_000, default: 180_000, envCeiling: 'SUBAGENT_TIMEOUT_MS' }, +}; + +export const BUDGET_BOUNDS = BOUNDS_SPEC; + +/** The effective ceiling: the built-in max, lowered (never raised) by env. */ +function ceilingFor(key: NumericKey): number { + const spec = BOUNDS_SPEC[key]; + const fromEnv = Number(process.env[spec.envCeiling]); + if (!Number.isFinite(fromEnv) || fromEnv <= 0) return spec.max; + return Math.min(spec.max, Math.round(fromEnv)); +} + +/** Absolute platform kill-switch. Defaults to OFF. */ +export function platformSubagentsEnabled(): boolean { + return process.env.SUBAGENTS_ENABLED === 'true'; +} + +export function clampBudget(input: Partial | null): SubagentBudgetConfig { + const result: Record = { enabled: input?.enabled === true }; + + for (const key of Object.keys(BOUNDS_SPEC) as NumericKey[]) { + const spec = BOUNDS_SPEC[key]; + const raw = Number(input?.[key]); + const value = Number.isFinite(raw) ? Math.round(raw) : spec.default; + result[key] = Math.min(ceilingFor(key), Math.max(spec.min, value)); + } + + return result as unknown as SubagentBudgetConfig; +} + +/** + * Read a tenant's budget, clamped. Never throws: a config-store failure must + * degrade to "sub-agents off", not break the chat run. + */ +export async function resolveSubagentBudget(tenantId: string): Promise { + if (!platformSubagentsEnabled()) { + return { ...clampBudget(null), enabled: false }; + } + + const stored = await TenantConfigService + .getConfig>(SUBAGENT_CONFIG_KEY, tenantId) + .catch(() => null); + + return clampBudget(stored); +} + +/** Validate a PUT payload. Returns an error string for a 400, or null. */ +export function validateBudgetInput(input: unknown): string | null { + if (!input || typeof input !== 'object' || Array.isArray(input)) { + return 'Request body must be an object'; + } + const body = input as Record; + + if (typeof body.enabled !== 'boolean') { + return 'enabled must be a boolean'; + } + + for (const key of Object.keys(BOUNDS_SPEC) as NumericKey[]) { + if (body[key] === undefined) continue; + const spec = BOUNDS_SPEC[key]; + const value = Number(body[key]); + const ceiling = ceilingFor(key); + if (!Number.isInteger(value) || value < spec.min || value > ceiling) { + return `${key} must be an integer between ${spec.min} and ${ceiling}`; + } + } + + return null; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/subagent-budget.test.ts` +Expected: PASS, 15 tests. + +- [ ] **Step 5: Declare the env vars** + +In `apps/web-ui/env.ts`, add to the `server:` block alongside the other optional string vars (near `LLM_AUDIT` on line 51): + +```typescript + SUBAGENTS_ENABLED: z.string().optional(), + SUBAGENT_MAX_CONCURRENCY: z.string().optional(), + SUBAGENT_MAX_PER_RUN: z.string().optional(), + SUBAGENT_MAX_TOKENS_PER_RUN: z.string().optional(), + SUBAGENT_MAX_ITERATIONS: z.string().optional(), + SUBAGENT_TIMEOUT_MS: z.string().optional(), + TOOL_CONCURRENCY: z.string().optional(), +``` + +Add matching commented entries to the repo-root `.env.example`: + +```bash +# AI Ops sub-agents (Layer B). SUBAGENTS_ENABLED is an absolute platform +# kill-switch — tenant settings cannot re-enable it when false. +# The SUBAGENT_MAX_* vars are CEILINGS that lower (never raise) the built-in +# maximums a tenant may configure. +SUBAGENTS_ENABLED=false +# SUBAGENT_MAX_CONCURRENCY=6 +# SUBAGENT_MAX_PER_RUN=16 +# SUBAGENT_MAX_TOKENS_PER_RUN=1000000 +# SUBAGENT_MAX_ITERATIONS=16 +# SUBAGENT_TIMEOUT_MS=300000 +# Max concurrent shell subprocesses from execute_command (Layer A). +# TOOL_CONCURRENCY=6 +``` + +- [ ] **Step 6: Verify typecheck** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +- [ ] **Step 7: Commit** + +```bash +git add apps/web-ui/lib/agent/subagent-budget.ts apps/web-ui/lib/agent/subagent-budget.test.ts apps/web-ui/env.ts .env.example +git commit -m "feat(agent): sub-agent budget config with platform ceilings + +Tenant-configurable sub-agent limits resolved as +clamp(tenantConfig ?? default, MIN, envCeiling), applied on read so lowering a +ceiling binds existing rows. SUBAGENTS_ENABLED is an absolute kill-switch." +``` + +--- + +### Task 5: AI Ops settings API route + +**Files:** +- Create: `apps/web-ui/app/api/settings/aiops/route.ts` +- Create: `apps/web-ui/app/api/settings/aiops/route.test.ts` + +**Interfaces:** +- Consumes: `resolveSubagentBudget`, `validateBudgetInput`, `clampBudget`, `platformSubagentsEnabled`, `BUDGET_BOUNDS`, `SUBAGENT_CONFIG_KEY` from Task 4. +- Produces: `GET /api/settings/aiops` → `{ success: true, data: { budget: SubagentBudgetConfig, bounds, platformEnabled: boolean } }`; `PUT` → `{ success: true, data: { budget: SubagentBudgetConfig } }`. + +- [ ] **Step 1: Write the failing test** + +Create `apps/web-ui/app/api/settings/aiops/route.test.ts`: + +```typescript +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('@/lib/rbac/authorize', () => ({ authorize: vi.fn() })); +vi.mock('@/lib/auth-session', () => ({ getSessionTenantId: vi.fn() })); +vi.mock('next-auth', () => ({ getServerSession: vi.fn() })); +vi.mock('@/lib/auth-options', () => ({ authOptions: {} })); +vi.mock('@/lib/tenant-config-service', () => ({ + TenantConfigService: { getConfig: vi.fn(), saveConfig: vi.fn() }, +})); +vi.mock('@/lib/audit-service', () => ({ + AuditService: { logUserAction: vi.fn().mockResolvedValue(undefined) }, +})); + +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getServerSession } from 'next-auth'; +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { AuditService } from '@/lib/audit-service'; +import { GET, PUT } from './route'; + +const putRequest = (body: unknown) => + ({ json: async () => body }) as unknown as Parameters[0]; + +beforeEach(() => { + vi.clearAllMocks(); + process.env.SUBAGENTS_ENABLED = 'true'; + vi.mocked(authorize).mockResolvedValue(null as never); + vi.mocked(getSessionTenantId).mockResolvedValue('t1' as never); + vi.mocked(getServerSession).mockResolvedValue({ user: { email: 'a@b.com' } } as never); + vi.mocked(TenantConfigService.getConfig).mockResolvedValue(null as never); +}); +afterEach(() => { delete process.env.SUBAGENTS_ENABLED; }); + +describe('GET /api/settings/aiops', () => { + it('returns the effective budget, bounds, and platform flag', async () => { + const body = await (await GET()).json(); + + expect(body.success).toBe(true); + expect(body.data.platformEnabled).toBe(true); + expect(body.data.budget.maxConcurrentSubagents).toBe(3); + expect(body.data.bounds.maxConcurrentSubagents.max).toBe(6); + }); + + it('propagates an RBAC denial', async () => { + const denied = { status: 403 }; + vi.mocked(authorize).mockResolvedValue(denied as never); + expect(await GET()).toBe(denied); + }); + + it('403s with no tenant context', async () => { + vi.mocked(getSessionTenantId).mockResolvedValue(null as never); + expect((await GET()).status).toBe(403); + }); +}); + +describe('PUT /api/settings/aiops', () => { + const valid = { + enabled: true, maxConcurrentSubagents: 4, maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400000, subagentMaxIterations: 8, subagentTimeoutMs: 180000, + }; + + it('saves a valid payload and audits it', async () => { + const res = await PUT(putRequest(valid)); + const body = await res.json(); + + expect(body.success).toBe(true); + expect(body.data.budget.maxConcurrentSubagents).toBe(4); + expect(TenantConfigService.saveConfig).toHaveBeenCalledWith( + 'aiops-subagents', expect.objectContaining({ maxConcurrentSubagents: 4 }), 't1', 'a@b.com', + ); + expect(AuditService.logUserAction).toHaveBeenCalled(); + }); + + it('400s on an out-of-range value', async () => { + const res = await PUT(putRequest({ ...valid, maxConcurrentSubagents: 999 })); + expect(res.status).toBe(400); + expect((await res.json()).error).toMatch(/maxConcurrentSubagents/); + }); + + it('400s when enabled is missing', async () => { + const { enabled, ...rest } = valid; + expect((await PUT(putRequest(rest))).status).toBe(400); + }); + + it('persists the clamped value, not the raw one', async () => { + process.env.SUBAGENT_MAX_CONCURRENCY = '2'; + await PUT(putRequest({ ...valid, maxConcurrentSubagents: 2 })); + expect(TenantConfigService.saveConfig).toHaveBeenCalledWith( + 'aiops-subagents', expect.objectContaining({ maxConcurrentSubagents: 2 }), 't1', 'a@b.com', + ); + delete process.env.SUBAGENT_MAX_CONCURRENCY; + }); + + it('propagates an RBAC denial', async () => { + const denied = { status: 403 }; + vi.mocked(authorize).mockResolvedValue(denied as never); + expect(await PUT(putRequest(valid))).toBe(denied); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run app/api/settings/aiops/route.test.ts` +Expected: FAIL — cannot resolve `./route`. + +- [ ] **Step 3: Implement the route** + +Create `apps/web-ui/app/api/settings/aiops/route.ts`: + +```typescript +import { NextRequest, NextResponse } from 'next/server'; +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getServerSession } from 'next-auth'; +import { authOptions } from '@/lib/auth-options'; +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { AuditService } from '@/lib/audit-service'; +import { + SUBAGENT_CONFIG_KEY, + BUDGET_BOUNDS, + clampBudget, + platformSubagentsEnabled, + resolveSubagentBudget, + validateBudgetInput, +} from '@/lib/agent/subagent-budget'; + +export async function GET() { + console.log('API - GET /api/settings/aiops - Fetching sub-agent budget'); + + const authError = await authorize('read', 'Agent'); + if (authError) return authError; + + try { + const tenantId = await getSessionTenantId(); + if (!tenantId) { + return NextResponse.json({ success: false, error: 'No tenant context' }, { status: 403 }); + } + + const budget = await resolveSubagentBudget(tenantId); + + return NextResponse.json({ + success: true, + data: { + budget, + // The UI renders sliders bounded by these, and explains why a + // ceiling is what it is rather than silently clipping input. + bounds: BUDGET_BOUNDS, + platformEnabled: platformSubagentsEnabled(), + }, + }); + } catch (error) { + console.error('API - Error fetching AI Ops settings:', error); + return NextResponse.json( + { success: false, error: error instanceof Error ? error.message : 'Failed to fetch AI Ops settings' }, + { status: 500 }, + ); + } +} + +export async function PUT(request: NextRequest) { + console.log('API - PUT /api/settings/aiops - Saving sub-agent budget'); + + const authError = await authorize('update', 'Agent'); + if (authError) return authError; + + try { + const tenantId = await getSessionTenantId(); + if (!tenantId) { + return NextResponse.json({ success: false, error: 'No tenant context' }, { status: 403 }); + } + + const body = await request.json(); + const validationError = validateBudgetInput(body); + if (validationError) { + return NextResponse.json({ success: false, error: validationError }, { status: 400 }); + } + + // Persist the CLAMPED value so a stored row can never exceed the ceiling + // that was in force when it was written. + const budget = clampBudget(body); + + const session = await getServerSession(authOptions); + const updatedBy = session?.user?.email || 'api-user'; + + await TenantConfigService.saveConfig(SUBAGENT_CONFIG_KEY, budget, tenantId, updatedBy); + + await AuditService.logUserAction({ + eventType: 'aiops.subagents.settings.updated', + severity: 'medium', + apiRoute: 'PUT /api/settings/aiops', + httpMethod: 'PUT', + action: 'Update AI Ops Sub-Agent Settings', + resourceType: 'settings', + resourceId: SUBAGENT_CONFIG_KEY, + resourceName: 'AI Ops Sub-Agent Budget', + user: updatedBy, + userType: 'user', + status: 'success', + details: `Sub-agents ${budget.enabled ? 'enabled' : 'disabled'}; concurrency=${budget.maxConcurrentSubagents}, perRun=${budget.maxSubagentsPerRun}, tokenBudget=${budget.maxSubagentTokensPerRun}`, + tenantId, + }); + + return NextResponse.json({ success: true, data: { budget } }); + } catch (error) { + console.error('API - Error saving AI Ops settings:', error); + return NextResponse.json( + { success: false, error: error instanceof Error ? error.message : 'Failed to save AI Ops settings' }, + { status: 500 }, + ); + } +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run app/api/settings/aiops/route.test.ts` +Expected: PASS, 9 tests. + +- [ ] **Step 5: Typecheck and commit** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +```bash +git add apps/web-ui/app/api/settings/aiops/ +git commit -m "feat(api): GET/PUT /api/settings/aiops for sub-agent budget + +Reuses the Agent RBAC subject (already mapped to the AIOps module) and stores +the clamped budget under the aiops-subagents tenant-config key." +``` + +--- + +### Task 6: AI Ops settings UI + +**Files:** +- Create: `apps/web-ui/lib/queries/aiops-settings.ts` +- Create: `apps/web-ui/components/settings/aiops-subagent-settings.tsx` +- Create: `apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx` +- Modify: `apps/web-ui/lib/queries/query-keys.ts` + +**Interfaces:** +- Consumes: `GET`/`PUT /api/settings/aiops` from Task 5; `SubagentBudgetConfig` from Task 4. +- Produces: `useAiopsSubagentSettings()`, `useSaveAiopsSubagentSettings()`, ``. + +- [ ] **Step 1: Add the query key** + +In `apps/web-ui/lib/queries/query-keys.ts`, add a domain entry alongside the others: + +```typescript + aiopsSettings: { + all: ['aiops-settings'] as const, + subagents: () => [...queryKeys.aiopsSettings.all, 'subagents'] as const, + }, +``` + +- [ ] **Step 2: Write the query hooks** + +Create `apps/web-ui/lib/queries/aiops-settings.ts`: + +```typescript +'use client'; + +/** + * TanStack Query hooks for the AI Ops sub-agent budget (per-tenant). + */ +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { queryKeys } from './query-keys'; + +export interface SubagentBudget { + enabled: boolean; + maxConcurrentSubagents: number; + maxSubagentsPerRun: number; + maxSubagentTokensPerRun: number; + subagentMaxIterations: number; + subagentTimeoutMs: number; +} + +export interface SubagentBound { min: number; max: number; default: number } + +export interface AiopsSubagentSettings { + budget: SubagentBudget; + bounds: Record, SubagentBound>; + /** False when SUBAGENTS_ENABLED is off for the whole deployment. */ + platformEnabled: boolean; +} + +export function useAiopsSubagentSettings() { + return useQuery({ + queryKey: queryKeys.aiopsSettings.subagents(), + queryFn: async (): Promise => { + const res = await fetch('/api/settings/aiops'); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) { + throw new Error(json.error || 'Failed to load AI Ops settings'); + } + return json.data as AiopsSubagentSettings; + }, + }); +} + +export function useSaveAiopsSubagentSettings() { + const qc = useQueryClient(); + return useMutation({ + mutationFn: async (body: SubagentBudget): Promise => { + const res = await fetch('/api/settings/aiops', { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) { + throw new Error(json.error || 'Failed to save AI Ops settings'); + } + return json.data.budget as SubagentBudget; + }, + onSuccess: () => qc.invalidateQueries({ queryKey: queryKeys.aiopsSettings.all }), + }); +} +``` + +- [ ] **Step 3: Build the settings panel** + +Create `apps/web-ui/components/settings/aiops-subagent-settings.tsx`. Note 2-space indentation (components convention) and React Hook Form + Zod per the global constraints: + +```tsx +'use client'; + +import { useEffect } from 'react'; +import { useForm } from 'react-hook-form'; +import { zodResolver } from '@hookform/resolvers/zod'; +import { z } from 'zod'; +import { toast } from 'sonner'; +import { Bot, Gauge, Loader2, Save } from 'lucide-react'; + +import { + useAiopsSubagentSettings, + useSaveAiopsSubagentSettings, + type SubagentBudget, +} from '@/lib/queries/aiops-settings'; +import { Button } from '@/components/ui/button'; +import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card'; +import { Input } from '@/components/ui/input'; +import { Label } from '@/components/ui/label'; +import { Switch } from '@/components/ui/switch'; +import { Alert, AlertDescription } from '@/components/ui/alert'; +import { Spinner } from '@/components/ui/spinner'; + +const schema = z.object({ + enabled: z.boolean(), + maxConcurrentSubagents: z.number().int(), + maxSubagentsPerRun: z.number().int(), + maxSubagentTokensPerRun: z.number().int(), + subagentMaxIterations: z.number().int(), + subagentTimeoutMs: z.number().int(), +}); + +type FormValues = z.infer; + +const FIELDS: Array<{ + name: keyof Omit; + label: string; + help: string; +}> = [ + { + name: 'maxConcurrentSubagents', + label: 'Concurrent sub-agents', + help: 'How many sub-agents may run at the same time. Higher finishes fan-out work faster but increases the chance of provider throttling.', + }, + { + name: 'maxSubagentsPerRun', + label: 'Sub-agents per run', + help: 'Total sub-agents a single chat run may dispatch before it falls back to working serially.', + }, + { + name: 'maxSubagentTokensPerRun', + label: 'Token budget per run', + help: 'Combined sub-agent token spend allowed in one run. When exhausted, the agent completes the work itself instead of failing.', + }, + { + name: 'subagentMaxIterations', + label: 'Iterations per sub-agent', + help: 'Reasoning/tool laps a single sub-agent may take before it must report what it has found.', + }, + { + name: 'subagentTimeoutMs', + label: 'Sub-agent timeout (ms)', + help: 'Wall-clock limit for one sub-agent. On timeout it returns partial findings rather than failing the run.', + }, +]; + +export function AiopsSubagentSettings() { + const { data, isLoading, error } = useAiopsSubagentSettings(); + const saveMutation = useSaveAiopsSubagentSettings(); + + const form = useForm({ + resolver: zodResolver(schema), + defaultValues: { + enabled: false, + maxConcurrentSubagents: 3, + maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400000, + subagentMaxIterations: 8, + subagentTimeoutMs: 180000, + }, + }); + + useEffect(() => { + if (data?.budget) form.reset(data.budget); + }, [data, form]); + + if (isLoading) { + return ( +
+ +
+ ); + } + + if (error) { + return ( + + {(error as Error).message} + + ); + } + + const platformEnabled = data?.platformEnabled ?? false; + const bounds = data?.bounds; + + const onSubmit = async (values: FormValues) => { + // Bounds are enforced server-side too; this only produces a friendlier message. + for (const field of FIELDS) { + const bound = bounds?.[field.name]; + const value = values[field.name]; + if (bound && (value < bound.min || value > bound.max)) { + toast.error(`${field.label} must be between ${bound.min} and ${bound.max}`); + return; + } + } + try { + await saveMutation.mutateAsync(values); + toast.success('Sub-agent settings saved — new runs will use this configuration'); + } catch (err) { + toast.error(err instanceof Error ? err.message : 'Failed to save sub-agent settings'); + } + }; + + return ( +
+
+

Sub-Agents

+

+ Sub-agents let one AI Ops run investigate several accounts or regions at the same time. + They are read-only: anything that changes infrastructure still runs on the main agent + under your normal approval flow. +

+
+ + {!platformEnabled && ( + + + Sub-agents are disabled for this deployment. These settings are saved but will not take + effect until an administrator enables the feature. + + + )} + + + + + + Enable sub-agents + + + When off, every run executes serially exactly as before. + + + +
+ form.setValue('enabled', checked, { shouldDirty: true })} + disabled={!platformEnabled} + /> + +
+ + + + + + + + Budget limits + + + These bound how much work and spend one run may fan out. Exceeding a limit never fails + the run — the agent finishes the work serially instead. + + + + {FIELDS.map((field) => { + const bound = bounds?.[field.name]; + return ( +
+ + +

{field.help}

+ {bound && ( +

+ Allowed: {bound.min}–{bound.max}. Default {bound.default}. +

+ )} +
+ ); + })} + + + + +
+ ); +} +``` + +- [ ] **Step 4: Add the page** + +Create `apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx`: + +```tsx +'use client'; + +import { AiopsSubagentSettings } from '@/components/settings/aiops-subagent-settings'; + +export default function AiopsSubagentSettingsPage() { + return ( +
+ +
+ ); +} +``` + +- [ ] **Step 5: Confirm the Switch and Spinner primitives exist** + +Run: `ls apps/web-ui/components/ui/switch.tsx apps/web-ui/components/ui/spinner.tsx` +Expected: both files listed. If `switch.tsx` is absent, add it with `bunx shadcn@latest add switch` from `apps/web-ui` — do not hand-write a primitive. + +- [ ] **Step 6: Typecheck, lint, and verify in the browser** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +Run: `cd apps/web-ui && bun run lint` +Expected: no new errors. + +Start the dev server and open `http://localhost:3001/app/agent-ops/settings/subagents`. Confirm: values load, the platform-disabled alert shows when `SUBAGENTS_ENABLED` is unset, saving fires a success toast, and an out-of-range value produces an error toast. + +- [ ] **Step 7: Commit** + +```bash +git add apps/web-ui/lib/queries/aiops-settings.ts apps/web-ui/lib/queries/query-keys.ts apps/web-ui/components/settings/aiops-subagent-settings.tsx apps/web-ui/app/app/agent-ops/settings/subagents/ +git commit -m "feat(ui): AI Ops sub-agent budget settings panel" +``` + +--- + +### Task 7: Sub-agent runtime with the read-only jail + +The core of Layer B. An ephemeral ReAct loop that cannot mutate anything, cannot spawn further sub-agents, and always returns a bounded report string. + +**Files:** +- Create: `apps/web-ui/lib/agent/subagent.ts` +- Create: `apps/web-ui/lib/agent/subagent.test.ts` + +**Interfaces:** +- Consumes: `classifyTool` from `./tool-classifier`; `Semaphore` from `./concurrency` (Task 3); `SubagentBudgetConfig` from `./subagent-budget` (Task 4); `contentToText`, `truncateOutput`, `sanitizeMessagesForBedrock` from `./agent-shared`. +- Produces: + - `const SUBAGENT_REPORT_MAX_CHARS = 6000` + - `interface SubagentSpec { role: string; task: string; expectedOutput: string }` + - `interface SubagentResult { report: string; toolCount: number; tokensIn: number; tokensOut: number; status: 'done' | 'failed'; transcript: Array<{ kind: 'ai' | 'tool'; name?: string; text: string }> }` + - `isReadOnlyForSubagent(name: string, args?: Record): { allowed: boolean; reason: string }` + - `filterReadOnlyTools(tools: T[]): T[]` + - `runSubagent(spec: SubagentSpec, deps: SubagentDeps): Promise` + - `interface SubagentDeps { model: { bindTools?: (t: unknown[]) => any; invoke: (m: unknown[]) => Promise }; tools: Array<{ name: string; invoke: (args: Record) => Promise }>; budget: SubagentBudgetConfig; onEvent?: (e: { toolCount: number; tokensIn: number; tokensOut: number }) => void }` + +- [ ] **Step 1: Write the failing test** + +Create `apps/web-ui/lib/agent/subagent.test.ts`: + +```typescript +import { describe, it, expect, vi } from 'vitest'; +import { + isReadOnlyForSubagent, + filterReadOnlyTools, + runSubagent, + SUBAGENT_REPORT_MAX_CHARS, + type SubagentSpec, +} from './subagent'; + +const BUDGET = { + enabled: true, + maxConcurrentSubagents: 3, + maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400_000, + subagentMaxIterations: 4, + subagentTimeoutMs: 5_000, +}; + +const SPEC: SubagentSpec = { + role: 'EC2 auditor', + task: 'List idle instances in account 111111111111', + expectedOutput: 'instance ids with CPU below 5%', +}; + +/** Model stub: returns each scripted response in turn. */ +function scriptedModel(responses: Array<{ content: string; tool_calls?: Array<{ id: string; name: string; args: Record }> }>) { + let i = 0; + const model: any = { + bindTools: () => model, + invoke: vi.fn(async () => { + const r = responses[Math.min(i, responses.length - 1)]; + i++; + return { content: r.content, tool_calls: r.tool_calls ?? [], usage_metadata: { input_tokens: 100, output_tokens: 20 } }; + }), + }; + return model; +} + +const readTool = { name: 'describe_instances', invoke: vi.fn(async () => 'i-123 running') }; +const shellRead = { name: 'execute_command', invoke: vi.fn(async () => 'ok') }; +const writeTool = { name: 'write_file', invoke: vi.fn(async () => 'written') }; + +describe('isReadOnlyForSubagent', () => { + it('allows an explicit read-only tool', () => { + expect(isReadOnlyForSubagent('describe_instances').allowed).toBe(true); + }); + + it('allows a read-only shell command', () => { + expect(isReadOnlyForSubagent('execute_command', { command: 'aws ec2 describe-instances' }).allowed).toBe(true); + }); + + it('blocks a mutative shell command', () => { + const verdict = isReadOnlyForSubagent('execute_command', { command: 'aws ec2 terminate-instances --instance-ids i-1' }); + expect(verdict.allowed).toBe(false); + expect(verdict.reason).toMatch(/mutat/i); + }); + + it('blocks a mutative tool name', () => { + expect(isReadOnlyForSubagent('write_file').allowed).toBe(false); + }); + + it('blocks an unknown-named tool (fail closed)', () => { + // classifyTool returns isMutative:false, matchedRule:false for unknowns. + // In the orchestrator that means "ask the human"; here there is no human, + // so it must mean "block". + const verdict = isReadOnlyForSubagent('some_mcp_thing'); + expect(verdict.allowed).toBe(false); + expect(verdict.reason).toMatch(/not on the verified read-only list/i); + }); + + it('blocks dispatch_agent so sub-agents cannot recurse', () => { + expect(isReadOnlyForSubagent('dispatch_agent').allowed).toBe(false); + }); + + it('blocks ask_user — no human is reachable inside a tool call', () => { + expect(isReadOnlyForSubagent('ask_user').allowed).toBe(false); + }); +}); + +describe('filterReadOnlyTools', () => { + it('keeps read-only tools and drops the rest', () => { + const kept = filterReadOnlyTools([readTool, shellRead, writeTool, { name: 'dispatch_agent' }]); + expect(kept.map(t => t.name)).toEqual(['describe_instances', 'execute_command']); + }); +}); + +describe('runSubagent', () => { + it('returns the final prose as the report', async () => { + const model = scriptedModel([{ content: 'Found 2 idle instances: i-1, i-2' }]); + const result = await runSubagent(SPEC, { model, tools: [readTool], budget: BUDGET }); + + expect(result.status).toBe('done'); + expect(result.report).toContain('i-1'); + expect(result.tokensIn).toBe(100); + }); + + it('executes an allowed tool call and loops', async () => { + const model = scriptedModel([ + { content: '', tool_calls: [{ id: '1', name: 'describe_instances', args: {} }] }, + { content: 'Instance i-123 is running' }, + ]); + const result = await runSubagent(SPEC, { model, tools: [readTool], budget: BUDGET }); + + expect(readTool.invoke).toHaveBeenCalled(); + expect(result.toolCount).toBe(1); + expect(result.report).toContain('i-123'); + }); + + it('refuses a mutative tool call instead of executing it', async () => { + const model = scriptedModel([ + { content: '', tool_calls: [{ id: '1', name: 'write_file', args: { file_path: 'x', content: 'y' } }] }, + { content: 'Understood — reporting instead.' }, + ]); + const result = await runSubagent(SPEC, { model, tools: [readTool, writeTool], budget: BUDGET }); + + expect(writeTool.invoke).not.toHaveBeenCalled(); + expect(result.status).toBe('done'); + }); + + it('stops at the iteration cap and marks the report incomplete', async () => { + const model = scriptedModel([ + { content: '', tool_calls: [{ id: '1', name: 'describe_instances', args: {} }] }, + ]); + const result = await runSubagent(SPEC, { model, tools: [readTool], budget: { ...BUDGET, subagentMaxIterations: 2 } }); + + expect(result.report).toMatch(/incomplete/i); + expect(model.invoke).toHaveBeenCalledTimes(2); + }); + + it('truncates an over-long report', async () => { + const model = scriptedModel([{ content: 'x'.repeat(SUBAGENT_REPORT_MAX_CHARS + 5000) }]); + const result = await runSubagent(SPEC, { model, tools: [readTool], budget: BUDGET }); + + expect(result.report.length).toBeLessThanOrEqual(SUBAGENT_REPORT_MAX_CHARS + 100); + expect(result.report).toMatch(/TRUNCATED/); + }); + + it('returns a failed status instead of throwing when the model errors', async () => { + const model: any = { bindTools: () => model, invoke: vi.fn().mockRejectedValue(new Error('provider down')) }; + const result = await runSubagent(SPEC, { model, tools: [readTool], budget: BUDGET }); + + expect(result.status).toBe('failed'); + expect(result.report).toMatch(/provider down/); + }); + + it('returns partial findings on timeout', async () => { + const model: any = { + bindTools: () => model, + invoke: vi.fn(() => new Promise(resolve => setTimeout(() => resolve({ content: 'late', tool_calls: [] }), 500))), + }; + const result = await runSubagent(SPEC, { model, tools: [readTool], budget: { ...BUDGET, subagentTimeoutMs: 50 } }); + + expect(result.report).toMatch(/timed out/i); + }); + + it('reports a failing tool without aborting the loop', async () => { + const boom = { name: 'describe_instances', invoke: vi.fn().mockRejectedValue(new Error('AccessDenied')) }; + const model = scriptedModel([ + { content: '', tool_calls: [{ id: '1', name: 'describe_instances', args: {} }] }, + { content: 'Could not read: AccessDenied' }, + ]); + const result = await runSubagent(SPEC, { model, tools: [boom], budget: BUDGET }); + + expect(result.status).toBe('done'); + expect(result.report).toContain('AccessDenied'); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/subagent.test.ts` +Expected: FAIL — cannot resolve `./subagent`. + +- [ ] **Step 3: Implement the sub-agent runtime** + +Create `apps/web-ui/lib/agent/subagent.ts`: + +```typescript +/** + * Sub-agent runtime — the Claude Code `Task` pattern. + * + * An ephemeral ReAct loop with its OWN message list. The orchestrator sees only + * the returned report, never the raw tool output, which is what keeps the + * orchestrator's context (and therefore its per-lap latency) flat. + * + * Sub-agents are strictly READ-ONLY. LangGraph cannot interrupt inside a tool + * call, so a mutation attempted here could never reach the guard node's human + * approval gate. Mutations stay on the orchestrator's guarded path. + * + * Not checkpointed: a sub-agent is not resumable, so a checkpointer would cost a + * Postgres write per lap for no benefit. + */ +import { AIMessage, HumanMessage, SystemMessage, ToolMessage } from '@langchain/core/messages'; +import { classifyTool } from './tool-classifier'; +import { contentToText, truncateOutput } from './agent-shared'; +import type { SubagentBudgetConfig } from './subagent-budget'; + +/** ~1500 tokens. Enforced in characters so the bound is deterministic and testable. */ +export const SUBAGENT_REPORT_MAX_CHARS = 6000; + +/** Per-tool output cap inside a sub-agent — it never leaves this context. */ +const SUBAGENT_TOOL_OUTPUT_MAX_CHARS = 4000; + +/** + * Never available to a sub-agent regardless of what the classifier says. + * - dispatch_agent: depth cap of 1. Recursion makes cost and latency unbounded. + * - ask_user: pauses for a human, and no human is reachable inside a tool call. + */ +const SUBAGENT_TOOL_DENYLIST = new Set(['dispatch_agent', 'ask_user']); + +export interface SubagentSpec { + role: string; + task: string; + expectedOutput: string; +} + +export interface SubagentTranscriptEntry { + kind: 'ai' | 'tool'; + name?: string; + text: string; +} + +export interface SubagentResult { + report: string; + toolCount: number; + tokensIn: number; + tokensOut: number; + status: 'done' | 'failed'; + transcript: SubagentTranscriptEntry[]; +} + +interface SubagentToolLike { + name: string; + invoke: (args: Record) => Promise; +} + +export interface SubagentDeps { + model: { bindTools?: (tools: unknown[]) => unknown; invoke: (messages: unknown[]) => Promise }; + tools: SubagentToolLike[]; + budget: SubagentBudgetConfig; + onEvent?: (progress: { toolCount: number; tokensIn: number; tokensOut: number }) => void; +} + +/** + * The jail. Fail-closed by design: `classifyTool` returns + * `{ isMutative: false, matchedRule: false }` for a tool whose name matched no + * rule at all. In the orchestrator that ambiguity routes to a human; inside a + * sub-agent there is no human, so an unverified tool is refused. + */ +export function isReadOnlyForSubagent( + name: string, + args?: Record, +): { allowed: boolean; reason: string } { + if (SUBAGENT_TOOL_DENYLIST.has(name)) { + return { allowed: false, reason: `${name} is not available to sub-agents` }; + } + + let classification; + try { + classification = classifyTool(name, args); + } catch (error) { + return { allowed: false, reason: `classifier error: ${error instanceof Error ? error.message : String(error)}` }; + } + + if (classification.isMutative) { + return { allowed: false, reason: `mutative call refused: ${classification.reason}` }; + } + if (!classification.matchedRule) { + return { allowed: false, reason: `${name} is not on the verified read-only list` }; + } + return { allowed: true, reason: classification.reason }; +} + +/** Drop everything a sub-agent may not call, before the model ever sees it. */ +export function filterReadOnlyTools(tools: T[]): T[] { + return tools.filter(tool => { + if (SUBAGENT_TOOL_DENYLIST.has(tool.name)) return false; + // Bash-like tools are judged per call (the command string decides), so keep + // them in the list and let the runtime check gate each invocation. + if (tool.name === 'execute_command') return true; + return isReadOnlyForSubagent(tool.name).allowed; + }); +} + +function buildSystemPrompt(spec: SubagentSpec): SystemMessage { + return new SystemMessage(`You are a focused read-only investigator working as a sub-agent for a cloud-operations platform. + +## Your role +${spec.role} + +## Your task +${spec.task} + +## What to return +${spec.expectedOutput} + +## Constraints + +- You are READ-ONLY. You cannot create, modify, delete, start, stop, or write anything. If the task appears to need a change, do NOT attempt it — describe the recommended change in your findings and the main agent will carry it out under human approval. +- You cannot ask the user questions. If something is ambiguous, investigate the most likely interpretation and say what you assumed. +- You see none of the parent conversation. Everything you need is in the task above. +- Batch independent read-only calls into a single turn — they run in parallel. + +## Output + +When you have gathered enough, reply with your findings as plain text. No preamble, no narration of what you did. Structure it as: + +**Findings** — what you established, with concrete evidence (resource ids, metric values, region and account names). +**Could not determine** — anything you could not establish, and why. Write "Nothing" if everything was resolved. + +Be dense. Your reply is consumed by another agent, not a human.`); +} + +function finishReport(text: string, note?: string): string { + const body = text.trim() || '(no findings produced)'; + const withNote = note ? `${body}\n\n[${note}]` : body; + return withNote.length > SUBAGENT_REPORT_MAX_CHARS + ? `${withNote.slice(0, SUBAGENT_REPORT_MAX_CHARS)}\n\n[TRUNCATED — report exceeded ${SUBAGENT_REPORT_MAX_CHARS} characters]` + : withNote; +} + +async function runSubagentLoop(spec: SubagentSpec, deps: SubagentDeps): Promise { + const { budget } = deps; + const allowedTools = filterReadOnlyTools(deps.tools); + const toolsByName = new Map(allowedTools.map(t => [t.name, t])); + + const boundModel = deps.model.bindTools + ? (deps.model.bindTools(allowedTools) as SubagentDeps['model']) + : deps.model; + + const messages: unknown[] = [buildSystemPrompt(spec), new HumanMessage({ content: spec.task })]; + const transcript: SubagentTranscriptEntry[] = []; + + let toolCount = 0; + let tokensIn = 0; + let tokensOut = 0; + let lastText = ''; + + for (let iteration = 0; iteration < budget.subagentMaxIterations; iteration++) { + const response = (await boundModel.invoke(messages)) as { + content: unknown; + tool_calls?: Array<{ id?: string; name: string; args?: Record }>; + usage_metadata?: { input_tokens?: number; output_tokens?: number }; + }; + + tokensIn += response.usage_metadata?.input_tokens ?? 0; + tokensOut += response.usage_metadata?.output_tokens ?? 0; + + const text = contentToText(response.content); + if (text.trim()) { + lastText = text; + transcript.push({ kind: 'ai', text }); + } + + const toolCalls = response.tool_calls ?? []; + if (toolCalls.length === 0) { + deps.onEvent?.({ toolCount, tokensIn, tokensOut }); + return { report: finishReport(lastText), toolCount, tokensIn, tokensOut, status: 'done', transcript }; + } + + messages.push(new AIMessage({ content: text, tool_calls: toolCalls as never })); + + // Independent calls in one turn run concurrently — the same parallelism + // the orchestrator gets from ToolNode. + const results = await Promise.all(toolCalls.map(async call => { + const verdict = isReadOnlyForSubagent(call.name, call.args ?? {}); + if (!verdict.allowed) { + return { + call, + output: `REFUSED: ${verdict.reason}. You cannot mutate state. Report the recommended change in your findings; the main agent will execute it under human approval.`, + }; + } + + const tool = toolsByName.get(call.name); + if (!tool) { + return { call, output: `REFUSED: ${call.name} is not available to sub-agents.` }; + } + + try { + const raw = await tool.invoke(call.args ?? {}); + return { call, output: truncateOutput(typeof raw === 'string' ? raw : JSON.stringify(raw), SUBAGENT_TOOL_OUTPUT_MAX_CHARS) }; + } catch (error) { + // A failing tool is data, not a crash — the sub-agent should report it. + return { call, output: `ERROR: ${error instanceof Error ? error.message : String(error)}` }; + } + })); + + for (const { call, output } of results) { + if (!output.startsWith('REFUSED:')) toolCount++; + transcript.push({ kind: 'tool', name: call.name, text: output }); + messages.push(new ToolMessage({ content: output, tool_call_id: call.id ?? `${call.name}-${toolCount}` })); + } + + deps.onEvent?.({ toolCount, tokensIn, tokensOut }); + } + + return { + report: finishReport(lastText, `INCOMPLETE — the sub-agent reached its ${budget.subagentMaxIterations}-iteration limit. Findings above are partial.`), + toolCount, tokensIn, tokensOut, status: 'done', transcript, + }; +} + +/** + * Run one sub-agent. Never throws: every failure path returns a report string, + * because a sub-agent failure must not abort the orchestrator's run. + */ +export async function runSubagent(spec: SubagentSpec, deps: SubagentDeps): Promise { + const timeoutMs = deps.budget.subagentTimeoutMs; + let timer: ReturnType | undefined; + + const timeout = new Promise<'timeout'>(resolve => { + timer = setTimeout(() => resolve('timeout'), timeoutMs); + }); + + try { + const outcome = await Promise.race([runSubagentLoop(spec, deps), timeout]); + if (outcome === 'timeout') { + return { + report: finishReport('', `TIMED OUT after ${Math.round(timeoutMs / 1000)}s — no findings were returned in time.`), + toolCount: 0, tokensIn: 0, tokensOut: 0, status: 'failed', transcript: [], + }; + } + return outcome; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + console.error(`[Subagent] "${spec.role}" failed: ${message}`); + return { + report: finishReport('', `FAILED — ${message}`), + toolCount: 0, tokensIn: 0, tokensOut: 0, status: 'failed', transcript: [], + }; + } finally { + if (timer) clearTimeout(timer); + } +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/subagent.test.ts` +Expected: PASS, 17 tests. + +- [ ] **Step 5: Typecheck and commit** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +```bash +git add apps/web-ui/lib/agent/subagent.ts apps/web-ui/lib/agent/subagent.test.ts +git commit -m "feat(agent): read-only sub-agent runtime + +Ephemeral ReAct loop with its own context that returns one bounded findings +report. Fail-closed jail refuses mutative and unverified tools, plus +dispatch_agent (depth cap 1) and ask_user (no human inside a tool call), so +LangGraph's inability to interrupt mid-tool-call is never a safety hole." +``` + +--- + +### Task 8: `dispatch_agent` tool and orchestrator wiring + +**Files:** +- Create: `apps/web-ui/lib/agent/dispatch-agent-tool.ts` +- Create: `apps/web-ui/lib/agent/dispatch-agent-tool.test.ts` +- Modify: `apps/web-ui/lib/agent/model-factory.ts` (`AssembleToolsOptions`, `assembleTools`) +- Modify: `apps/web-ui/lib/agent/planning-agent.ts:226` (tool assembly), `:384-420` (executor prompt) +- Modify: `apps/web-ui/lib/agent/tool-classifier.ts` (allowlist entry) + +**Interfaces:** +- Consumes: `runSubagent`, `filterReadOnlyTools`, `SubagentSpec`, `SubagentResult` from Task 7; `SubagentBudgetConfig`, `resolveSubagentBudget` from Task 4; `Semaphore` from Task 3. +- Produces: + - `createRunBudgetLedger(budget: SubagentBudgetConfig): RunBudgetLedger` + - `interface RunBudgetLedger { tryReserve(): { ok: true } | { ok: false; reason: string }; recordSpend(tokensIn: number, tokensOut: number): void; semaphore: Semaphore }` + - `createDispatchAgentTool(deps: DispatchAgentDeps)` — returns a LangChain tool named `dispatch_agent`. + - `interface DispatchAgentDeps { model: unknown; subagentTools: Array<{ name: string; invoke: (a: Record) => Promise }>; ledger: RunBudgetLedger; budget: SubagentBudgetConfig; onSubagentEvent?: (e: SubagentEvent) => void }` + - `interface SubagentEvent { id: string; role: string; task: string; status: 'running' | 'done' | 'failed'; toolCount: number; tokensIn: number; tokensOut: number; summary?: string; transcript?: SubagentTranscriptEntry[] }` + +- [ ] **Step 1: Write the failing test** + +Create `apps/web-ui/lib/agent/dispatch-agent-tool.test.ts`: + +```typescript +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +vi.mock('./subagent', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, runSubagent: vi.fn() }; +}); + +import { runSubagent } from './subagent'; +import { createRunBudgetLedger, createDispatchAgentTool } from './dispatch-agent-tool'; + +const BUDGET = { + enabled: true, + maxConcurrentSubagents: 2, + maxSubagentsPerRun: 3, + maxSubagentTokensPerRun: 1000, + subagentMaxIterations: 4, + subagentTimeoutMs: 5000, +}; + +const okResult = { + report: 'Findings: i-123 idle', toolCount: 2, tokensIn: 300, tokensOut: 50, + status: 'done' as const, transcript: [], +}; + +beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(runSubagent).mockResolvedValue(okResult); +}); + +describe('createRunBudgetLedger', () => { + it('allows reservations up to maxSubagentsPerRun', () => { + const ledger = createRunBudgetLedger(BUDGET); + expect(ledger.tryReserve().ok).toBe(true); + expect(ledger.tryReserve().ok).toBe(true); + expect(ledger.tryReserve().ok).toBe(true); + + const fourth = ledger.tryReserve(); + expect(fourth.ok).toBe(false); + expect((fourth as { reason: string }).reason).toMatch(/per-run sub-agent limit/i); + }); + + it('refuses once the token budget is spent', () => { + const ledger = createRunBudgetLedger(BUDGET); + ledger.recordSpend(900, 200); + + const verdict = ledger.tryReserve(); + expect(verdict.ok).toBe(false); + expect((verdict as { reason: string }).reason).toMatch(/token budget/i); + }); + + it('refuses everything when the budget is disabled', () => { + const ledger = createRunBudgetLedger({ ...BUDGET, enabled: false }); + expect(ledger.tryReserve().ok).toBe(false); + }); +}); + +describe('dispatch_agent tool', () => { + const makeTool = (budget = BUDGET) => createDispatchAgentTool({ + model: {}, + subagentTools: [{ name: 'describe_instances', invoke: async () => 'ok' }], + ledger: createRunBudgetLedger(budget), + budget, + }); + + it('returns the sub-agent report', async () => { + const result = await makeTool().invoke({ + role: 'EC2 auditor', task: 'audit account 1', expectedOutput: 'idle instances', + }); + expect(result).toContain('i-123 idle'); + }); + + it('returns ONLY the report — raw sub-agent tool output never reaches the orchestrator', async () => { + // The tool's return value becomes a ToolMessage in the orchestrator's + // message list. If the transcript leaked into it, context isolation — the + // entire reason sub-agents exist — would be defeated. + vi.mocked(runSubagent).mockResolvedValue({ + ...okResult, + transcript: [ + { kind: 'tool', name: 'describe_instances', text: 'RAW_TOOL_DUMP_MARKER' }, + { kind: 'ai', text: 'INTERNAL_REASONING_MARKER' }, + ], + }); + + const result = await makeTool().invoke({ role: 'a', task: 't', expectedOutput: 'e' }); + + expect(result).toBe(okResult.report); + expect(result).not.toContain('RAW_TOOL_DUMP_MARKER'); + expect(result).not.toContain('INTERNAL_REASONING_MARKER'); + }); + + it('degrades gracefully instead of throwing when the budget is exhausted', async () => { + const tool = makeTool({ ...BUDGET, maxSubagentsPerRun: 1 }); + await tool.invoke({ role: 'a', task: 't', expectedOutput: 'e' }); + + const second = await tool.invoke({ role: 'b', task: 't', expectedOutput: 'e' }); + expect(second).toMatch(/perform this work yourself/i); + expect(runSubagent).toHaveBeenCalledTimes(1); + }); + + it('emits running then done events', async () => { + const events: Array<{ status: string }> = []; + const budget = BUDGET; + const tool = createDispatchAgentTool({ + model: {}, + subagentTools: [], + ledger: createRunBudgetLedger(budget), + budget, + onSubagentEvent: e => events.push({ status: e.status }), + }); + + await tool.invoke({ role: 'a', task: 't', expectedOutput: 'e' }); + + expect(events[0].status).toBe('running'); + expect(events[events.length - 1].status).toBe('done'); + }); + + it('never throws when the sub-agent runtime rejects', async () => { + vi.mocked(runSubagent).mockRejectedValue(new Error('unexpected')); + const result = await makeTool().invoke({ role: 'a', task: 't', expectedOutput: 'e' }); + expect(result).toMatch(/unexpected/); + }); + + it('bounds concurrency to maxConcurrentSubagents', async () => { + let active = 0; + let peak = 0; + vi.mocked(runSubagent).mockImplementation(async () => { + active++; peak = Math.max(peak, active); + await new Promise(r => setTimeout(r, 10)); + active--; + return okResult; + }); + + const budget = { ...BUDGET, maxConcurrentSubagents: 2, maxSubagentsPerRun: 6 }; + const tool = makeTool(budget); + await Promise.all(Array.from({ length: 5 }, (_, i) => + tool.invoke({ role: `r${i}`, task: 't', expectedOutput: 'e' }))); + + expect(peak).toBeLessThanOrEqual(2); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/dispatch-agent-tool.test.ts` +Expected: FAIL — cannot resolve `./dispatch-agent-tool`. + +- [ ] **Step 3: Implement the ledger and the tool** + +Create `apps/web-ui/lib/agent/dispatch-agent-tool.ts`: + +```typescript +/** + * dispatch_agent — the sub-agent fan-out tool. + * + * One sub-agent per tool call, deliberately: the orchestrator emits N calls in a + * single turn and ToolNode's existing concurrency performs the fan-out, so there + * is no second concurrency mechanism and each sub-agent gets its own tool card + * in the UI for free. + * + * Budget exhaustion NEVER fails the run. The tool returns an instruction to do + * the work serially, so behaviour degrades to the pre-sub-agent baseline. + */ +import { tool } from '@langchain/core/tools'; +import { z } from 'zod'; +import { randomUUID } from 'crypto'; +import { Semaphore } from './concurrency'; +import { runSubagent, type SubagentTranscriptEntry } from './subagent'; +import type { SubagentBudgetConfig } from './subagent-budget'; + +export interface SubagentEvent { + id: string; + role: string; + task: string; + status: 'running' | 'done' | 'failed'; + toolCount: number; + tokensIn: number; + tokensOut: number; + summary?: string; + transcript?: SubagentTranscriptEntry[]; +} + +export interface RunBudgetLedger { + tryReserve(): { ok: true } | { ok: false; reason: string }; + recordSpend(tokensIn: number, tokensOut: number): void; + semaphore: Semaphore; +} + +/** + * Per-run ledger. Runs execute on a single ECS replica, so in-process counters + * are sufficient — no distributed coordination needed. + */ +export function createRunBudgetLedger(budget: SubagentBudgetConfig): RunBudgetLedger { + let dispatched = 0; + let tokensSpent = 0; + const semaphore = new Semaphore(budget.maxConcurrentSubagents); + + return { + tryReserve() { + if (!budget.enabled) { + return { ok: false, reason: 'sub-agents are disabled for this organization' }; + } + if (dispatched >= budget.maxSubagentsPerRun) { + return { ok: false, reason: `per-run sub-agent limit reached (${budget.maxSubagentsPerRun})` }; + } + if (tokensSpent >= budget.maxSubagentTokensPerRun) { + return { ok: false, reason: `sub-agent token budget exhausted (${budget.maxSubagentTokensPerRun})` }; + } + dispatched++; + return { ok: true }; + }, + recordSpend(tokensIn: number, tokensOut: number) { + tokensSpent += tokensIn + tokensOut; + }, + semaphore, + }; +} + +export interface DispatchAgentDeps { + model: unknown; + subagentTools: Array<{ name: string; invoke: (args: Record) => Promise }>; + ledger: RunBudgetLedger; + budget: SubagentBudgetConfig; + onSubagentEvent?: (event: SubagentEvent) => void; +} + +const DEGRADE_PREFIX = 'Sub-agent budget exhausted'; + +export function createDispatchAgentTool(deps: DispatchAgentDeps) { + return tool( + async ({ role, task, expectedOutput }: { role: string; task: string; expectedOutput: string }) => { + const reservation = deps.ledger.tryReserve(); + if (!reservation.ok) { + return `${DEGRADE_PREFIX}: ${reservation.reason}. Perform this work yourself, serially, using your own tools.`; + } + + const id = randomUUID(); + const emit = (event: Partial & Pick) => + deps.onSubagentEvent?.({ + id, role, task, toolCount: 0, tokensIn: 0, tokensOut: 0, ...event, + }); + + emit({ status: 'running' }); + + try { + const result = await deps.ledger.semaphore.run(() => runSubagent( + { role, task, expectedOutput }, + { + model: deps.model as never, + tools: deps.subagentTools, + budget: deps.budget, + onEvent: progress => emit({ status: 'running', ...progress }), + }, + )); + + deps.ledger.recordSpend(result.tokensIn, result.tokensOut); + + emit({ + status: result.status === 'failed' ? 'failed' : 'done', + toolCount: result.toolCount, + tokensIn: result.tokensIn, + tokensOut: result.tokensOut, + summary: result.report, + transcript: result.transcript, + }); + + return result.report; + } catch (error) { + // runSubagent is already total, but a semaphore or emit failure must + // not propagate into ToolNode and abort the orchestrator's turn. + const message = error instanceof Error ? error.message : String(error); + console.error(`[dispatch_agent] "${role}" failed: ${message}`); + emit({ status: 'failed', summary: message }); + return `The sub-agent "${role}" failed: ${message}. Continue with the information you already have, or perform this work yourself.`; + } + }, + { + name: 'dispatch_agent', + description: `Delegate an INDEPENDENT read-only investigation to a sub-agent that works in its own context and returns a compressed findings report. + +Use this when the task splits into parts that do not depend on each other — one account each, one region each, one service each. Emit SEVERAL dispatch_agent calls in a SINGLE turn and they run in parallel; that is the entire point. + +Do NOT use it for: +- anything that changes state (sub-agents are read-only — do mutations yourself) +- work that depends on another sub-agent's output (they cannot see each other) +- a single quick lookup you could do in one tool call + +CRITICAL: "task" must be completely self-contained. The sub-agent sees NONE of this conversation — no account ids, no prior findings, no user context unless you write them into the task. A vague brief returns a useless report.`, + schema: z.object({ + role: z.string().describe('Short identity for this sub-agent, e.g. "EC2 idle-resource auditor for account 123456789012"'), + task: z.string().describe('Complete standalone brief: what to investigate, which account ids and regions, which tools to prefer, and any constraints. Assume zero shared context.'), + expectedOutput: z.string().describe('Exactly what the report must contain, e.g. "instance ids with average CPU below 5% over 14 days, with the metric value for each"'), + }), + }, + ); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run lib/agent/dispatch-agent-tool.test.ts` +Expected: PASS, 9 tests. + +- [ ] **Step 5: Mark `dispatch_agent` read-only for the orchestrator's guard** + +Without this the guard classifies `dispatch_agent` as mutative (it matches `/\bdispatch\b/`? no — but it is unknown-named, so it goes to the LLM risk batch on every call, adding a round-trip). Add it to the allowlist in `apps/web-ui/lib/agent/tool-classifier.ts`, in `READ_ONLY_ALLOWLIST` next to `'read_file'`: + +```typescript + 'dispatch_agent', // spawns a read-only sub-agent; the sub-agent's own jail enforces that +``` + +- [ ] **Step 6: Assemble the tool into the orchestrator** + +In `apps/web-ui/lib/agent/model-factory.ts`, extend `AssembleToolsOptions`: + +```typescript + /** When set, adds the dispatch_agent fan-out tool built from these deps. */ + dispatchAgentTool?: unknown; +``` + +and in `assembleTools`, add it to `customTools` after `askUserTool`: + +```typescript + ...(options.dispatchAgentTool ? [options.dispatchAgentTool as never] : []), +``` + +In `apps/web-ui/lib/agent/planning-agent.ts`, add imports: + +```typescript +import { resolveSubagentBudget } from "./subagent-budget"; +import { createRunBudgetLedger, createDispatchAgentTool } from "./dispatch-agent-tool"; +import { filterReadOnlyTools } from "./subagent"; +``` + +Then replace the tool-assembly block at line 226 with: + +```typescript + // Sub-agent tools are assembled FIRST and without dispatch_agent, so the + // sub-agents' tool list can never contain the tool that spawns sub-agents. + const baseTools = await assembleTools({ includeS3Tools: true, includeMemoryTools: false, includeSkillTool: autoLoadSkills, userId: config.userId, mcpServerIds, tenantId, accounts, knowledgeBaseIds: config.knowledgeBaseIds }); + + const subagentBudget = tenantId + ? await resolveSubagentBudget(tenantId) + : { enabled: false, maxConcurrentSubagents: 1, maxSubagentsPerRun: 1, maxSubagentTokensPerRun: 0, subagentMaxIterations: 2, subagentTimeoutMs: 30_000 }; + + const dispatchAgentTool = subagentBudget.enabled + ? createDispatchAgentTool({ + model, + subagentTools: filterReadOnlyTools(baseTools as never[]), + ledger: createRunBudgetLedger(subagentBudget), + budget: subagentBudget, + onSubagentEvent: config.onSubagentEvent, + }) + : null; + + const tools = dispatchAgentTool ? [...baseTools, dispatchAgentTool] : baseTools; + console.log(`[PlanningAgent] Sub-agents ${subagentBudget.enabled ? `enabled (concurrency=${subagentBudget.maxConcurrentSubagents})` : 'disabled'}`); + const modelWithTools = model.bindTools!(tools); + const toolNode = new ToolNode(tools); +``` + +Add the callback to `GraphConfig` in `apps/web-ui/lib/agent/agent-shared.ts` (after `autoLoadSkills`, line 859): + +```typescript + /** Live sub-agent progress sink. Set by the chat route so dispatch_agent + * activity can be streamed as data-subagent parts. */ + onSubagentEvent?: (event: import('./dispatch-agent-tool').SubagentEvent) => void; +``` + +- [ ] **Step 7: Teach the executor when to fan out** + +In `apps/web-ui/lib/agent/planning-agent.ts`, append to the `## Execution Discipline` block in `generateNode` (after the batching rules added in Task 3): + +``` +## Delegation + +- When the current step splits into INDEPENDENT investigations — one per account, region, or service — emit several dispatch_agent calls in THIS turn. They run in parallel and each returns a compressed report. +- Write each sub-agent brief so it stands completely alone: it sees none of this conversation, so spell out account ids, regions, time windows, and what the report must contain. +- Do NOT delegate work that changes state — sub-agents are read-only. Do NOT delegate a single quick lookup. Do NOT chain sub-agents; they cannot see each other's results. +- If dispatch_agent replies that the budget is exhausted, simply do that work yourself with your own tools. +``` + +- [ ] **Step 8: Verify typecheck and the full suite** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +Run: `cd apps/web-ui && bunx vitest run lib/agent/` +Expected: PASS with no new failures. + +- [ ] **Step 9: Commit** + +```bash +git add apps/web-ui/lib/agent/dispatch-agent-tool.ts apps/web-ui/lib/agent/dispatch-agent-tool.test.ts apps/web-ui/lib/agent/model-factory.ts apps/web-ui/lib/agent/planning-agent.ts apps/web-ui/lib/agent/agent-shared.ts apps/web-ui/lib/agent/tool-classifier.ts +git commit -m "feat(agent): dispatch_agent fan-out tool with per-run budget ledger + +One sub-agent per tool call so ToolNode's existing concurrency performs the +fan-out. Budget exhaustion returns a do-it-yourself instruction rather than an +error, so runs degrade to the serial baseline instead of failing." +``` + +--- + +### Task 9: Stream sub-agent progress with a heartbeat + +CloudFront's `originReadTimeout` is 60s (`infra/compute/index.ts:962`) while the ALB allows 1200s. Today a 10-minute run survives only because streamed tokens keep resetting the 60s timer. A silent fan-out phase would drop the connection, so the heartbeat is a correctness requirement, not decoration. + +**Files:** +- Modify: `apps/web-ui/app/api/chat/stream-parts.ts` +- Modify: `apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts` +- Modify: `apps/web-ui/app/api/chat/route.ts` + +**Interfaces:** +- Consumes: `SubagentEvent` from Task 8. +- Produces: `buildSubagentPart(event: SubagentEvent): DataPart` emitting `{ type: 'data-subagent', id: 'subagent-', data: {...} }`. + +- [ ] **Step 1: Write the failing test** + +Append to `apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts`: + +```typescript +import { buildSubagentPart } from '../stream-parts'; + +describe('buildSubagentPart', () => { + const base = { + id: 'sa-1', role: 'EC2 auditor', task: 'audit account 1', + toolCount: 3, tokensIn: 900, tokensOut: 120, + }; + + it('builds a stable id so updates replace rather than append', () => { + const part = buildSubagentPart({ ...base, status: 'running' }); + expect(part.type).toBe('data-subagent'); + expect(part.id).toBe('subagent-sa-1'); + }); + + it('carries progress counters', () => { + const part = buildSubagentPart({ ...base, status: 'running' }); + expect(part.data).toMatchObject({ role: 'EC2 auditor', status: 'running', toolCount: 3, tokensIn: 900 }); + }); + + it('omits the transcript from the stream payload', () => { + const part = buildSubagentPart({ + ...base, status: 'done', summary: 'found things', + transcript: [{ kind: 'ai', text: 'internal reasoning' }], + }); + expect(JSON.stringify(part.data)).not.toContain('internal reasoning'); + expect((part.data as { summary?: string }).summary).toBe('found things'); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run app/api/chat/__tests__/stream-parts.test.ts` +Expected: FAIL — `buildSubagentPart` is not exported. + +- [ ] **Step 3: Add the stream part** + +In `apps/web-ui/app/api/chat/stream-parts.ts`, add after `buildMemoryPart` (line 28): + +```typescript +/** + * One data-subagent part per sub-agent state change. The `id` is stable per + * sub-agent so the client replaces the card in place rather than appending a new + * one on every progress tick. + * + * The transcript is deliberately NOT sent: it can be large, and the whole point + * of sub-agents is keeping bulk out of the transcript. It is persisted + * separately and fetched on demand when a card is expanded. + */ +export function buildSubagentPart(event: { + id: string; + role: string; + task: string; + status: 'running' | 'done' | 'failed'; + toolCount: number; + tokensIn: number; + tokensOut: number; + summary?: string; +}): DataPart { + return { + type: 'data-subagent', + id: `subagent-${event.id}`, + data: { + id: event.id, + role: event.role, + task: event.task, + status: event.status, + toolCount: event.toolCount, + tokensIn: event.tokensIn, + tokensOut: event.tokensOut, + ...(event.summary ? { summary: event.summary } : {}), + }, + }; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run app/api/chat/__tests__/stream-parts.test.ts` +Expected: PASS. + +- [ ] **Step 5: Wire the sink and the heartbeat into the chat route** + +In `apps/web-ui/app/api/chat/route.ts`, add the imports: + +```typescript +import { buildSubagentPart } from './stream-parts'; +import type { SubagentEvent } from '@/lib/agent/dispatch-agent-tool'; +``` + +(`stream-parts` is already imported — extend the existing import list rather than adding a second statement.) + +Inside the `ReadableStream` `start(controller)` body, before the graph stream is created, add the live sink and heartbeat: + +```typescript + // --- Sub-agent progress --------------------------------------------- + // Sub-agent tokens deliberately never reach the transcript: three + // concurrent streams interleaved into one column are unreadable, and the + // narration was never the deliverable. Collapsed cards instead. + const liveSubagents = new Map(); + + const emitSubagent = (event: SubagentEvent) => { + liveSubagents.set(event.id, event); + if (event.status !== 'running') liveSubagents.delete(event.id); + try { + controller.enqueue(buildSubagentPart(event)); + } catch { + // Client disconnected — the stream teardown handles it. + } + }; + + // CloudFront's originReadTimeout is 60s (infra/compute/index.ts:962). A + // sub-agent can think for 90s without producing a byte, so re-emit the + // in-flight cards on a timer to keep the connection alive. + const HEARTBEAT_MS = 15_000; + const heartbeat = setInterval(() => { + for (const event of liveSubagents.values()) { + try { + controller.enqueue(buildSubagentPart(event)); + } catch { + // ignore — teardown will clear the interval + } + } + }, HEARTBEAT_MS); +``` + +Pass `emitSubagent` into the graph config where `GraphConfig` is assembled (line 230, alongside `autoApprove`): + +```typescript + onSubagentEvent: emitSubagent, +``` + +Clear the interval in the same teardown path where `logRunSummary(threadId)` was added in Task 2: + +```typescript + clearInterval(heartbeat); +``` + +- [ ] **Step 6: Verify the heartbeat manually** + +With `SUBAGENTS_ENABLED=true` and sub-agents enabled for the tenant, run a fan-out task and watch the browser network tab: `data-subagent` frames must arrive at least every 15 seconds during the fan-out phase, with no gap approaching 60 seconds. + +- [ ] **Step 7: Typecheck and commit** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +```bash +git add apps/web-ui/app/api/chat/stream-parts.ts apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts apps/web-ui/app/api/chat/route.ts +git commit -m "feat(chat): stream sub-agent progress as data-subagent parts + +Collapsed-card progress instead of interleaved token streams, plus a 15s +heartbeat so a silent fan-out phase cannot exceed CloudFront's 60s +originReadTimeout and drop the connection." +``` + +--- + +### Task 10: Sub-agent cards in the chat UI + +**Files:** +- Modify: `apps/web-ui/components/agent/chat/run-state.ts` +- Modify: `apps/web-ui/components/agent/chat/__tests__/run-state.test.ts` (create if absent) +- Create: `apps/web-ui/components/agent/chat/subagent-card.tsx` +- Modify: `apps/web-ui/components/agent/chat/run-rail.tsx` + +**Interfaces:** +- Consumes: the `data-subagent` part from Task 9. +- Produces: + - `interface SubagentState { id: string; role: string; task: string; status: 'running' | 'done' | 'failed'; toolCount: number; tokensIn: number; tokensOut: number; summary?: string }` + - `RunState.subagents: SubagentState[]` + - `` + +- [ ] **Step 1: Write the failing reducer test** + +Create `apps/web-ui/components/agent/chat/__tests__/run-state.test.ts` (or append to it if it exists): + +```typescript +import { describe, it, expect } from 'vitest'; +import { deriveRunState } from '../run-state'; + +const msg = (parts: unknown[]) => ({ role: 'assistant', id: 'm1', parts } as never); + +const subagentPart = (data: Record) => ({ type: 'data-subagent', data }); + +describe('deriveRunState — subagents', () => { + it('collects sub-agents in first-seen order', () => { + const state = deriveRunState([msg([ + subagentPart({ id: 'a', role: 'A', task: 't', status: 'running', toolCount: 0, tokensIn: 0, tokensOut: 0 }), + subagentPart({ id: 'b', role: 'B', task: 't', status: 'running', toolCount: 0, tokensIn: 0, tokensOut: 0 }), + ])], new Set()); + + expect(state.subagents.map(s => s.id)).toEqual(['a', 'b']); + }); + + it('replaces an earlier update for the same id rather than duplicating', () => { + const state = deriveRunState([msg([ + subagentPart({ id: 'a', role: 'A', task: 't', status: 'running', toolCount: 1, tokensIn: 10, tokensOut: 2 }), + subagentPart({ id: 'a', role: 'A', task: 't', status: 'done', toolCount: 5, tokensIn: 900, tokensOut: 80, summary: 'found it' }), + ])], new Set()); + + expect(state.subagents).toHaveLength(1); + expect(state.subagents[0]).toMatchObject({ status: 'done', toolCount: 5, summary: 'found it' }); + }); + + it('marks the run as having structured data', () => { + const state = deriveRunState([msg([ + subagentPart({ id: 'a', role: 'A', task: 't', status: 'running', toolCount: 0, tokensIn: 0, tokensOut: 0 }), + ])], new Set()); + + expect(state.hasStructuredData).toBe(true); + }); + + it('returns an empty list when no sub-agent parts are present', () => { + expect(deriveRunState([msg([{ type: 'text', text: 'hi' }])], new Set()).subagents).toEqual([]); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run components/agent/chat/__tests__/run-state.test.ts` +Expected: FAIL — `state.subagents` is undefined. + +- [ ] **Step 3: Extend the reducer** + +In `apps/web-ui/components/agent/chat/run-state.ts`, add the interface after `PendingClarification` (line 14): + +```typescript +export interface SubagentState { + id: string; + role: string; + task: string; + status: 'running' | 'done' | 'failed'; + toolCount: number; + tokensIn: number; + tokensOut: number; + summary?: string; +} +``` + +Add to `RunState` (after `tokenUsage`, line 28): + +```typescript + /** Sub-agents dispatched in this thread, in first-seen order. Later parts for + * the same id replace the earlier entry in place. */ + subagents: SubagentState[]; +``` + +Inside `deriveRunState`, declare the accumulator next to the other locals (after line 46): + +```typescript + const subagents = new Map(); +``` + +Add the case to the `switch` (after `case 'data-usage'`, line 103): + +```typescript + case 'data-subagent': { + hasStructuredData = true; + const id = String(part.data?.id ?? ''); + if (!id) break; + // Map.set on an existing key preserves insertion order, so the + // card stays put as it updates from running → done. + subagents.set(id, { + id, + role: String(part.data?.role ?? ''), + task: String(part.data?.task ?? ''), + status: (part.data?.status === 'done' || part.data?.status === 'failed') + ? part.data.status : 'running', + toolCount: Number(part.data?.toolCount) || 0, + tokensIn: Number(part.data?.tokensIn) || 0, + tokensOut: Number(part.data?.tokensOut) || 0, + ...(part.data?.summary ? { summary: String(part.data.summary) } : {}), + }); + break; + } +``` + +Add to the returned object (after `tokenUsage`, line 124): + +```typescript + subagents: Array.from(subagents.values()), +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run components/agent/chat/__tests__/run-state.test.ts` +Expected: PASS, 4 tests. + +- [ ] **Step 5: Build the card** + +Create `apps/web-ui/components/agent/chat/subagent-card.tsx`: + +```tsx +'use client'; + +import { useState } from 'react'; +import { Bot, CheckCircle2, ChevronDown, ChevronRight, Loader2, XCircle } from 'lucide-react'; + +import { cn } from '@/lib/utils'; +import type { SubagentState } from './run-state'; + +function formatTokens(n: number): string { + return n >= 1000 ? `${(n / 1000).toFixed(1)}k` : String(n); +} + +const STATUS_ICON = { + running: , + done: , + failed: , +} as const; + +export function SubagentCard({ subagent }: { subagent: SubagentState }) { + const [expanded, setExpanded] = useState(false); + const canExpand = subagent.status !== 'running' && !!subagent.summary; + + return ( +
+ + + {expanded && subagent.summary && ( +
+

Task

+

{subagent.task}

+

Findings

+

{subagent.summary}

+
+ )} +
+ ); +} +``` + +- [ ] **Step 6: Fix the existing RunRail test fixture** + +`apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx` builds a complete `RunState` literal (`EMPTY_RUN_STATE`, line 7). Adding a required field to `RunState` breaks its typecheck, so update it now — add after `tokenUsage` (line 15): + +```tsx + subagents: [], +``` + +Run: `cd apps/web-ui && bunx vitest run components/agent/chat/__tests__/run-rail.test.tsx` +Expected: PASS (unchanged behaviour, fixture now complete). + +- [ ] **Step 7: Render the cards in the run rail** + +`RunRail` currently takes `{ runState, isStreaming, context }` (line 45) and has no thread id, so add one. In `apps/web-ui/components/agent/chat/run-rail.tsx`: + +Add to the imports: + +```tsx +import { Bot } from "lucide-react"; +import { SubagentCard } from "./subagent-card"; +``` + +(`lucide-react` is already imported — add `Bot` to the existing named import list rather than adding a second statement.) + +Extend the props: + +```tsx +export function RunRail({ + runState, + isStreaming, + context, + threadId, +}: { + runState: RunState; + isStreaming: boolean; + context: { accountNames: string[]; modelLabel: string; skillName: string | null; toolCount: number | null; kbLabel: string }; + /** Needed to fetch a persisted sub-agent transcript when a card is expanded + * after a reload. Optional so the existing rail tests keep compiling. */ + threadId?: string; +}) { +``` + +Render the group using the file's own `RailSection` wrapper (line 33), placed immediately after the plan `RailSection` block that ends at line 102's closing `)}`: + +```tsx + {runState.subagents.length > 0 && ( + s.status === "running").length} running)`} + > +
+ {runState.subagents.map((subagent) => ( + + ))} +
+ + )} +``` + +- [ ] **Step 8: Pass the thread id from the session view** + +In `apps/web-ui/components/agent/workspace/session-view.tsx:372`, add the prop: + +```tsx + +``` + +`threadId` is already a `SessionView` prop (line 45), so nothing else needs threading. + +- [ ] **Step 9: Typecheck, lint, and verify visually** + +Run: `cd apps/web-ui && bunx tsc --noEmit` +Expected: no new errors. + +Run: `cd apps/web-ui && bun run lint` +Expected: no new errors. + +Run: `cd apps/web-ui && bunx vitest run components/agent/chat/` +Expected: PASS. + +With sub-agents enabled, run a fan-out task and confirm: one card per sub-agent, spinners while running, live tool/token counters, green ticks on completion, cards expand to show task and findings, and no sub-agent prose leaks into the transcript. + +- [ ] **Step 10: Commit** + +```bash +git add apps/web-ui/components/agent/chat/run-state.ts apps/web-ui/components/agent/chat/__tests__/ apps/web-ui/components/agent/chat/subagent-card.tsx apps/web-ui/components/agent/chat/run-rail.tsx apps/web-ui/components/agent/workspace/session-view.tsx +git commit -m "feat(ui): collapsed sub-agent cards in the chat run rail" +``` + +--- + +### Task 11: Persist sub-agent transcripts for expansion after reload + +Stream data parts are not persisted — chat history is rebuilt from the LangGraph checkpointer. Live expansion works from client state, but a page reload loses it. The transcript must never travel in the `dispatch_agent` ToolMessage, because that message enters the orchestrator's context and would defeat the isolation the whole feature exists to provide. + +**Files:** +- Modify: `libs/prisma/schema.prisma` +- Create: `apps/web-ui/lib/db/repositories/subagent/interface.ts` +- Create: `apps/web-ui/lib/db/repositories/subagent/postgres.ts` +- Modify: `apps/web-ui/lib/db/repository-factory.ts` +- Create: `apps/web-ui/app/api/chat/subagents/[threadId]/route.ts` +- Create: `apps/web-ui/app/api/chat/subagents/[threadId]/route.test.ts` +- Modify: `apps/web-ui/app/api/chat/route.ts` (persist on terminal events) +- Modify: `apps/web-ui/components/agent/chat/subagent-card.tsx` (fetch on expand) + +**Interfaces:** +- Consumes: `SubagentEvent` from Task 8; `emitSubagent` sink from Task 9. +- Produces: + - `interface SubagentRunRepository { save(record: SubagentRunRecord): Promise; listByThread(tenantId: string, threadId: string): Promise }` + - `getSubagentRunRepository(): SubagentRunRepository` + - `GET /api/chat/subagents/[threadId]` → `{ success: true, data: SubagentRunRecord[] }` + +- [ ] **Step 1: Add the Prisma model** + +In `libs/prisma/schema.prisma`, add next to `ChatMessage` (line 615): + +```prisma +// AgentSubagentRun — one row per dispatch_agent sub-agent, so a collapsed card +// can be expanded after a page reload. The transcript lives here and NEVER in +// the orchestrator's message list: putting it there would defeat the context +// isolation that makes sub-agents worthwhile. +model AgentSubagentRun { + id String @id @default(cuid()) + tenantId String + threadId String + subagentId String + role String + task String + status String // running|done|failed + toolCount Int @default(0) + tokensIn Int @default(0) + tokensOut Int @default(0) + summary String? + transcript Json? + createdAt DateTime @default(now()) + expiresAt DateTime // 30-day TTL, matching ChatMessage + + @@unique([tenantId, threadId, subagentId]) + @@index([tenantId, threadId, createdAt]) + @@index([expiresAt]) + @@map("agent_subagent_runs") +} +``` + +- [ ] **Step 2: Generate the migration and audit it** + +Run: `cd apps/web-ui && bun run db:migrate` +When prompted, name it `add_agent_subagent_runs`. + +**Then open the generated SQL under `libs/prisma/migrations/` and read every statement.** Per the project's recorded history, `prisma migrate dev` silently emits `DROP INDEX` for raw-SQL-created HNSW/GIN/IVFFlat indexes it treats as drift. If you find any `DROP INDEX` unrelated to this new table, delete those lines before the migration is applied anywhere else, and add a follow-up idempotent migration recreating anything already dropped locally. Never edit a migration that has already been applied in a shared environment. + +Run: `cd apps/web-ui && bun run db:generate && cd ../workers && bun run db:generate` +Expected: both clients regenerate without error. + +- [ ] **Step 3: Write the failing route test** + +Create `apps/web-ui/app/api/chat/subagents/[threadId]/route.test.ts`: + +```typescript +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +vi.mock('@/lib/rbac/authorize', () => ({ authorize: vi.fn() })); +vi.mock('@/lib/auth-session', () => ({ getSessionTenantId: vi.fn() })); +vi.mock('@/lib/db/repository-factory', () => ({ + getSubagentRunRepository: vi.fn(), +})); + +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getSubagentRunRepository } from '@/lib/db/repository-factory'; +import { GET } from './route'; + +const params = (threadId: string) => ({ params: Promise.resolve({ threadId }) }); +const listByThread = vi.fn(); + +beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(authorize).mockResolvedValue(null as never); + vi.mocked(getSessionTenantId).mockResolvedValue('t1' as never); + vi.mocked(getSubagentRunRepository).mockReturnValue({ save: vi.fn(), listByThread } as never); + listByThread.mockResolvedValue([{ subagentId: 'a', role: 'A', transcript: [] }]); +}); + +describe('GET /api/chat/subagents/[threadId]', () => { + it('returns the thread\'s sub-agent runs', async () => { + const body = await (await GET({} as never, params('thread-1'))).json(); + + expect(body.success).toBe(true); + expect(body.data).toHaveLength(1); + expect(listByThread).toHaveBeenCalledWith('t1', 'thread-1'); + }); + + it('propagates an RBAC denial', async () => { + const denied = { status: 403 }; + vi.mocked(authorize).mockResolvedValue(denied as never); + expect(await GET({} as never, params('thread-1'))).toBe(denied); + }); + + it('403s with no tenant context', async () => { + vi.mocked(getSessionTenantId).mockResolvedValue(null as never); + expect((await GET({} as never, params('thread-1'))).status).toBe(403); + }); + + it('500s when the repository throws', async () => { + listByThread.mockRejectedValue(new Error('db down')); + expect((await GET({} as never, params('thread-1'))).status).toBe(500); + }); +}); +``` + +- [ ] **Step 4: Run the test to verify it fails** + +Run: `cd apps/web-ui && bunx vitest run app/api/chat/subagents/` +Expected: FAIL — cannot resolve `./route`. + +- [ ] **Step 5: Implement the repository** + +Create `apps/web-ui/lib/db/repositories/subagent/interface.ts`: + +```typescript +export interface SubagentTranscriptEntry { + kind: 'ai' | 'tool'; + name?: string; + text: string; +} + +export interface SubagentRunRecord { + tenantId: string; + threadId: string; + subagentId: string; + role: string; + task: string; + status: string; + toolCount: number; + tokensIn: number; + tokensOut: number; + summary?: string | null; + transcript?: SubagentTranscriptEntry[] | null; +} + +export interface SubagentRunRepository { + /** Upsert by (tenantId, threadId, subagentId) — a sub-agent is written once + * on completion, but a retried write must not create a duplicate row. */ + save(record: SubagentRunRecord): Promise; + listByThread(tenantId: string, threadId: string): Promise; +} +``` + +Create `apps/web-ui/lib/db/repositories/subagent/postgres.ts`: + +```typescript +import { getTenantClient } from '@/lib/db/pg-config'; +import type { SubagentRunRecord, SubagentRunRepository, SubagentTranscriptEntry } from './interface'; + +const TTL_30_DAYS_MS = 30 * 24 * 60 * 60 * 1000; + +export class SubagentRunPostgresRepository implements SubagentRunRepository { + async save(record: SubagentRunRecord): Promise { + const db = getTenantClient(record.tenantId); + const data = { + tenantId: record.tenantId, + threadId: record.threadId, + subagentId: record.subagentId, + role: record.role, + task: record.task, + status: record.status, + toolCount: record.toolCount, + tokensIn: record.tokensIn, + tokensOut: record.tokensOut, + summary: record.summary ?? null, + transcript: (record.transcript ?? null) as never, + expiresAt: new Date(Date.now() + TTL_30_DAYS_MS), + }; + + await db.agentSubagentRun.upsert({ + where: { + tenantId_threadId_subagentId: { + tenantId: record.tenantId, + threadId: record.threadId, + subagentId: record.subagentId, + }, + }, + create: data, + update: data, + }); + } + + async listByThread(tenantId: string, threadId: string): Promise { + const db = getTenantClient(tenantId); + const rows = await db.agentSubagentRun.findMany({ + where: { threadId }, + orderBy: { createdAt: 'asc' }, + }); + + return rows.map(row => ({ + tenantId: row.tenantId, + threadId: row.threadId, + subagentId: row.subagentId, + role: row.role, + task: row.task, + status: row.status, + toolCount: row.toolCount, + tokensIn: row.tokensIn, + tokensOut: row.tokensOut, + summary: row.summary, + transcript: (row.transcript ?? null) as SubagentTranscriptEntry[] | null, + })); + } +} +``` + +In `apps/web-ui/lib/db/repository-factory.ts`, add the accessor following the existing pattern in that file: + +```typescript +import { SubagentRunPostgresRepository } from './repositories/subagent/postgres'; +import type { SubagentRunRepository } from './repositories/subagent/interface'; + +let subagentRunRepository: SubagentRunRepository | null = null; + +export function getSubagentRunRepository(): SubagentRunRepository { + if (!subagentRunRepository) subagentRunRepository = new SubagentRunPostgresRepository(); + return subagentRunRepository; +} +``` + +- [ ] **Step 6: Implement the route** + +Create `apps/web-ui/app/api/chat/subagents/[threadId]/route.ts`: + +```typescript +import { NextRequest, NextResponse } from 'next/server'; +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getSubagentRunRepository } from '@/lib/db/repository-factory'; + +export async function GET( + _request: NextRequest, + { params }: { params: Promise<{ threadId: string }> }, +) { + console.log('API - GET /api/chat/subagents/[threadId] - Fetching sub-agent runs'); + + const authError = await authorize('read', 'Agent'); + if (authError) return authError; + + try { + const tenantId = await getSessionTenantId(); + if (!tenantId) { + return NextResponse.json({ success: false, error: 'No tenant context' }, { status: 403 }); + } + + const { threadId } = await params; + // listByThread goes through getTenantClient, so the query is tenant-scoped + // regardless of what threadId the caller supplies. + const runs = await getSubagentRunRepository().listByThread(tenantId, threadId); + + return NextResponse.json({ success: true, data: runs }); + } catch (error) { + console.error('API - Error fetching sub-agent runs:', error); + return NextResponse.json( + { success: false, error: error instanceof Error ? error.message : 'Failed to fetch sub-agent runs' }, + { status: 500 }, + ); + } +} +``` + +- [ ] **Step 7: Run the route test to verify it passes** + +Run: `cd apps/web-ui && bunx vitest run app/api/chat/subagents/` +Expected: PASS, 4 tests. + +- [ ] **Step 8: Persist on terminal sub-agent events** + +In `apps/web-ui/app/api/chat/route.ts`, extend the `emitSubagent` sink added in Task 9 so terminal events are written: + +```typescript + const emitSubagent = (event: SubagentEvent) => { + liveSubagents.set(event.id, event); + if (event.status !== 'running') { + liveSubagents.delete(event.id); + // Fire-and-forget: a persistence failure must never break the run. + if (tenantId) { + getSubagentRunRepository().save({ + tenantId, + threadId, + subagentId: event.id, + role: event.role, + task: event.task, + status: event.status, + toolCount: event.toolCount, + tokensIn: event.tokensIn, + tokensOut: event.tokensOut, + summary: event.summary ?? null, + transcript: (event.transcript ?? null) as never, + }).catch(err => console.error('[Chat] Failed to persist sub-agent run:', err)); + } + } + try { + controller.enqueue(buildSubagentPart(event)); + } catch { + // Client disconnected — the stream teardown handles it. + } + }; +``` + +Add the import: + +```typescript +import { getSubagentRunRepository } from '@/lib/db/repository-factory'; +``` + +- [ ] **Step 9: Fetch the transcript when a card is expanded** + +In `apps/web-ui/components/agent/chat/subagent-card.tsx`, accept a `threadId` prop and lazily load the persisted transcript. Replace the component signature and add the fetch: + +```tsx +export function SubagentCard({ subagent, threadId }: { subagent: SubagentState; threadId?: string }) { + const [expanded, setExpanded] = useState(false); + const canExpand = subagent.status !== 'running' && !!subagent.summary; + + // Loaded only on expand, and only when this card came from history (no live + // summary). Keeps the default render free of network work. + const { data: transcript } = useQuery({ + queryKey: ['subagent-transcript', threadId, subagent.id], + enabled: expanded && !!threadId, + queryFn: async () => { + const res = await fetch(`/api/chat/subagents/${threadId}`); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) throw new Error(json.error || 'Failed to load sub-agent transcript'); + const match = (json.data as Array<{ subagentId: string; transcript?: Array<{ kind: string; name?: string; text: string }> }>) + .find(r => r.subagentId === subagent.id); + return match?.transcript ?? []; + }, + }); +``` + +Add the import: + +```tsx +import { useQuery } from '@tanstack/react-query'; +``` + +And render the transcript below the findings block inside the `expanded` branch: + +```tsx + {transcript && transcript.length > 0 && ( +
+

Transcript

+
+ {transcript.map((entry, i) => ( +
+ + {entry.kind === 'tool' ? `${entry.name ?? 'tool'} → ` : 'thinking: '} + + {entry.text} +
+ ))} +
+
+ )} +``` + +`run-rail.tsx` already passes `threadId` into `SubagentCard` (Task 10, Step 7), so no further wiring is needed here. + +- [ ] **Step 10: Verify end to end** + +Run: `cd apps/web-ui && bunx tsc --noEmit` — no new errors. +Run: `cd apps/web-ui && bun run lint` — no new errors. +Run: `cd apps/web-ui && bun run test` — full suite, no new failures. + +Manually: run a fan-out task, wait for completion, **reload the page**, expand a sub-agent card, and confirm the task, findings, and transcript all render from the persisted row. + +- [ ] **Step 11: Commit** + +```bash +git add libs/prisma/schema.prisma libs/prisma/migrations apps/web-ui/lib/db/repositories/subagent apps/web-ui/lib/db/repository-factory.ts apps/web-ui/app/api/chat/subagents apps/web-ui/app/api/chat/route.ts apps/web-ui/components/agent/chat/subagent-card.tsx apps/web-ui/components/agent/chat/run-rail.tsx +git commit -m "feat(chat): persist sub-agent transcripts for expansion after reload + +Transcripts live in agent_subagent_runs (30-day TTL) and are fetched on demand, +never carried in the dispatch_agent ToolMessage — that message enters the +orchestrator's context and would undo the isolation sub-agents exist to give." +``` + +--- + +## Final verification + +- [ ] Run the full web-ui suite: `cd apps/web-ui && bun run test`. Compare failures against the pre-Task-1 baseline; there must be no new ones. +- [ ] Run `cd apps/web-ui && bunx tsc --noEmit` and `bun run lint` — clean. +- [ ] Re-run the representative slow task from Task 2 Step 8 with `SUBAGENTS_ENABLED=true` and sub-agents enabled for the tenant. Record the `📊 [RUN SUMMARY]` and compare against both the Task 2 baseline and the Task 3 measurement. +- [ ] Confirm the safety property by inspection: with sub-agents enabled, ask the agent to stop an EC2 instance across several accounts. Every mutation must still surface in the approval card — no mutation may occur inside a sub-agent. From 600d27122e1a15df8d5d50cc54275318b1fecbdd Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 01:28:24 +0530 Subject: [PATCH 04/42] fix(agent): make AWS session-profile writes concurrency-safe Unlocked read-modify-write on the shared per-tenant credentials file let concurrent get_aws_credentials calls silently drop each other's profiles, surfacing as 'The config profile could not be found'. Serialize per file, write atomically via rename, and cache profiles with refresh-on-near-expiry so fan-out does not multiply AssumeRole calls. --- apps/web-ui/lib/agent/aws-credentials-tool.ts | 31 +++-- apps/web-ui/lib/agent/session-manager.test.ts | 91 +++++++++++++++ apps/web-ui/lib/agent/session-manager.ts | 106 +++++++++++++++--- 3 files changed, 198 insertions(+), 30 deletions(-) create mode 100644 apps/web-ui/lib/agent/session-manager.test.ts diff --git a/apps/web-ui/lib/agent/aws-credentials-tool.ts b/apps/web-ui/lib/agent/aws-credentials-tool.ts index 38e71966..5d90853b 100644 --- a/apps/web-ui/lib/agent/aws-credentials-tool.ts +++ b/apps/web-ui/lib/agent/aws-credentials-tool.ts @@ -2,7 +2,7 @@ import { tool } from '@langchain/core/tools'; import { z } from 'zod'; import { STSClient, AssumeRoleCommand } from '@aws-sdk/client-sts'; import { AccountService } from '../account-service'; -import { createSessionProfile } from './session-manager'; +import { getOrCreateSessionProfile } from './session-manager'; import { env } from '@/env'; /** @@ -97,28 +97,25 @@ export function createGetAwsCredentialsTool(tenantId: string) { }); } - // 2. Assume the role to get temporary credentials - const { credentials, expiration } = await assumeRoleForAccount( - account.roleArn, - account.externalId - ); - - // 3. Determine region (use first region from account, or default) + // 2/3/4. Resolve region, then get a cached-or-fresh session profile. + // getOrCreateSessionProfile only calls STS when there is no live + // profile left for this (tenant, account) pair. const region = account.regions?.[0] || env.AWS_REGION || env.NEXT_PUBLIC_AWS_REGION || 'us-east-1'; - // 4. Create session profile in the tenant-isolated credentials file - const profile = await createSessionProfile( - accountId, - { + const profile = await getOrCreateSessionProfile(accountId, tenantId, async () => { + const { credentials } = await assumeRoleForAccount( + account.roleArn!, + account.externalId, + ); + return { accessKeyId: credentials.AccessKeyId!, secretAccessKey: credentials.SecretAccessKey!, sessionToken: credentials.SessionToken!, - region: region - }, - tenantId - ); + region, + }; + }); - console.log(`[Tool] Created profile: ${profile.profileName} for account: ${accountId}`); + console.log(`[Tool] Profile: ${profile.profileName} for account: ${accountId}`); console.log(`[Tool] Profile expires at: ${profile.expiresAt.toISOString()}`); return JSON.stringify({ diff --git a/apps/web-ui/lib/agent/session-manager.test.ts b/apps/web-ui/lib/agent/session-manager.test.ts new file mode 100644 index 00000000..0a940c1d --- /dev/null +++ b/apps/web-ui/lib/agent/session-manager.test.ts @@ -0,0 +1,91 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs/promises'; +import * as os from 'os'; +import * as path from 'path'; + +import { + createSessionProfile, + getOrCreateSessionProfile, + getTenantCredentialsFilePath, + __resetProfileCacheForTests, +} from './session-manager'; + +const TENANT = 'tenant-concurrency-test'; + +function creds(region = 'us-east-1') { + return { + accessKeyId: 'AKIAEXAMPLE', + secretAccessKey: 'secret', + sessionToken: 'token', + region, + }; +} + +beforeEach(() => { + __resetProfileCacheForTests(); +}); + +afterEach(async () => { + await fs.rm(path.dirname(getTenantCredentialsFilePath(TENANT)), { recursive: true, force: true }); +}); + +describe('createSessionProfile concurrency', () => { + it('keeps every profile when 10 callers write at the same time', async () => { + const accountIds = Array.from({ length: 10 }, (_, i) => `10000000000${i}`); + + const profiles = await Promise.all( + accountIds.map(id => createSessionProfile(id, creds(), TENANT)), + ); + + const contents = await fs.readFile(getTenantCredentialsFilePath(TENANT), 'utf-8'); + for (const profile of profiles) { + expect(contents).toContain(`[${profile.profileName}]`); + } + }); + + it('writes the credentials file atomically (no partial content observed)', async () => { + await createSessionProfile('222222222222', creds(), TENANT); + const filePath = getTenantCredentialsFilePath(TENANT); + const stat = await fs.stat(filePath); + expect(stat.mode & 0o777).toBe(0o600); + + const dir = path.dirname(filePath); + const leftovers = (await fs.readdir(dir)).filter(f => f.endsWith('.tmp')); + expect(leftovers).toEqual([]); + }); +}); + +describe('getOrCreateSessionProfile caching', () => { + it('reuses a fresh profile instead of assuming again', async () => { + const assume = vi.fn().mockResolvedValue(creds()); + + const first = await getOrCreateSessionProfile('333333333333', TENANT, assume); + const second = await getOrCreateSessionProfile('333333333333', TENANT, assume); + + expect(assume).toHaveBeenCalledTimes(1); + expect(second.profileName).toBe(first.profileName); + }); + + it('re-assumes when the cached profile is near expiry', async () => { + const assume = vi.fn().mockResolvedValue(creds()); + + const first = await getOrCreateSessionProfile('444444444444', TENANT, assume); + // Force the cached entry to be 30s from expiry — inside the 120s refresh margin. + first.expiresAt = new Date(Date.now() + 30_000); + + const second = await getOrCreateSessionProfile('444444444444', TENANT, assume); + + expect(assume).toHaveBeenCalledTimes(2); + expect(second.profileName).not.toBe(first.profileName); + }); + + it('scopes the cache by tenant', async () => { + const assume = vi.fn().mockResolvedValue(creds()); + + await getOrCreateSessionProfile('555555555555', TENANT, assume); + await getOrCreateSessionProfile('555555555555', `${TENANT}-other`, assume); + + expect(assume).toHaveBeenCalledTimes(2); + await fs.rm(path.dirname(getTenantCredentialsFilePath(`${TENANT}-other`)), { recursive: true, force: true }); + }); +}); diff --git a/apps/web-ui/lib/agent/session-manager.ts b/apps/web-ui/lib/agent/session-manager.ts index 6db32992..51e3327a 100644 --- a/apps/web-ui/lib/agent/session-manager.ts +++ b/apps/web-ui/lib/agent/session-manager.ts @@ -1,6 +1,7 @@ import * as fs from 'fs/promises'; import * as path from 'path'; import * as os from 'os'; +import { randomUUID } from 'crypto'; /** * AWS Session Profile Manager @@ -52,6 +53,30 @@ const TENANT_CREDS_ROOT = path.join(os.tmpdir(), 'nucleus-aws-creds'); // Profile name prefix for identification during cleanup const PROFILE_PREFIX = 'nucleus_agent_'; +/** + * Serializes read-modify-write cycles per credentials file. + * + * Node is single-threaded, but `await` between the read and the write yields the + * event loop: two concurrent callers both read the old content and the second + * write clobbers the first caller's profile. The victim then runs + * `aws --profile ` against a profile that is not in the file and gets + * "The config profile could not be found". Chaining every mutation onto a + * per-path promise removes the interleaving window. + * + * The map is keyed by resolved file path, so the legacy shared-file path is + * covered too. Key count is bounded by tenant count, so it is not a leak. + */ +const fileLocks = new Map>(); + +function withFileLock(filePath: string, fn: () => Promise): Promise { + const previous = fileLocks.get(filePath) ?? Promise.resolve(); + // Run `fn` whether the predecessor resolved or rejected — one caller's + // failure must not deadlock every later caller on the same file. + const next = previous.then(fn, fn); + fileLocks.set(filePath, next.catch(() => undefined)); + return next; +} + /** * Resolve the absolute credentials-file path for a tenant. * Deterministic so execute_command (which only has the tenantId) can point @@ -90,7 +115,17 @@ async function readCredentialsFile(filePath: string): Promise { async function writeCredentialsFile(filePath: string, content: string): Promise { const dir = path.dirname(filePath); await fs.mkdir(dir, { recursive: true }); - await fs.writeFile(filePath, content, { mode: 0o600 }); + // Write-then-rename: a concurrent reader (the AWS CLI) never observes a + // half-written credentials file. The temp file is created in the same + // directory so the rename stays on one filesystem and is therefore atomic. + const tmpPath = path.join(dir, `.credentials.${process.pid}.${randomUUID()}.tmp`); + try { + await fs.writeFile(tmpPath, content, { mode: 0o600 }); + await fs.rename(tmpPath, filePath); + } catch (error) { + await fs.rm(tmpPath, { force: true }).catch(() => undefined); + throw error; + } } /** @@ -156,20 +191,19 @@ export async function createSessionProfile( credentialsFile, }; - // Read current credentials file - const content = await readCredentialsFile(credentialsFile); - const profiles = parseCredentialsFile(content); + await withFileLock(credentialsFile, async () => { + const content = await readCredentialsFile(credentialsFile); + const profiles = parseCredentialsFile(content); - // Add new profile - const profileCreds = new Map(); - profileCreds.set('aws_access_key_id', credentials.accessKeyId); - profileCreds.set('aws_secret_access_key', credentials.secretAccessKey); - profileCreds.set('aws_session_token', credentials.sessionToken); - profileCreds.set('region', credentials.region); - profiles.set(profileName, profileCreds); + const profileCreds = new Map(); + profileCreds.set('aws_access_key_id', credentials.accessKeyId); + profileCreds.set('aws_secret_access_key', credentials.secretAccessKey); + profileCreds.set('aws_session_token', credentials.sessionToken); + profileCreds.set('region', credentials.region); + profiles.set(profileName, profileCreds); - // Write back to file - await writeCredentialsFile(credentialsFile, serializeCredentialsFile(profiles)); + await writeCredentialsFile(credentialsFile, serializeCredentialsFile(profiles)); + }); console.log(`[SessionManager] Created profile: ${profileName} in ${credentialsFile}`); @@ -229,5 +263,51 @@ export async function cleanupAllAgentProfiles(): Promise { } } +/** + * Re-assume when a cached profile has less than this much life left. STS + * credentials here last 900s (see assumeRoleForAccount); handing back a profile + * with 10s remaining guarantees the next AWS CLI call fails mid-flight. + */ +const PROFILE_REFRESH_MARGIN_MS = 120_000; + +const profileCache = new Map(); + +function profileCacheKey(tenantId: string, accountId: string): string { + return `${tenantId}:${accountId}`; +} + +/** + * Return a usable session profile for (tenant, account), assuming fresh + * credentials only when there is no cached profile or the cached one is within + * PROFILE_REFRESH_MARGIN_MS of expiry. + * + * Fan-out makes this matter twice over: N sub-agents auditing the same account + * previously triggered N AssumeRole calls, and a long run could hand out a + * profile that expired before it was used. + */ +export async function getOrCreateSessionProfile( + accountId: string, + tenantId: string, + assume: () => Promise, +): Promise { + const key = profileCacheKey(tenantId, accountId); + const cached = profileCache.get(key); + if (cached && cached.expiresAt.getTime() - Date.now() > PROFILE_REFRESH_MARGIN_MS) { + console.log(`[SessionManager] Reusing cached profile ${cached.profileName} for ${key}`); + return cached; + } + + const credentials = await assume(); + const profile = await createSessionProfile(accountId, credentials, tenantId); + profileCache.set(key, profile); + return profile; +} + +/** Test seam — clears the in-process profile cache between test cases. */ +export function __resetProfileCacheForTests(): void { + profileCache.clear(); + fileLocks.clear(); +} + // Run cleanup on module load (server startup) cleanupAllAgentProfiles().catch(console.error); From a6eddad55e2726a91a177c1fe9349d4524377b13 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 01:38:06 +0530 Subject: [PATCH 05/42] feat(agent): per-run LLM timing summary Aggregates the latency and token counts llmAuditLog already computes into a per-node breakdown logged at run end, so the cost of each graph node is attributable before parallelism changes land. --- apps/web-ui/app/api/chat/route.ts | 5 ++ apps/web-ui/lib/agent/planning-agent.ts | 17 +++- apps/web-ui/lib/agent/run-timings.test.ts | 43 ++++++++++ apps/web-ui/lib/agent/run-timings.ts | 96 +++++++++++++++++++++++ 4 files changed, 158 insertions(+), 3 deletions(-) create mode 100644 apps/web-ui/lib/agent/run-timings.test.ts create mode 100644 apps/web-ui/lib/agent/run-timings.ts diff --git a/apps/web-ui/app/api/chat/route.ts b/apps/web-ui/app/api/chat/route.ts index 87eb8b65..93717b58 100644 --- a/apps/web-ui/app/api/chat/route.ts +++ b/apps/web-ui/app/api/chat/route.ts @@ -12,6 +12,7 @@ import { acquireThreadLock, releaseThreadLock as releaseThreadLockDb } from '@/l import { buildPlanPart, buildPhasePart, buildInterruptParts, buildMemoryPart, buildUsagePart, humanizeReflection, humanizePlanning, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from './stream-parts'; import { resolveResumedToolCallId, type ResumedPendingCall } from './resume-tool-id'; import type { PlanStep } from '@/lib/agent/agent-shared'; +import { logRunSummary } from '@/lib/agent/run-timings'; export const maxDuration = 300; // 5 minutes for complex multi-iteration tasks @@ -1170,6 +1171,10 @@ function processStream( } } finally { + // Emit the per-run timing breakdown regardless of how the run ended, + // so aborted and errored runs are measured too. + if (threadId) logRunSummary(threadId); + // Persist NEW messages from this turn to chat history if (threadId && graph && config && resolvedUserId) { try { diff --git a/apps/web-ui/lib/agent/planning-agent.ts b/apps/web-ui/lib/agent/planning-agent.ts index 252e6537..c9603f18 100644 --- a/apps/web-ui/lib/agent/planning-agent.ts +++ b/apps/web-ui/lib/agent/planning-agent.ts @@ -36,6 +36,7 @@ import { createMemoryRecallNode, createMemorySaveNode } from "./memory-nodes"; import { prepareContext, buildWorkingMemorySection, estimateTokens } from "./memory/working-memory"; import { createGuardNode } from "./guard"; import { routeAfterGuard } from "./gate-routing"; +import { recordNodeTiming } from "./run-timings"; const PLAN_STATUSES = new Set(['completed', 'in_progress', 'pending', 'failed']); @@ -253,7 +254,7 @@ export async function createReflectionGraph(config: GraphConfig) { // --------------------------------------------------------------------------- // PLANNER NODE // --------------------------------------------------------------------------- - async function planNode(state: ReflectionState): Promise> { + async function planNode(state: ReflectionState, runtimeConfig?: any): Promise> { const { messages, memoryContext } = state; const workingMemorySection = buildWorkingMemorySection( state.runningSummary || (state.scratchpad?.openGoals?.length ?? 0) > 0 @@ -313,6 +314,8 @@ Only return the JSON array, nothing else.`); try { response = await model.invoke(_auditInputs_plan) as AIMessage; llmAuditLog('PLANNER', _auditInputs_plan, response, _auditStart_plan); + recordNodeTiming(runtimeConfig?.configurable?.thread_id, 'PLANNER', Date.now() - _auditStart_plan, + (response as any).usage_metadata?.input_tokens ?? 0, (response as any).usage_metadata?.output_tokens ?? 0); } catch (err: any) { // A provider hiccup while planning must not abort the run — fall back to a // trivial single-step plan (the empty content flows through the parse fallback below). @@ -430,6 +433,8 @@ The user sees plan progress in a UI rail — your prose must NEVER duplicate it: try { response = await modelWithTools.invoke(_auditInputs_exec) as AIMessage; llmAuditLog('EXECUTOR', _auditInputs_exec, response, _auditStart_exec); + recordNodeTiming(runtimeConfig?.configurable?.thread_id, 'EXECUTOR', Date.now() - _auditStart_exec, + (response as any).usage_metadata?.input_tokens ?? 0, (response as any).usage_metadata?.output_tokens ?? 0); } catch (err: any) { // Provider error mid-step: don't crash the run. Emit a text note (no tool calls) // so the graph routes on to reflection/finalization with whatever was gathered. @@ -533,7 +538,7 @@ The user sees plan progress in a UI rail — your prose must NEVER duplicate it: // --------------------------------------------------------------------------- // REFLECTOR NODE // --------------------------------------------------------------------------- - async function reflectNode(state: ReflectionState): Promise> { + async function reflectNode(state: ReflectionState, runtimeConfig?: any): Promise> { const { messages, taskDescription, iterationCount, plan, toolResults, memoryContext } = state; console.log(`\n================================================================================`); @@ -646,6 +651,8 @@ ${plan.map((s, i) => `${i + 1}. [${s.status}] ${s.step}`).join('\n')}` try { response = await reflectorModel.invoke(_auditInputs_ref); llmAuditLog('REFLECTOR', _auditInputs_ref, response, _auditStart_ref); + recordNodeTiming(runtimeConfig?.configurable?.thread_id, 'REFLECTOR', Date.now() - _auditStart_ref, + (response as any).usage_metadata?.input_tokens ?? 0, (response as any).usage_metadata?.output_tokens ?? 0); } catch (err: any) { // If the reflector fails (throttle, context overflow, parse-impossible), treat the // work as complete and force finalization rather than aborting the whole run. @@ -770,6 +777,8 @@ ${accountContext}`); try { response = await modelWithTools.invoke(_auditInputs_rev) as AIMessage; llmAuditLog('REVISER', _auditInputs_rev, response, _auditStart_rev); + recordNodeTiming(runtimeConfig?.configurable?.thread_id, 'REVISER', Date.now() - _auditStart_rev, + (response as any).usage_metadata?.input_tokens ?? 0, (response as any).usage_metadata?.output_tokens ?? 0); } catch (err: any) { // Provider error while revising: emit a text note (no tool calls) so the graph // routes back to reflection and can finalize instead of aborting. @@ -804,7 +813,7 @@ ${accountContext}`); // --------------------------------------------------------------------------- // FINAL OUTPUT NODE // --------------------------------------------------------------------------- - async function finalNode(state: ReflectionState): Promise> { + async function finalNode(state: ReflectionState, runtimeConfig?: any): Promise> { const { taskDescription, toolResults } = state; console.log(`\n================================================================================`); @@ -869,6 +878,8 @@ ${groundingRule}`); try { const summaryResponse = await model.invoke(_auditInputs_fin); llmAuditLog('FINAL', _auditInputs_fin, summaryResponse, _auditStart_fin); + recordNodeTiming(runtimeConfig?.configurable?.thread_id, 'FINAL', Date.now() - _auditStart_fin, + (summaryResponse as any).usage_metadata?.input_tokens ?? 0, (summaryResponse as any).usage_metadata?.output_tokens ?? 0); summaryContent = contentToText(summaryResponse.content); } catch (err: any) { // A finalize failure must never crash the run — assemble a best-effort answer diff --git a/apps/web-ui/lib/agent/run-timings.test.ts b/apps/web-ui/lib/agent/run-timings.test.ts new file mode 100644 index 00000000..caf17a96 --- /dev/null +++ b/apps/web-ui/lib/agent/run-timings.test.ts @@ -0,0 +1,43 @@ +import { describe, it, expect, beforeEach } from 'vitest'; +import { recordNodeTiming, summarizeRun, logRunSummary, __resetRunTimingsForTests } from './run-timings'; + +beforeEach(() => __resetRunTimingsForTests()); + +describe('run timings', () => { + it('aggregates calls per node', () => { + recordNodeTiming('t1', 'EXECUTOR', 1000, 500, 100); + recordNodeTiming('t1', 'EXECUTOR', 2000, 600, 150); + recordNodeTiming('t1', 'REFLECTOR', 500, 200, 50); + + const summary = summarizeRun('t1')!; + + expect(summary.totalLlmMs).toBe(3500); + expect(summary.totalTokensIn).toBe(1300); + expect(summary.totalTokensOut).toBe(300); + expect(summary.byNode.EXECUTOR).toEqual({ calls: 2, ms: 3000, tokensIn: 1100, tokensOut: 250 }); + expect(summary.byNode.REFLECTOR.calls).toBe(1); + }); + + it('keeps runs isolated by threadId', () => { + recordNodeTiming('t1', 'EXECUTOR', 1000, 10, 10); + recordNodeTiming('t2', 'EXECUTOR', 5000, 20, 20); + + expect(summarizeRun('t1')!.totalLlmMs).toBe(1000); + expect(summarizeRun('t2')!.totalLlmMs).toBe(5000); + }); + + it('ignores calls with no threadId rather than throwing', () => { + expect(() => recordNodeTiming(undefined, 'EXECUTOR', 100, 1, 1)).not.toThrow(); + expect(summarizeRun('t1')).toBeNull(); + }); + + it('discards the run after logging its summary', () => { + recordNodeTiming('t1', 'EXECUTOR', 100, 1, 1); + logRunSummary('t1'); + expect(summarizeRun('t1')).toBeNull(); + }); + + it('returns null for an unknown thread', () => { + expect(summarizeRun('never-seen')).toBeNull(); + }); +}); diff --git a/apps/web-ui/lib/agent/run-timings.ts b/apps/web-ui/lib/agent/run-timings.ts new file mode 100644 index 00000000..96b7908a --- /dev/null +++ b/apps/web-ui/lib/agent/run-timings.ts @@ -0,0 +1,96 @@ +/** + * Per-run LLM timing accumulator. + * + * llmAuditLog already measures latency and tokens for every model call, but + * throws the numbers away unless LLM_AUDIT is enabled. This module keeps a + * lightweight always-on aggregate so a run's wall time can be attributed to + * specific graph nodes — the baseline needed to tell whether parallelism + * actually helped. + * + * Keyed by threadId because runs are concurrent within one process. + */ + +export interface NodeTiming { + calls: number; + ms: number; + tokensIn: number; + tokensOut: number; +} + +export interface RunTimingSummary { + totalLlmMs: number; + totalTokensIn: number; + totalTokensOut: number; + byNode: Record; +} + +/** Backstop against unbounded growth if a run never reaches its teardown. */ +const MAX_TRACKED_RUNS = 200; + +const runs = new Map>(); + +export function recordNodeTiming( + threadId: string | undefined, + node: string, + latencyMs: number, + tokensIn: number, + tokensOut: number, +): void { + if (!threadId) return; + + let byNode = runs.get(threadId); + if (!byNode) { + if (runs.size >= MAX_TRACKED_RUNS) { + // Evict the oldest insertion — Map preserves insertion order. + const oldest = runs.keys().next().value; + if (oldest !== undefined) runs.delete(oldest); + } + byNode = {}; + runs.set(threadId, byNode); + } + + const entry = byNode[node] ?? { calls: 0, ms: 0, tokensIn: 0, tokensOut: 0 }; + entry.calls += 1; + entry.ms += latencyMs; + entry.tokensIn += tokensIn; + entry.tokensOut += tokensOut; + byNode[node] = entry; +} + +export function summarizeRun(threadId: string): RunTimingSummary | null { + const byNode = runs.get(threadId); + if (!byNode) return null; + + let totalLlmMs = 0; + let totalTokensIn = 0; + let totalTokensOut = 0; + for (const entry of Object.values(byNode)) { + totalLlmMs += entry.ms; + totalTokensIn += entry.tokensIn; + totalTokensOut += entry.tokensOut; + } + + return { totalLlmMs, totalTokensIn, totalTokensOut, byNode }; +} + +export function logRunSummary(threadId: string): void { + const summary = summarizeRun(threadId); + runs.delete(threadId); + if (!summary) return; + + const rows = Object.entries(summary.byNode) + .sort((a, b) => b[1].ms - a[1].ms) + .map(([node, t]) => + ` ${node.padEnd(12)} calls=${String(t.calls).padStart(3)} llm=${(t.ms / 1000).toFixed(1)}s in=${t.tokensIn} out=${t.tokensOut}`); + + console.log( + `\n📊 [RUN SUMMARY] thread=${threadId}\n` + + ` TOTAL llm=${(summary.totalLlmMs / 1000).toFixed(1)}s in=${summary.totalTokensIn} out=${summary.totalTokensOut}\n` + + rows.join('\n'), + ); +} + +/** Test seam. */ +export function __resetRunTimingsForTests(): void { + runs.clear(); +} From 007d70a5fb86ee2a1f5080785f89d6917575c5e5 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 01:47:04 +0530 Subject: [PATCH 06/42] docs(aiops): fix run-timings eviction defect in the plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Map keeps INSERTION order and .set() on an existing key does not refresh it, so the plan's eviction snippet targeted the oldest-STARTED run — precisely the long run whose measurement matters most. Delete-then-set makes eviction least-recently-ACTIVE, plus two regression tests. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-07-26-aiops-subagent-orchestration.md | 36 +++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md index a7d0d8dd..0208320d 100644 --- a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md +++ b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md @@ -401,6 +401,33 @@ describe('run timings', () => { it('returns null for an unknown thread', () => { expect(summarizeRun('never-seen')).toBeNull(); }); + + it('does not evict a run that is still recording', () => { + // The eviction backstop must target the most IDLE run, not the + // oldest-started one. A 10-minute agent run is exactly the run whose + // numbers matter most, and it is also the one that has been in the map + // longest — evicting it would silently discard the measurement. + recordNodeTiming('long-runner', 'EXECUTOR', 1000, 10, 10); + + // Fill past capacity with other threads, while the long runner keeps working. + for (let i = 0; i < 250; i++) { + recordNodeTiming(`other-${i}`, 'EXECUTOR', 1, 1, 1); + recordNodeTiming('long-runner', 'EXECUTOR', 1000, 10, 10); + } + + const summary = summarizeRun('long-runner'); + expect(summary).not.toBeNull(); + expect(summary!.byNode.EXECUTOR.calls).toBe(251); + }); + + it('still bounds the map when runs never reach teardown', () => { + for (let i = 0; i < 300; i++) { + recordNodeTiming(`abandoned-${i}`, 'EXECUTOR', 1, 1, 1); + } + // The earliest abandoned runs must have been evicted. + expect(summarizeRun('abandoned-0')).toBeNull(); + expect(summarizeRun('abandoned-299')).not.toBeNull(); + }); }); ``` @@ -457,13 +484,18 @@ export function recordNodeTiming( let byNode = runs.get(threadId); if (!byNode) { if (runs.size >= MAX_TRACKED_RUNS) { - // Evict the oldest insertion — Map preserves insertion order. const oldest = runs.keys().next().value; if (oldest !== undefined) runs.delete(oldest); } byNode = {}; - runs.set(threadId, byNode); + } else { + // Refresh position: Map keeps INSERTION order, and .set() on an + // existing key does not move it. Without this, eviction targets the + // oldest-STARTED run — which is exactly the long run we most want to + // measure. Delete-then-set makes it least-recently-ACTIVE. + runs.delete(threadId); } + runs.set(threadId, byNode); const entry = byNode[node] ?? { calls: 0, ms: 0, tokensIn: 0, tokensOut: 0 }; entry.calls += 1; From 0e005657b46a2d558c7d5601968fc66fdc7fca48 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 01:48:19 +0530 Subject: [PATCH 07/42] fix(agent): stop run-timing eviction from truncating long-running threads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MAX_TRACKED_RUNS eviction targeted the oldest-INSERTED key, but Map.set() on an existing key never moves it — so a long agent run (up to 30 iterations) got evicted mid-run once 200 other threads had started, silently discarding exactly the numbers this instrumentation exists to capture. Delete-then-set on every write now refreshes position to least-recently-active instead. --- apps/web-ui/lib/agent/run-timings.test.ts | 27 +++++++++++++++++++++++ apps/web-ui/lib/agent/run-timings.ts | 8 ++++++- 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/apps/web-ui/lib/agent/run-timings.test.ts b/apps/web-ui/lib/agent/run-timings.test.ts index caf17a96..4fc57ced 100644 --- a/apps/web-ui/lib/agent/run-timings.test.ts +++ b/apps/web-ui/lib/agent/run-timings.test.ts @@ -40,4 +40,31 @@ describe('run timings', () => { it('returns null for an unknown thread', () => { expect(summarizeRun('never-seen')).toBeNull(); }); + + it('does not evict a run that is still recording', () => { + // The eviction backstop must target the most IDLE run, not the + // oldest-started one. A 10-minute agent run is exactly the run whose + // numbers matter most, and it is also the one that has been in the map + // longest — evicting it would silently discard the measurement. + recordNodeTiming('long-runner', 'EXECUTOR', 1000, 10, 10); + + // Fill past capacity with other threads, while the long runner keeps working. + for (let i = 0; i < 250; i++) { + recordNodeTiming(`other-${i}`, 'EXECUTOR', 1, 1, 1); + recordNodeTiming('long-runner', 'EXECUTOR', 1000, 10, 10); + } + + const summary = summarizeRun('long-runner'); + expect(summary).not.toBeNull(); + expect(summary!.byNode.EXECUTOR.calls).toBe(251); + }); + + it('still bounds the map when runs never reach teardown', () => { + for (let i = 0; i < 300; i++) { + recordNodeTiming(`abandoned-${i}`, 'EXECUTOR', 1, 1, 1); + } + // The earliest abandoned runs must have been evicted. + expect(summarizeRun('abandoned-0')).toBeNull(); + expect(summarizeRun('abandoned-299')).not.toBeNull(); + }); }); diff --git a/apps/web-ui/lib/agent/run-timings.ts b/apps/web-ui/lib/agent/run-timings.ts index 96b7908a..0beb33c1 100644 --- a/apps/web-ui/lib/agent/run-timings.ts +++ b/apps/web-ui/lib/agent/run-timings.ts @@ -46,8 +46,14 @@ export function recordNodeTiming( if (oldest !== undefined) runs.delete(oldest); } byNode = {}; - runs.set(threadId, byNode); + } else { + // Refresh position: Map keeps INSERTION order, and .set() on an + // existing key does not move it. Without this, eviction targets the + // oldest-STARTED run — which is exactly the long run we most want to + // measure. Delete-then-set makes it least-recently-ACTIVE. + runs.delete(threadId); } + runs.set(threadId, byNode); const entry = byNode[node] ?? { calls: 0, ms: 0, tokensIn: 0, tokensOut: 0 }; entry.calls += 1; From 08c7a646d0def30987bb7b9548a58b01df9c3e94 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 01:58:21 +0530 Subject: [PATCH 08/42] perf(agent): allow batched parallel tool calls in one executor turn MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ToolNode already ran multiple tool_calls concurrently; the executor prompt forbade emitting them, so an N-way independent sweep cost N serial LLM laps. Independent read-only calls now batch into one turn while dependent calls and mutations stay sequential. Shell subprocesses are bounded by TOOL_CONCURRENCY via a new Semaphore (apps/web-ui/lib/agent/concurrency.ts). Live before/after call-count and wall-time measurement (brief Step 8) was deferred — this environment has no AWS credentials or running Postgres to exercise the agent end-to-end. Co-Authored-By: Claude Opus 5 (1M context) --- apps/web-ui/lib/agent/concurrency.test.ts | 56 +++++++++++++++++++++++ apps/web-ui/lib/agent/concurrency.ts | 45 ++++++++++++++++++ apps/web-ui/lib/agent/planning-agent.ts | 8 +++- apps/web-ui/lib/agent/tools.ts | 13 +++++- 4 files changed, 118 insertions(+), 4 deletions(-) create mode 100644 apps/web-ui/lib/agent/concurrency.test.ts create mode 100644 apps/web-ui/lib/agent/concurrency.ts diff --git a/apps/web-ui/lib/agent/concurrency.test.ts b/apps/web-ui/lib/agent/concurrency.test.ts new file mode 100644 index 00000000..bdb36e68 --- /dev/null +++ b/apps/web-ui/lib/agent/concurrency.test.ts @@ -0,0 +1,56 @@ +import { describe, it, expect } from 'vitest'; +import { Semaphore } from './concurrency'; + +const defer = () => { + let resolve!: () => void; + const promise = new Promise(r => { resolve = r; }); + return { promise, resolve }; +}; + +describe('Semaphore', () => { + it('never runs more than `limit` tasks at once', async () => { + const sem = new Semaphore(2); + let active = 0; + let peak = 0; + + const task = async () => sem.run(async () => { + active++; + peak = Math.max(peak, active); + await new Promise(r => setTimeout(r, 5)); + active--; + }); + + await Promise.all(Array.from({ length: 10 }, task)); + + expect(peak).toBe(2); + expect(active).toBe(0); + }); + + it('releases the slot when a task throws', async () => { + const sem = new Semaphore(1); + + await expect(sem.run(async () => { throw new Error('boom'); })).rejects.toThrow('boom'); + + // If the slot leaked, this would hang rather than resolve. + await expect(sem.run(async () => 'ok')).resolves.toBe('ok'); + }); + + it('queues callers beyond the limit and runs them as slots free', async () => { + const sem = new Semaphore(1); + const first = defer(); + const order: string[] = []; + + const a = sem.run(async () => { order.push('a-start'); await first.promise; order.push('a-end'); }); + const b = sem.run(async () => { order.push('b-start'); }); + + expect(order).toEqual(['a-start']); + first.resolve(); + await Promise.all([a, b]); + expect(order).toEqual(['a-start', 'a-end', 'b-start']); + }); + + it('treats a limit below 1 as 1', async () => { + const sem = new Semaphore(0); + await expect(sem.run(async () => 'ok')).resolves.toBe('ok'); + }); +}); diff --git a/apps/web-ui/lib/agent/concurrency.ts b/apps/web-ui/lib/agent/concurrency.ts new file mode 100644 index 00000000..be3949ed --- /dev/null +++ b/apps/web-ui/lib/agent/concurrency.ts @@ -0,0 +1,45 @@ +/** + * Minimal counting semaphore. + * + * Once the executor is allowed to emit many tool calls in one turn, ToolNode + * runs them all concurrently. That is the point — but execute_command spawns a + * real subprocess per call, so an unbounded 45-call turn would fork 45 shells + * inside the web-ui container. This bounds that specific blast radius. + */ +export class Semaphore { + private available: number; + private readonly waiters: Array<() => void> = []; + + constructor(limit: number) { + this.available = Math.max(1, Math.floor(limit)); + } + + async run(fn: () => Promise): Promise { + // Deliberately NOT `await this.acquire()` here: awaiting any promise — + // even one already resolved — yields a microtask before continuing, so + // a caller that finds a free slot would still run `fn` one tick late. + // Callers (and this file's own tests) rely on a free slot starting + // `fn` synchronously within the `run()` call. + if (this.available > 0) { + this.available--; + } else { + await new Promise(resolve => this.waiters.push(resolve)); + } + try { + return await fn(); + } finally { + this.release(); + } + } + + private release(): void { + const next = this.waiters.shift(); + if (next) { + // Hand the slot straight to the next waiter — do not increment, or a + // third caller could take the slot we just promised away. + next(); + return; + } + this.available++; + } +} diff --git a/apps/web-ui/lib/agent/planning-agent.ts b/apps/web-ui/lib/agent/planning-agent.ts index c9603f18..7f16cc47 100644 --- a/apps/web-ui/lib/agent/planning-agent.ts +++ b/apps/web-ui/lib/agent/planning-agent.ts @@ -290,7 +290,7 @@ Work through three phases when building the plan: - The plan is INTERNAL COORDINATION. The user sees it in a progress rail — it is never part of the answer, so optimize it for execution, not for presentation. - Match plan depth to the request. A simple lookup or single-command request needs only 1–2 steps (credentials + the query); a conversational or no-tool request gets exactly one step: ["Answer the user's request directly"]. Only genuinely multi-phase investigations warrant 5+ steps. -- Keep the plan SHORT: never more than 7 steps. Merge related read-only queries into a single step (e.g., one step for "inventory EC2 + RDS + EBS across regions", not one step per service per region). +- Keep the plan SHORT: never more than 7 steps. Merge related read-only queries into a single step (e.g., one step for "inventory EC2 + RDS + EBS across regions", not one step per service per region). The executor issues the individual queries in that step as parallel tool calls, so a merged step costs no more time than a narrow one. - Every step except the last must be a concrete tool action. Do NOT create steps for aggregating, analyzing, summarizing, cross-referencing, or "evaluating" data — that thinking happens inside execution, not as plan line items. - Do NOT add a knowledge-base search step unless the user asked for it or the task clearly depends on tenant-specific documented context. - The LAST step is always: compose the final answer to the user's request from the gathered data. @@ -404,7 +404,10 @@ ${accountContext} ## Execution Discipline -- Execute exactly the current step — do not skip ahead or bundle future steps into a single call. +- Execute the current step. When that step covers several INDEPENDENT read-only lookups (different accounts, regions, or services), issue them as MULTIPLE tool calls in this single turn — they run in parallel. One call per turn for independent reads wastes minutes. +- Calls that DEPEND on each other must stay sequential: if call B needs call A's output (an account id, a resource id, a profile name), make call A now and call B on the next turn. +- Never batch mutations. Issue at most ONE state-changing call per turn so each is reviewed and approved individually. +- Do not skip ahead to later plan steps. - If a tool call returns an error, capture the full error message and diagnose it; do not silently suppress it. - If the current step is a simple question or greeting that requires no tools, answer directly and concisely. - If the current step has nothing to execute (empty inventory, prerequisite returned no data), move on silently — NEVER run echo or other no-op commands just to mark a step done, and never write an explanation of why there was nothing to do. @@ -764,6 +767,7 @@ Issues to Address: ${errors.join(', ') || 'None'} 6. For errors returned by tools: diagnose the root cause (permissions, resource not found, wrong region, wrong account) and fix the underlying issue rather than retrying the same command unchanged. 7. Do not repeat actions that the reviewer marked as correctly completed — focus only on the open issues. 8. After fixing all issues, provide a brief summary of what was corrected and what the result now shows. +9. When the fix requires several independent read-only re-checks, issue them as multiple tool calls in this single turn rather than one per turn. ${accountContext}`); const recentMessages = windowMessages; diff --git a/apps/web-ui/lib/agent/tools.ts b/apps/web-ui/lib/agent/tools.ts index 315c344b..b950617d 100644 --- a/apps/web-ui/lib/agent/tools.ts +++ b/apps/web-ui/lib/agent/tools.ts @@ -10,6 +10,7 @@ import { Readable } from 'stream'; import { env } from '@/env'; import { getTenantCredentialsFilePath } from './session-manager'; import { AuditService } from '@/lib/audit-service'; +import { Semaphore } from './concurrency'; // Re-export AWS credentials tool factories export { createGetAwsCredentialsTool, createListAwsAccountsTool } from './aws-credentials-tool'; @@ -81,6 +82,14 @@ function buildCommandEnv(tenantId?: string): Record { // Helper to truncate large tool outputs to prevent prompt token limits const MAX_OUTPUT_LENGTH = 100000; +/** + * Caps concurrent shell subprocesses across the whole process. Other tools + * (AWS SDK, Prisma, MCP) are network/DB calls that are already pooled by their + * own clients, so only execute_command needs an explicit bound. + */ +const TOOL_CONCURRENCY = Number(process.env.TOOL_CONCURRENCY) || 6; +const commandSemaphore = new Semaphore(TOOL_CONCURRENCY); + const truncateToolOutput = (output: string) => { if (output && output.length > MAX_OUTPUT_LENGTH) { return output.substring(0, MAX_OUTPUT_LENGTH) + `\n\n...[Output truncated due to length. Total length: ${output.length} characters]...`; @@ -128,12 +137,12 @@ export const executeCommandTool = tool( }; try { - const { stdout, stderr } = await execAsync(command, { + const { stdout, stderr } = await commandSemaphore.run(() => execAsync(command, { shell: '/bin/bash', timeout: 120000, // 2 minute timeout for long-running AWS CLI commands maxBuffer: 1024 * 1024 * 10, // 10MB buffer env: buildCommandEnv(tenantId) as NodeJS.ProcessEnv, - }); + })); const output = stdout || stderr || 'Command executed successfully (no output)'; console.log(`[Tool] Command Output Length: ${output.length}`); From 8c651203ddf56bb3176785149ab9e5221b12243a Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:00:52 +0530 Subject: [PATCH 09/42] docs(aiops): plan fixes for semaphore fast path and profile-name collisions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects in plan-authored code, both surfaced during Task 3: - Semaphore.run awaited acquire() even on the free-slot path, deferring a microtask so fn() started one tick late — the plan's own test asserted synchronous start, so the plan code failed the plan's test. Inline the slot check. - generateProfileName derives names from Date.now() alone, so two profiles for one account in the same millisecond collide and the second overwrites the first. Added as Task 1 Step 9 with a regression test. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-07-26-aiops-subagent-orchestration.md | 95 +++++++++++++++++-- 1 file changed, 86 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md index 0208320d..8eb2e451 100644 --- a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md +++ b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md @@ -334,6 +334,82 @@ write atomically via rename, and cache profiles with refresh-on-near-expiry so fan-out does not multiply AssumeRole calls." ``` +- [ ] **Step 9: Make profile names collision-proof** + +Added after Task 3 surfaced this as an intermittent failure of Step 1's +"re-assumes when the cached profile is near expiry" test (roughly 1 run in 3). + +`generateProfileName` derives the name from `Date.now()` alone: + +```typescript +function generateProfileName(accountId: string): string { + const timestamp = Date.now(); + return `${PROFILE_PREFIX}${accountId}_${timestamp}`; +} +``` + +Two profiles for the same account created within the same millisecond therefore +get **identical names**, and the second silently overwrites the first's section in +the credentials file — the same lost-profile bug class this task exists to remove, +just reached by a different route. The mutex cannot help: each call is individually +serialized and still produces a colliding name. Parallel fan-out (Task 3) and +parallel sub-agents (Task 8) make same-millisecond creation routine rather than +theoretical. + +Replace it with: + +```typescript +function generateProfileName(accountId: string): string { + // Date.now() alone collides when two profiles for one account are created in + // the same millisecond — which parallel get_aws_credentials calls do routinely. + // A colliding name means the second profile overwrites the first's section and + // the first caller's handle silently points at someone else's credentials. + return `${PROFILE_PREFIX}${accountId}_${Date.now()}_${randomUUID().slice(0, 8)}`; +} +``` + +`randomUUID` is already imported at the top of the file from Step 3. + +Add this regression test to `session-manager.test.ts`, inside the +`describe('createSessionProfile concurrency', ...)` block: + +```typescript + it('gives every profile a unique name even within the same millisecond', async () => { + // 50 back-to-back creations for ONE account will land many in the same ms. + const profiles = await Promise.all( + Array.from({ length: 50 }, () => createSessionProfile('999999999999', creds(), TENANT)), + ); + + const names = new Set(profiles.map(p => p.profileName)); + expect(names.size).toBe(50); + + // And every one of them must actually be present in the file. + const contents = await fs.readFile(getTenantCredentialsFilePath(TENANT), 'utf-8'); + for (const name of names) { + expect(contents).toContain(`[${name}]`); + } + }); +``` + +Run: `cd apps/web-ui && bunx vitest run lib/agent/session-manager.test.ts` — expect 6/6. + +Then run it **five times in a row** and confirm 6/6 every time; the bug this fixes +is intermittent, so a single green run proves nothing: + +```bash +cd apps/web-ui && for i in 1 2 3 4 5; do bunx vitest run lib/agent/session-manager.test.ts 2>&1 | grep -E "^ +Tests "; done +``` + +```bash +git add apps/web-ui/lib/agent/session-manager.ts apps/web-ui/lib/agent/session-manager.test.ts +git commit -m "fix(agent): make session profile names collision-proof + +Date.now() alone collides when two profiles for one account are created in the +same millisecond, so the second silently overwrote the first's section — the +same lost-profile bug the mutex was added to prevent, reached by a different +route. Surfaced as an intermittent test failure once parallel tool calls landed." +``` + --- ### Task 2: Per-run timing summary @@ -730,7 +806,16 @@ export class Semaphore { } async run(fn: () => Promise): Promise { - await this.acquire(); + // Deliberately NOT `await this.acquire()` here: awaiting any promise — + // even one already resolved — yields a microtask before continuing, so + // a caller that finds a free slot would still run `fn` one tick late. + // Callers (and this file's own tests) rely on a free slot starting + // `fn` synchronously within the `run()` call. + if (this.available > 0) { + this.available--; + } else { + await new Promise(resolve => this.waiters.push(resolve)); + } try { return await fn(); } finally { @@ -738,14 +823,6 @@ export class Semaphore { } } - private acquire(): Promise { - if (this.available > 0) { - this.available--; - return Promise.resolve(); - } - return new Promise(resolve => this.waiters.push(resolve)); - } - private release(): void { const next = this.waiters.shift(); if (next) { From 3efb5773ad9299d099b2e110d036dab3472ca964 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:02:27 +0530 Subject: [PATCH 10/42] fix(agent): make session-profile names unique within the same millisecond generateProfileName() derived the name from Date.now() alone, so two profiles for the same account created in the same millisecond collided; the second silently overwrote the first's section in the credentials file even though the write itself was correctly serialized. Append 8 hex chars of randomUUID() to guarantee uniqueness, since parallel get_aws_credentials calls (Task 3+) make same-millisecond collisions routine rather than theoretical. --- apps/web-ui/lib/agent/session-manager.test.ts | 16 ++++++++++++++++ apps/web-ui/lib/agent/session-manager.ts | 7 +++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/apps/web-ui/lib/agent/session-manager.test.ts b/apps/web-ui/lib/agent/session-manager.test.ts index 0a940c1d..cf4daeec 100644 --- a/apps/web-ui/lib/agent/session-manager.test.ts +++ b/apps/web-ui/lib/agent/session-manager.test.ts @@ -53,6 +53,22 @@ describe('createSessionProfile concurrency', () => { const leftovers = (await fs.readdir(dir)).filter(f => f.endsWith('.tmp')); expect(leftovers).toEqual([]); }); + + it('gives every profile a unique name even within the same millisecond', async () => { + // 50 back-to-back creations for ONE account will land many in the same ms. + const profiles = await Promise.all( + Array.from({ length: 50 }, () => createSessionProfile('999999999999', creds(), TENANT)), + ); + + const names = new Set(profiles.map(p => p.profileName)); + expect(names.size).toBe(50); + + // And every one of them must actually be present in the file. + const contents = await fs.readFile(getTenantCredentialsFilePath(TENANT), 'utf-8'); + for (const name of names) { + expect(contents).toContain(`[${name}]`); + } + }); }); describe('getOrCreateSessionProfile caching', () => { diff --git a/apps/web-ui/lib/agent/session-manager.ts b/apps/web-ui/lib/agent/session-manager.ts index 51e3327a..0a53994b 100644 --- a/apps/web-ui/lib/agent/session-manager.ts +++ b/apps/web-ui/lib/agent/session-manager.ts @@ -91,8 +91,11 @@ export function getTenantCredentialsFilePath(tenantId: string): string { * Generate a unique profile name */ function generateProfileName(accountId: string): string { - const timestamp = Date.now(); - return `${PROFILE_PREFIX}${accountId}_${timestamp}`; + // Date.now() alone collides when two profiles for one account are created in + // the same millisecond — which parallel get_aws_credentials calls do routinely. + // A colliding name means the second profile overwrites the first's section and + // the first caller's handle silently points at someone else's credentials. + return `${PROFILE_PREFIX}${accountId}_${Date.now()}_${randomUUID().slice(0, 8)}`; } /** From 51ff20af26e54b5f7635b44108b035224c5f1442 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:06:41 +0530 Subject: [PATCH 11/42] feat(agent): sub-agent budget config with platform ceilings Tenant-configurable sub-agent limits resolved as clamp(tenantConfig ?? default, MIN, envCeiling), applied on read so lowering a ceiling binds existing rows. SUBAGENTS_ENABLED is an absolute kill-switch. Co-Authored-By: Claude Opus 5 (1M context) --- .env.example | 13 ++ apps/web-ui/env.ts | 7 ++ apps/web-ui/lib/agent/subagent-budget.test.ts | 117 ++++++++++++++++++ apps/web-ui/lib/agent/subagent-budget.ts | 112 +++++++++++++++++ 4 files changed, 249 insertions(+) create mode 100644 apps/web-ui/lib/agent/subagent-budget.test.ts create mode 100644 apps/web-ui/lib/agent/subagent-budget.ts diff --git a/.env.example b/.env.example index 297032dc..75dbf9a7 100644 --- a/.env.example +++ b/.env.example @@ -148,6 +148,19 @@ LANGFUSE_HOST=https://cloud.langfuse.com # Skip audit DB writes in local dev # SKIP_AUDIT_LOGGING=true +# AI Ops sub-agents (Layer B). SUBAGENTS_ENABLED is an absolute platform +# kill-switch — tenant settings cannot re-enable it when false. +# The SUBAGENT_MAX_* vars are CEILINGS that lower (never raise) the built-in +# maximums a tenant may configure. +SUBAGENTS_ENABLED=false +# SUBAGENT_MAX_CONCURRENCY=6 +# SUBAGENT_MAX_PER_RUN=16 +# SUBAGENT_MAX_TOKENS_PER_RUN=1000000 +# SUBAGENT_MAX_ITERATIONS=16 +# SUBAGENT_TIMEOUT_MS=300000 +# Max concurrent shell subprocesses from execute_command (Layer A). +# TOOL_CONCURRENCY=6 + # Internal worker<->web-ui auth key (must match across both apps). # REQUIRED for the agent-ops scheduler to work: the workers process sends this # as x-internal-key when triggering a scheduled task, and web-ui only trusts the diff --git a/apps/web-ui/env.ts b/apps/web-ui/env.ts index 5e595ebf..94f9a422 100644 --- a/apps/web-ui/env.ts +++ b/apps/web-ui/env.ts @@ -50,6 +50,13 @@ export const env = createEnv({ LANGFUSE_SECRET_KEY: z.string().optional(), LLM_AUDIT: z.string().optional(), SKIP_AUDIT_LOGGING: z.string().optional(), + SUBAGENTS_ENABLED: z.string().optional(), + SUBAGENT_MAX_CONCURRENCY: z.string().optional(), + SUBAGENT_MAX_PER_RUN: z.string().optional(), + SUBAGENT_MAX_TOKENS_PER_RUN: z.string().optional(), + SUBAGENT_MAX_ITERATIONS: z.string().optional(), + SUBAGENT_TIMEOUT_MS: z.string().optional(), + TOOL_CONCURRENCY: z.string().optional(), // Storage APP_BUCKET_NAME: z.string().optional(), diff --git a/apps/web-ui/lib/agent/subagent-budget.test.ts b/apps/web-ui/lib/agent/subagent-budget.test.ts new file mode 100644 index 00000000..96e12ffc --- /dev/null +++ b/apps/web-ui/lib/agent/subagent-budget.test.ts @@ -0,0 +1,117 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('@/lib/tenant-config-service', () => ({ + TenantConfigService: { getConfig: vi.fn() }, +})); + +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { + clampBudget, + resolveSubagentBudget, + validateBudgetInput, + platformSubagentsEnabled, + BUDGET_BOUNDS, +} from './subagent-budget'; + +beforeEach(() => { + vi.clearAllMocks(); + process.env.SUBAGENTS_ENABLED = 'true'; +}); +afterEach(() => { + delete process.env.SUBAGENTS_ENABLED; + delete process.env.SUBAGENT_MAX_CONCURRENCY; +}); + +describe('platformSubagentsEnabled', () => { + it('is false unless SUBAGENTS_ENABLED is exactly "true"', () => { + delete process.env.SUBAGENTS_ENABLED; + expect(platformSubagentsEnabled()).toBe(false); + process.env.SUBAGENTS_ENABLED = 'false'; + expect(platformSubagentsEnabled()).toBe(false); + process.env.SUBAGENTS_ENABLED = 'true'; + expect(platformSubagentsEnabled()).toBe(true); + }); +}); + +describe('clampBudget', () => { + it('returns defaults for null input', () => { + const budget = clampBudget(null); + expect(budget.maxConcurrentSubagents).toBe(BUDGET_BOUNDS.maxConcurrentSubagents.default); + expect(budget.enabled).toBe(false); + }); + + it('clamps a value above the ceiling down', () => { + expect(clampBudget({ maxConcurrentSubagents: 50 }).maxConcurrentSubagents) + .toBe(BUDGET_BOUNDS.maxConcurrentSubagents.max); + }); + + it('clamps a value below the floor up', () => { + expect(clampBudget({ maxSubagentsPerRun: 0 }).maxSubagentsPerRun) + .toBe(BUDGET_BOUNDS.maxSubagentsPerRun.min); + }); + + it('rounds non-integers', () => { + expect(clampBudget({ subagentMaxIterations: 7.6 }).subagentMaxIterations).toBe(8); + }); + + it('falls back to the default for non-numeric values', () => { + expect(clampBudget({ subagentTimeoutMs: 'abc' as unknown as number }).subagentTimeoutMs) + .toBe(BUDGET_BOUNDS.subagentTimeoutMs.default); + }); + + it('honours an env ceiling lower than the built-in max', () => { + process.env.SUBAGENT_MAX_CONCURRENCY = '2'; + expect(clampBudget({ maxConcurrentSubagents: 6 }).maxConcurrentSubagents).toBe(2); + }); +}); + +describe('resolveSubagentBudget', () => { + it('returns a disabled budget when the platform kill-switch is off', async () => { + process.env.SUBAGENTS_ENABLED = 'false'; + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ enabled: true } as never); + + expect((await resolveSubagentBudget('t1')).enabled).toBe(false); + }); + + it('returns tenant config clamped when the platform allows it', async () => { + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ + enabled: true, maxConcurrentSubagents: 99, + } as never); + + const budget = await resolveSubagentBudget('t1'); + expect(budget.enabled).toBe(true); + expect(budget.maxConcurrentSubagents).toBe(BUDGET_BOUNDS.maxConcurrentSubagents.max); + }); + + it('returns defaults (disabled) when the tenant has no config', async () => { + vi.mocked(TenantConfigService.getConfig).mockResolvedValue(null as never); + expect((await resolveSubagentBudget('t1')).enabled).toBe(false); + }); + + it('never throws when the config read fails', async () => { + vi.mocked(TenantConfigService.getConfig).mockRejectedValue(new Error('db down')); + await expect(resolveSubagentBudget('t1')).resolves.toMatchObject({ enabled: false }); + }); +}); + +describe('validateBudgetInput', () => { + it('accepts a well-formed payload', () => { + expect(validateBudgetInput({ + enabled: true, maxConcurrentSubagents: 3, maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400000, subagentMaxIterations: 8, subagentTimeoutMs: 180000, + })).toBeNull(); + }); + + it('rejects a non-object', () => { + expect(validateBudgetInput(null)).toMatch(/object/i); + }); + + it('rejects a non-boolean enabled', () => { + expect(validateBudgetInput({ enabled: 'yes' })).toMatch(/enabled/i); + }); + + it('rejects an out-of-range number with a message naming the field', () => { + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: 999 })) + .toMatch(/maxConcurrentSubagents/); + }); +}); diff --git a/apps/web-ui/lib/agent/subagent-budget.ts b/apps/web-ui/lib/agent/subagent-budget.ts new file mode 100644 index 00000000..8976c6c0 --- /dev/null +++ b/apps/web-ui/lib/agent/subagent-budget.ts @@ -0,0 +1,112 @@ +/** + * Sub-agent budget configuration. + * + * web-ui runs as a SHARED ECS task, so these limits cannot be purely + * tenant-controlled: one tenant setting concurrency to 50 would saturate the + * container's event loop for every co-tenant and flood their Bedrock quota. + * Env vars are therefore CEILINGS, not overrides: + * + * effective = clamp(tenantConfig ?? default, MIN, envCeiling) + * + * Clamping runs on READ (not only on write) so lowering a ceiling retroactively + * binds config rows written while it was higher. + * + * SUBAGENTS_ENABLED is an absolute platform kill-switch: tenant config can + * never re-enable the feature when it is off. + */ +import { TenantConfigService } from '@/lib/tenant-config-service'; + +export const SUBAGENT_CONFIG_KEY = 'aiops-subagents'; + +export interface SubagentBudgetConfig { + enabled: boolean; + maxConcurrentSubagents: number; + maxSubagentsPerRun: number; + maxSubagentTokensPerRun: number; + subagentMaxIterations: number; + subagentTimeoutMs: number; +} + +type NumericKey = keyof Omit; + +interface Bound { min: number; max: number; default: number; envCeiling: string } + +/** + * `max` is the hard platform ceiling. `envCeiling` names an env var that may + * lower it further (never raise it) for a given deployment. + */ +const BOUNDS_SPEC: Record = { + maxConcurrentSubagents: { min: 1, max: 6, default: 3, envCeiling: 'SUBAGENT_MAX_CONCURRENCY' }, + maxSubagentsPerRun: { min: 1, max: 16, default: 8, envCeiling: 'SUBAGENT_MAX_PER_RUN' }, + maxSubagentTokensPerRun: { min: 50_000, max: 1_000_000, default: 400_000, envCeiling: 'SUBAGENT_MAX_TOKENS_PER_RUN' }, + subagentMaxIterations: { min: 2, max: 16, default: 8, envCeiling: 'SUBAGENT_MAX_ITERATIONS' }, + subagentTimeoutMs: { min: 30_000, max: 300_000, default: 180_000, envCeiling: 'SUBAGENT_TIMEOUT_MS' }, +}; + +export const BUDGET_BOUNDS = BOUNDS_SPEC; + +/** The effective ceiling: the built-in max, lowered (never raised) by env. */ +function ceilingFor(key: NumericKey): number { + const spec = BOUNDS_SPEC[key]; + const fromEnv = Number(process.env[spec.envCeiling]); + if (!Number.isFinite(fromEnv) || fromEnv <= 0) return spec.max; + return Math.min(spec.max, Math.round(fromEnv)); +} + +/** Absolute platform kill-switch. Defaults to OFF. */ +export function platformSubagentsEnabled(): boolean { + return process.env.SUBAGENTS_ENABLED === 'true'; +} + +export function clampBudget(input: Partial | null): SubagentBudgetConfig { + const result: Record = { enabled: input?.enabled === true }; + + for (const key of Object.keys(BOUNDS_SPEC) as NumericKey[]) { + const spec = BOUNDS_SPEC[key]; + const raw = Number(input?.[key]); + const value = Number.isFinite(raw) ? Math.round(raw) : spec.default; + result[key] = Math.min(ceilingFor(key), Math.max(spec.min, value)); + } + + return result as unknown as SubagentBudgetConfig; +} + +/** + * Read a tenant's budget, clamped. Never throws: a config-store failure must + * degrade to "sub-agents off", not break the chat run. + */ +export async function resolveSubagentBudget(tenantId: string): Promise { + if (!platformSubagentsEnabled()) { + return { ...clampBudget(null), enabled: false }; + } + + const stored = await TenantConfigService + .getConfig>(SUBAGENT_CONFIG_KEY, tenantId) + .catch(() => null); + + return clampBudget(stored); +} + +/** Validate a PUT payload. Returns an error string for a 400, or null. */ +export function validateBudgetInput(input: unknown): string | null { + if (!input || typeof input !== 'object' || Array.isArray(input)) { + return 'Request body must be an object'; + } + const body = input as Record; + + if (typeof body.enabled !== 'boolean') { + return 'enabled must be a boolean'; + } + + for (const key of Object.keys(BOUNDS_SPEC) as NumericKey[]) { + if (body[key] === undefined) continue; + const spec = BOUNDS_SPEC[key]; + const value = Number(body[key]); + const ceiling = ceilingFor(key); + if (!Number.isInteger(value) || value < spec.min || value > ceiling) { + return `${key} must be an integer between ${spec.min} and ${ceiling}`; + } + } + + return null; +} From 500d57697b25fa17e19fab5ed785ebdb63a36dbb Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:10:40 +0530 Subject: [PATCH 12/42] docs(aiops): harden subagent-budget trust boundary in the plan - validateBudgetInput type-checks before coercing; it guards the PUT /api/settings/aiops boundary in Task 5, where Number() would have accepted true as 1 and "3" as 3. - Add the missing test for the load-bearing isolation invariant: an env ceiling ABOVE the built-in max must be ignored. Correct in code, untested. - Add a test for non-number rejection. - Drop stale Bedrock-specific wording (multi-provider port already landed). Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-07-26-aiops-subagent-orchestration.md | 28 ++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md index 8eb2e451..81683256 100644 --- a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md +++ b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md @@ -1025,6 +1025,15 @@ describe('clampBudget', () => { process.env.SUBAGENT_MAX_CONCURRENCY = '2'; expect(clampBudget({ maxConcurrentSubagents: 6 }).maxConcurrentSubagents).toBe(2); }); + + it('IGNORES an env ceiling above the built-in max', () => { + // The load-bearing multi-tenant isolation invariant: web-ui is a shared + // ECS task, so no operator misconfiguration may let a tenant exceed the + // hard cap and saturate the box for co-tenants. + process.env.SUBAGENT_MAX_CONCURRENCY = '100'; + expect(clampBudget({ maxConcurrentSubagents: 100 }).maxConcurrentSubagents) + .toBe(BUDGET_BOUNDS.maxConcurrentSubagents.max); + }); }); describe('resolveSubagentBudget', () => { @@ -1076,6 +1085,17 @@ describe('validateBudgetInput', () => { expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: 999 })) .toMatch(/maxConcurrentSubagents/); }); + + it('rejects non-number types rather than coercing them', () => { + // This function guards the PUT /api/settings/aiops trust boundary, so it + // must not accept `true` as 1 or "3" as 3. + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: true })) + .toMatch(/maxConcurrentSubagents/); + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: '3' })) + .toMatch(/maxConcurrentSubagents/); + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: null })) + .toMatch(/maxConcurrentSubagents/); + }); }); ``` @@ -1094,7 +1114,7 @@ Create `apps/web-ui/lib/agent/subagent-budget.ts`: * * web-ui runs as a SHARED ECS task, so these limits cannot be purely * tenant-controlled: one tenant setting concurrency to 50 would saturate the - * container's event loop for every co-tenant and flood their Bedrock quota. + * container's event loop for every co-tenant and flood their LLM provider quota. * Env vars are therefore CEILINGS, not overrides: * * effective = clamp(tenantConfig ?? default, MIN, envCeiling) @@ -1192,6 +1212,12 @@ export function validateBudgetInput(input: unknown): string | null { for (const key of Object.keys(BOUNDS_SPEC) as NumericKey[]) { if (body[key] === undefined) continue; const spec = BOUNDS_SPEC[key]; + // Type-check BEFORE coercing: this function is the trust boundary for the + // PUT /api/settings/aiops route, and Number() would silently accept + // `true` as 1 or "3" as 3. + if (typeof body[key] !== 'number') { + return `${key} must be a number`; + } const value = Number(body[key]); const ceiling = ceilingFor(key); if (!Number.isInteger(value) || value < spec.min || value > ceiling) { From 6fb173f8188655f6f3fc7817af4778068efb2dde Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:12:09 +0530 Subject: [PATCH 13/42] fix(agent): type-check before coercing in validateBudgetInput Number(body[key]) silently accepted booleans (true -> 1) and numeric strings ("3" -> 3). Since this function is about to become the trust boundary for PUT /api/settings/aiops (Task 5), require typeof number before coercion. Also covers the load-bearing invariant that an env ceiling set ABOVE the built-in max is ignored, and fixes stale "Bedrock quota" wording now that the platform is multi-provider. Co-Authored-By: Claude Opus 5 (1M context) --- apps/web-ui/lib/agent/subagent-budget.test.ts | 20 +++++++++++++++++++ apps/web-ui/lib/agent/subagent-budget.ts | 8 +++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/apps/web-ui/lib/agent/subagent-budget.test.ts b/apps/web-ui/lib/agent/subagent-budget.test.ts index 96e12ffc..9d057960 100644 --- a/apps/web-ui/lib/agent/subagent-budget.test.ts +++ b/apps/web-ui/lib/agent/subagent-budget.test.ts @@ -63,6 +63,15 @@ describe('clampBudget', () => { process.env.SUBAGENT_MAX_CONCURRENCY = '2'; expect(clampBudget({ maxConcurrentSubagents: 6 }).maxConcurrentSubagents).toBe(2); }); + + it('IGNORES an env ceiling above the built-in max', () => { + // The load-bearing multi-tenant isolation invariant: web-ui is a shared + // ECS task, so no operator misconfiguration may let a tenant exceed the + // hard cap and saturate the box for co-tenants. + process.env.SUBAGENT_MAX_CONCURRENCY = '100'; + expect(clampBudget({ maxConcurrentSubagents: 100 }).maxConcurrentSubagents) + .toBe(BUDGET_BOUNDS.maxConcurrentSubagents.max); + }); }); describe('resolveSubagentBudget', () => { @@ -114,4 +123,15 @@ describe('validateBudgetInput', () => { expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: 999 })) .toMatch(/maxConcurrentSubagents/); }); + + it('rejects non-number types rather than coercing them', () => { + // This function guards the PUT /api/settings/aiops trust boundary, so it + // must not accept `true` as 1 or "3" as 3. + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: true })) + .toMatch(/maxConcurrentSubagents/); + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: '3' })) + .toMatch(/maxConcurrentSubagents/); + expect(validateBudgetInput({ enabled: true, maxConcurrentSubagents: null })) + .toMatch(/maxConcurrentSubagents/); + }); }); diff --git a/apps/web-ui/lib/agent/subagent-budget.ts b/apps/web-ui/lib/agent/subagent-budget.ts index 8976c6c0..959e9d96 100644 --- a/apps/web-ui/lib/agent/subagent-budget.ts +++ b/apps/web-ui/lib/agent/subagent-budget.ts @@ -3,7 +3,7 @@ * * web-ui runs as a SHARED ECS task, so these limits cannot be purely * tenant-controlled: one tenant setting concurrency to 50 would saturate the - * container's event loop for every co-tenant and flood their Bedrock quota. + * container's event loop for every co-tenant and flood their LLM provider quota. * Env vars are therefore CEILINGS, not overrides: * * effective = clamp(tenantConfig ?? default, MIN, envCeiling) @@ -100,6 +100,12 @@ export function validateBudgetInput(input: unknown): string | null { for (const key of Object.keys(BOUNDS_SPEC) as NumericKey[]) { if (body[key] === undefined) continue; + // Type-check BEFORE coercing: this function is the trust boundary for the + // PUT /api/settings/aiops route, and Number() would silently accept + // `true` as 1 or "3" as 3. + if (typeof body[key] !== 'number') { + return `${key} must be a number`; + } const spec = BOUNDS_SPEC[key]; const value = Number(body[key]); const ceiling = ceilingFor(key); From d81d98ab048f3bd70ac00f1637b9477a38f0cd3c Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:15:37 +0530 Subject: [PATCH 14/42] feat(api): GET/PUT /api/settings/aiops for sub-agent budget Reuses the Agent RBAC subject (already mapped to the AIOps module) and stores the clamped budget under the aiops-subagents tenant-config key. --- .../app/api/settings/aiops/route.test.ts | 99 +++++++++++++++++ apps/web-ui/app/api/settings/aiops/route.ts | 101 ++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 apps/web-ui/app/api/settings/aiops/route.test.ts create mode 100644 apps/web-ui/app/api/settings/aiops/route.ts diff --git a/apps/web-ui/app/api/settings/aiops/route.test.ts b/apps/web-ui/app/api/settings/aiops/route.test.ts new file mode 100644 index 00000000..9b5d8c23 --- /dev/null +++ b/apps/web-ui/app/api/settings/aiops/route.test.ts @@ -0,0 +1,99 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('@/lib/rbac/authorize', () => ({ authorize: vi.fn() })); +vi.mock('@/lib/auth-session', () => ({ getSessionTenantId: vi.fn() })); +vi.mock('next-auth', () => ({ getServerSession: vi.fn() })); +vi.mock('@/lib/auth-options', () => ({ authOptions: {} })); +vi.mock('@/lib/tenant-config-service', () => ({ + TenantConfigService: { getConfig: vi.fn(), saveConfig: vi.fn() }, +})); +vi.mock('@/lib/audit-service', () => ({ + AuditService: { logUserAction: vi.fn().mockResolvedValue(undefined) }, +})); + +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getServerSession } from 'next-auth'; +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { AuditService } from '@/lib/audit-service'; +import { GET, PUT } from './route'; + +const putRequest = (body: unknown) => + ({ json: async () => body }) as unknown as Parameters[0]; + +beforeEach(() => { + vi.clearAllMocks(); + process.env.SUBAGENTS_ENABLED = 'true'; + vi.mocked(authorize).mockResolvedValue(null as never); + vi.mocked(getSessionTenantId).mockResolvedValue('t1' as never); + vi.mocked(getServerSession).mockResolvedValue({ user: { email: 'a@b.com' } } as never); + vi.mocked(TenantConfigService.getConfig).mockResolvedValue(null as never); +}); +afterEach(() => { delete process.env.SUBAGENTS_ENABLED; }); + +describe('GET /api/settings/aiops', () => { + it('returns the effective budget, bounds, and platform flag', async () => { + const body = await (await GET()).json(); + + expect(body.success).toBe(true); + expect(body.data.platformEnabled).toBe(true); + expect(body.data.budget.maxConcurrentSubagents).toBe(3); + expect(body.data.bounds.maxConcurrentSubagents.max).toBe(6); + }); + + it('propagates an RBAC denial', async () => { + const denied = { status: 403 }; + vi.mocked(authorize).mockResolvedValue(denied as never); + expect(await GET()).toBe(denied); + }); + + it('403s with no tenant context', async () => { + vi.mocked(getSessionTenantId).mockResolvedValue(null as never); + expect((await GET()).status).toBe(403); + }); +}); + +describe('PUT /api/settings/aiops', () => { + const valid = { + enabled: true, maxConcurrentSubagents: 4, maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400000, subagentMaxIterations: 8, subagentTimeoutMs: 180000, + }; + + it('saves a valid payload and audits it', async () => { + const res = await PUT(putRequest(valid)); + const body = await res.json(); + + expect(body.success).toBe(true); + expect(body.data.budget.maxConcurrentSubagents).toBe(4); + expect(TenantConfigService.saveConfig).toHaveBeenCalledWith( + 'aiops-subagents', expect.objectContaining({ maxConcurrentSubagents: 4 }), 't1', 'a@b.com', + ); + expect(AuditService.logUserAction).toHaveBeenCalled(); + }); + + it('400s on an out-of-range value', async () => { + const res = await PUT(putRequest({ ...valid, maxConcurrentSubagents: 999 })); + expect(res.status).toBe(400); + expect((await res.json()).error).toMatch(/maxConcurrentSubagents/); + }); + + it('400s when enabled is missing', async () => { + const { enabled, ...rest } = valid; + expect((await PUT(putRequest(rest))).status).toBe(400); + }); + + it('persists the clamped value, not the raw one', async () => { + process.env.SUBAGENT_MAX_CONCURRENCY = '2'; + await PUT(putRequest({ ...valid, maxConcurrentSubagents: 2 })); + expect(TenantConfigService.saveConfig).toHaveBeenCalledWith( + 'aiops-subagents', expect.objectContaining({ maxConcurrentSubagents: 2 }), 't1', 'a@b.com', + ); + delete process.env.SUBAGENT_MAX_CONCURRENCY; + }); + + it('propagates an RBAC denial', async () => { + const denied = { status: 403 }; + vi.mocked(authorize).mockResolvedValue(denied as never); + expect(await PUT(putRequest(valid))).toBe(denied); + }); +}); diff --git a/apps/web-ui/app/api/settings/aiops/route.ts b/apps/web-ui/app/api/settings/aiops/route.ts new file mode 100644 index 00000000..aac85abf --- /dev/null +++ b/apps/web-ui/app/api/settings/aiops/route.ts @@ -0,0 +1,101 @@ +import { NextRequest, NextResponse } from 'next/server'; +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getServerSession } from 'next-auth'; +import { authOptions } from '@/lib/auth-options'; +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { AuditService } from '@/lib/audit-service'; +import { + SUBAGENT_CONFIG_KEY, + BUDGET_BOUNDS, + clampBudget, + platformSubagentsEnabled, + resolveSubagentBudget, + validateBudgetInput, +} from '@/lib/agent/subagent-budget'; + +export async function GET() { + console.log('API - GET /api/settings/aiops - Fetching sub-agent budget'); + + const authError = await authorize('read', 'Agent'); + if (authError) return authError; + + try { + const tenantId = await getSessionTenantId(); + if (!tenantId) { + return NextResponse.json({ success: false, error: 'No tenant context' }, { status: 403 }); + } + + const budget = await resolveSubagentBudget(tenantId); + + return NextResponse.json({ + success: true, + data: { + budget, + // The UI renders sliders bounded by these, and explains why a + // ceiling is what it is rather than silently clipping input. + bounds: BUDGET_BOUNDS, + platformEnabled: platformSubagentsEnabled(), + }, + }); + } catch (error) { + console.error('API - Error fetching AI Ops settings:', error); + return NextResponse.json( + { success: false, error: error instanceof Error ? error.message : 'Failed to fetch AI Ops settings' }, + { status: 500 }, + ); + } +} + +export async function PUT(request: NextRequest) { + console.log('API - PUT /api/settings/aiops - Saving sub-agent budget'); + + const authError = await authorize('update', 'Agent'); + if (authError) return authError; + + try { + const tenantId = await getSessionTenantId(); + if (!tenantId) { + return NextResponse.json({ success: false, error: 'No tenant context' }, { status: 403 }); + } + + const body = await request.json(); + const validationError = validateBudgetInput(body); + if (validationError) { + return NextResponse.json({ success: false, error: validationError }, { status: 400 }); + } + + // Persist the CLAMPED value so a stored row can never exceed the ceiling + // that was in force when it was written. + const budget = clampBudget(body); + + const session = await getServerSession(authOptions); + const updatedBy = session?.user?.email || 'api-user'; + + await TenantConfigService.saveConfig(SUBAGENT_CONFIG_KEY, budget, tenantId, updatedBy); + + await AuditService.logUserAction({ + eventType: 'aiops.subagents.settings.updated', + severity: 'medium', + apiRoute: 'PUT /api/settings/aiops', + httpMethod: 'PUT', + action: 'Update AI Ops Sub-Agent Settings', + resourceType: 'settings', + resourceId: SUBAGENT_CONFIG_KEY, + resourceName: 'AI Ops Sub-Agent Budget', + user: updatedBy, + userType: 'user', + status: 'success', + details: `Sub-agents ${budget.enabled ? 'enabled' : 'disabled'}; concurrency=${budget.maxConcurrentSubagents}, perRun=${budget.maxSubagentsPerRun}, tokenBudget=${budget.maxSubagentTokensPerRun}`, + tenantId, + }); + + return NextResponse.json({ success: true, data: { budget } }); + } catch (error) { + console.error('API - Error saving AI Ops settings:', error); + return NextResponse.json( + { success: false, error: error instanceof Error ? error.message : 'Failed to save AI Ops settings' }, + { status: 500 }, + ); + } +} From 41a244670280a874815a48ff33b8f5670966a9cc Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:19:02 +0530 Subject: [PATCH 15/42] feat(ui): AI Ops sub-agent budget settings panel --- .../app/agent-ops/settings/subagents/page.tsx | 11 + .../settings/aiops-subagent-settings.tsx | 214 ++++++++++++++++++ apps/web-ui/lib/queries/aiops-settings.ts | 58 +++++ apps/web-ui/lib/queries/query-keys.ts | 4 + 4 files changed, 287 insertions(+) create mode 100644 apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx create mode 100644 apps/web-ui/components/settings/aiops-subagent-settings.tsx create mode 100644 apps/web-ui/lib/queries/aiops-settings.ts diff --git a/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx b/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx new file mode 100644 index 00000000..6b184a5b --- /dev/null +++ b/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx @@ -0,0 +1,11 @@ +'use client'; + +import { AiopsSubagentSettings } from '@/components/settings/aiops-subagent-settings'; + +export default function AiopsSubagentSettingsPage() { + return ( +
+ +
+ ); +} diff --git a/apps/web-ui/components/settings/aiops-subagent-settings.tsx b/apps/web-ui/components/settings/aiops-subagent-settings.tsx new file mode 100644 index 00000000..32a4d994 --- /dev/null +++ b/apps/web-ui/components/settings/aiops-subagent-settings.tsx @@ -0,0 +1,214 @@ +'use client'; + +import { useEffect } from 'react'; +import { useForm } from 'react-hook-form'; +import { zodResolver } from '@hookform/resolvers/zod'; +import { z } from 'zod'; +import { toast } from 'sonner'; +import { Bot, Gauge, Loader2, Save } from 'lucide-react'; + +import { + useAiopsSubagentSettings, + useSaveAiopsSubagentSettings, + type SubagentBudget, +} from '@/lib/queries/aiops-settings'; +import { Button } from '@/components/ui/button'; +import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card'; +import { Input } from '@/components/ui/input'; +import { Label } from '@/components/ui/label'; +import { Switch } from '@/components/ui/switch'; +import { Alert, AlertDescription } from '@/components/ui/alert'; +import { Spinner } from '@/components/ui/spinner'; + +const schema = z.object({ + enabled: z.boolean(), + maxConcurrentSubagents: z.number().int(), + maxSubagentsPerRun: z.number().int(), + maxSubagentTokensPerRun: z.number().int(), + subagentMaxIterations: z.number().int(), + subagentTimeoutMs: z.number().int(), +}); + +type FormValues = z.infer; + +const FIELDS: Array<{ + name: keyof Omit; + label: string; + help: string; +}> = [ + { + name: 'maxConcurrentSubagents', + label: 'Concurrent sub-agents', + help: 'How many sub-agents may run at the same time. Higher finishes fan-out work faster but increases the chance of provider throttling.', + }, + { + name: 'maxSubagentsPerRun', + label: 'Sub-agents per run', + help: 'Total sub-agents a single chat run may dispatch before it falls back to working serially.', + }, + { + name: 'maxSubagentTokensPerRun', + label: 'Token budget per run', + help: 'Combined sub-agent token spend allowed in one run. When exhausted, the agent completes the work itself instead of failing.', + }, + { + name: 'subagentMaxIterations', + label: 'Iterations per sub-agent', + help: 'Reasoning/tool laps a single sub-agent may take before it must report what it has found.', + }, + { + name: 'subagentTimeoutMs', + label: 'Sub-agent timeout (ms)', + help: 'Wall-clock limit for one sub-agent. On timeout it returns partial findings rather than failing the run.', + }, +]; + +export function AiopsSubagentSettings() { + const { data, isLoading, error } = useAiopsSubagentSettings(); + const saveMutation = useSaveAiopsSubagentSettings(); + + const form = useForm({ + resolver: zodResolver(schema), + defaultValues: { + enabled: false, + maxConcurrentSubagents: 3, + maxSubagentsPerRun: 8, + maxSubagentTokensPerRun: 400000, + subagentMaxIterations: 8, + subagentTimeoutMs: 180000, + }, + }); + + useEffect(() => { + if (data?.budget) form.reset(data.budget); + }, [data, form]); + + if (isLoading) { + return ( +
+ +
+ ); + } + + if (error) { + return ( + + {(error as Error).message} + + ); + } + + const platformEnabled = data?.platformEnabled ?? false; + const bounds = data?.bounds; + + const onSubmit = async (values: FormValues) => { + // Bounds are enforced server-side too; this only produces a friendlier message. + for (const field of FIELDS) { + const bound = bounds?.[field.name]; + const value = values[field.name]; + if (bound && (value < bound.min || value > bound.max)) { + toast.error(`${field.label} must be between ${bound.min} and ${bound.max}`); + return; + } + } + try { + await saveMutation.mutateAsync(values); + toast.success('Sub-agent settings saved — new runs will use this configuration'); + } catch (err) { + toast.error(err instanceof Error ? err.message : 'Failed to save sub-agent settings'); + } + }; + + return ( +
+
+

Sub-Agents

+

+ Sub-agents let one AI Ops run investigate several accounts or regions at the same time. + They are read-only: anything that changes infrastructure still runs on the main agent + under your normal approval flow. +

+
+ + {!platformEnabled && ( + + + Sub-agents are disabled for this deployment. These settings are saved but will not take + effect until an administrator enables the feature. + + + )} + + + + + + Enable sub-agents + + + When off, every run executes serially exactly as before. + + + +
+ form.setValue('enabled', checked, { shouldDirty: true })} + disabled={!platformEnabled} + /> + +
+ + + + + + + + Budget limits + + + These bound how much work and spend one run may fan out. Exceeding a limit never fails + the run — the agent finishes the work serially instead. + + + + {FIELDS.map((field) => { + const bound = bounds?.[field.name]; + return ( +
+ + +

{field.help}

+ {bound && ( +

+ Allowed: {bound.min}–{bound.max}. Default {bound.default}. +

+ )} +
+ ); + })} + + + + +
+ ); +} diff --git a/apps/web-ui/lib/queries/aiops-settings.ts b/apps/web-ui/lib/queries/aiops-settings.ts new file mode 100644 index 00000000..b095bfce --- /dev/null +++ b/apps/web-ui/lib/queries/aiops-settings.ts @@ -0,0 +1,58 @@ +'use client'; + +/** + * TanStack Query hooks for the AI Ops sub-agent budget (per-tenant). + */ +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { queryKeys } from './query-keys'; + +export interface SubagentBudget { + enabled: boolean; + maxConcurrentSubagents: number; + maxSubagentsPerRun: number; + maxSubagentTokensPerRun: number; + subagentMaxIterations: number; + subagentTimeoutMs: number; +} + +export interface SubagentBound { min: number; max: number; default: number } + +export interface AiopsSubagentSettings { + budget: SubagentBudget; + bounds: Record, SubagentBound>; + /** False when SUBAGENTS_ENABLED is off for the whole deployment. */ + platformEnabled: boolean; +} + +export function useAiopsSubagentSettings() { + return useQuery({ + queryKey: queryKeys.aiopsSettings.subagents(), + queryFn: async (): Promise => { + const res = await fetch('/api/settings/aiops'); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) { + throw new Error(json.error || 'Failed to load AI Ops settings'); + } + return json.data as AiopsSubagentSettings; + }, + }); +} + +export function useSaveAiopsSubagentSettings() { + const qc = useQueryClient(); + return useMutation({ + mutationFn: async (body: SubagentBudget): Promise => { + const res = await fetch('/api/settings/aiops', { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) { + throw new Error(json.error || 'Failed to save AI Ops settings'); + } + return json.data.budget as SubagentBudget; + }, + onSuccess: () => qc.invalidateQueries({ queryKey: queryKeys.aiopsSettings.all }), + }); +} diff --git a/apps/web-ui/lib/queries/query-keys.ts b/apps/web-ui/lib/queries/query-keys.ts index 19f084b7..1845262b 100644 --- a/apps/web-ui/lib/queries/query-keys.ts +++ b/apps/web-ui/lib/queries/query-keys.ts @@ -100,6 +100,10 @@ export const queryKeys = { all: ['threads'] as const, lists: () => [...queryKeys.threads.all, 'list'] as const, }, + aiopsSettings: { + all: ['aiops-settings'] as const, + subagents: () => [...queryKeys.aiopsSettings.all, 'subagents'] as const, + }, dashboard: { all: ['dashboard'] as const, hero: (range: string) => [...queryKeys.dashboard.all, 'hero', range] as const, From a84c71771bb144ea317c9a5209fb89a59bc3d545 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 02:24:48 +0530 Subject: [PATCH 16/42] docs(aiops): fix settings-panel defects in the plan - Important: blank number input made Save a silent no-op. Zod rejects NaN client-side but the panel rendered no formState.errors and passed no invalid handler, so the click did nothing with zero feedback. Add an onInvalid toast and per-field error text. - Alert copy contradicted the spec and the code: the panel is read-only when the platform kill-switch is off, not 'saved but inert'. - Use the Spinner primitive instead of bespoke Loader2 markup, per CLAUDE.md. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-07-26-aiops-subagent-orchestration.md | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md index 81683256..d424f11b 100644 --- a/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md +++ b/docs/superpowers/plans/2026-07-26-aiops-subagent-orchestration.md @@ -1632,7 +1632,7 @@ import { useForm } from 'react-hook-form'; import { zodResolver } from '@hookform/resolvers/zod'; import { z } from 'zod'; import { toast } from 'sonner'; -import { Bot, Gauge, Loader2, Save } from 'lucide-react'; +import { Bot, Gauge, Save } from 'lucide-react'; import { useAiopsSubagentSettings, @@ -1729,6 +1729,12 @@ export function AiopsSubagentSettings() { const platformEnabled = data?.platformEnabled ?? false; const bounds = data?.bounds; + // Zod rejects a blank number input as NaN before onSubmit ever runs. Without an + // invalid handler the click is a silent no-op with zero feedback, so surface it. + const onInvalid = () => { + toast.error('Some values are invalid — check the highlighted fields.'); + }; + const onSubmit = async (values: FormValues) => { // Bounds are enforced server-side too; this only produces a friendlier message. for (const field of FIELDS) { @@ -1748,7 +1754,7 @@ export function AiopsSubagentSettings() { }; return ( -
+

Sub-Agents

@@ -1761,8 +1767,8 @@ export function AiopsSubagentSettings() { {!platformEnabled && ( - Sub-agents are disabled for this deployment. These settings are saved but will not take - effect until an administrator enables the feature. + Sub-agents are disabled for this deployment, so these settings are read-only. An + administrator must enable the feature before they can be changed. )} @@ -1822,6 +1828,11 @@ export function AiopsSubagentSettings() { Allowed: {bound.min}–{bound.max}. Default {bound.default}.

)} + {form.formState.errors[field.name] && ( +

+ Enter a whole number between {bound?.min} and {bound?.max}. +

+ )}
); })} @@ -1830,7 +1841,7 @@ export function AiopsSubagentSettings() { + + {expanded && subagent.summary && ( +
+

Task

+

{subagent.task}

+

Findings

+

{subagent.summary}

+
+ )} + + ); +} diff --git a/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx b/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx index bc4b6880..22ed3fed 100644 --- a/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx +++ b/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx @@ -19,6 +19,7 @@ const EMPTY_RUN_STATE: RunState = { hasStructuredData: false, hasApprovalData: false, tokenUsage: { input: 0, output: 0 }, + subagents: [], } function baseProps(events: TranscriptEvent[]) { diff --git a/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx b/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx index 0989d44e..7b5e24d4 100644 --- a/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx +++ b/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx @@ -30,6 +30,7 @@ const EMPTY_RUN_STATE: RunState = { hasStructuredData: false, hasApprovalData: false, tokenUsage: { input: 0, output: 0 }, + subagents: [], } function planOf(doneCount: number, total: number): RunState['plan'] { diff --git a/apps/web-ui/components/agent/workspace/session-view.tsx b/apps/web-ui/components/agent/workspace/session-view.tsx index e6a555e7..0435cdfc 100644 --- a/apps/web-ui/components/agent/workspace/session-view.tsx +++ b/apps/web-ui/components/agent/workspace/session-view.tsx @@ -369,7 +369,7 @@ export function SessionView({ threadId, ownerUserId, active, onStatusChange, onT )} >
- +
From facf53b875b5fa4f7413f69eae2e8d4ca643a0b7 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 18:32:08 +0530 Subject: [PATCH 36/42] feat(chat): persist redacted sub-agent transcripts for expansion after reload MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Transcripts live in agent_subagent_runs (30-day TTL) and are fetched on demand, never carried in the dispatch_agent ToolMessage — that message enters the orchestrator's context and would undo the isolation sub-agents exist to give. Secrets are redacted in SubagentRunPostgresRepository.save(), not at the call site, so no present or future caller can bypass it: aws_read permits `lambda get-function-configuration`, whose Environment.Variables are plaintext, and those must never reach at-rest storage in a multi-tenant database. data-subagent parts are not persisted, so a reloaded thread has no live sub-agent state at all. RunState now carries usedSubagents (set by a persisted dispatch_agent tool call), which is what lets the rail rebuild the cards from agent_subagent_runs instead of rendering nothing. Co-Authored-By: Claude Opus 5 (1M context) --- apps/web-ui/app/api/chat/route.ts | 21 ++- .../chat/subagents/[threadId]/route.test.ts | 57 ++++++ .../api/chat/subagents/[threadId]/route.ts | 42 +++++ .../agent/chat/__tests__/run-rail.test.tsx | 64 ++++++- .../agent/chat/__tests__/run-state.test.ts | 44 +++++ .../chat/__tests__/subagent-card.test.tsx | 93 ++++++++++ .../web-ui/components/agent/chat/run-rail.tsx | 28 ++- .../web-ui/components/agent/chat/run-state.ts | 13 ++ .../components/agent/chat/subagent-card.tsx | 29 ++- .../workspace/__tests__/agent-turn.test.tsx | 1 + .../__tests__/transcript-header.test.tsx | 1 + .../agent/__tests__/subagent-redact.test.ts | 148 ++++++++++++++++ apps/web-ui/lib/agent/subagent-redact.ts | 167 ++++++++++++++++++ apps/web-ui/lib/db/pg-config.ts | 1 + .../lib/db/repositories/subagent/interface.ts | 36 ++++ .../db/repositories/subagent/postgres.test.ts | 131 ++++++++++++++ .../lib/db/repositories/subagent/postgres.ts | 75 ++++++++ apps/web-ui/lib/db/repository-factory.ts | 7 + apps/web-ui/lib/queries/query-keys.ts | 4 + apps/web-ui/lib/queries/subagents.ts | 52 ++++++ .../migration.sql | 42 +++++ libs/prisma/schema.prisma | 30 ++++ 22 files changed, 1076 insertions(+), 10 deletions(-) create mode 100644 apps/web-ui/app/api/chat/subagents/[threadId]/route.test.ts create mode 100644 apps/web-ui/app/api/chat/subagents/[threadId]/route.ts create mode 100644 apps/web-ui/components/agent/chat/__tests__/subagent-card.test.tsx create mode 100644 apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts create mode 100644 apps/web-ui/lib/agent/subagent-redact.ts create mode 100644 apps/web-ui/lib/db/repositories/subagent/interface.ts create mode 100644 apps/web-ui/lib/db/repositories/subagent/postgres.test.ts create mode 100644 apps/web-ui/lib/db/repositories/subagent/postgres.ts create mode 100644 apps/web-ui/lib/queries/subagents.ts create mode 100644 libs/prisma/migrations/20260726124500_add_agent_subagent_runs/migration.sql diff --git a/apps/web-ui/app/api/chat/route.ts b/apps/web-ui/app/api/chat/route.ts index 2444eaed..6c7a6bf3 100644 --- a/apps/web-ui/app/api/chat/route.ts +++ b/apps/web-ui/app/api/chat/route.ts @@ -14,6 +14,7 @@ import { resolveResumedToolCallId, type ResumedPendingCall } from './resume-tool import type { PlanStep } from '@/lib/agent/agent-shared'; import { logRunSummary } from '@/lib/agent/run-timings'; import type { SubagentEvent } from '@/lib/agent/dispatch-agent-tool'; +import { getSubagentRunRepository } from '@/lib/db/repository-factory'; export const maxDuration = 300; // 5 minutes for complex multi-iteration tasks @@ -241,7 +242,25 @@ export async function POST(req: Request) { const subagentSinkRef: { sink: ((chunk: UIMessageChunk) => void) | null } = { sink: null }; const emitSubagent = (event: SubagentEvent) => { liveSubagents.set(event.id, event); - if (event.status !== 'running') liveSubagents.delete(event.id); + if (event.status !== 'running') { + liveSubagents.delete(event.id); + // Persist so a collapsed card survives a reload. Fire-and-forget: a + // persistence failure must never break the run. The repository redacts + // secrets before the write. + getSubagentRunRepository().save({ + tenantId: resolvedTenantId, + threadId, + subagentId: event.id, + role: event.role, + task: event.task, + status: event.status, + toolCount: event.toolCount, + tokensIn: event.tokensIn, + tokensOut: event.tokensOut, + summary: event.summary ?? null, + transcript: event.transcript ?? null, + }).catch(err => console.error('[Chat] Failed to persist sub-agent run:', err)); + } try { subagentSinkRef.sink?.(buildSubagentPart(event) as UIMessageChunk); } catch { diff --git a/apps/web-ui/app/api/chat/subagents/[threadId]/route.test.ts b/apps/web-ui/app/api/chat/subagents/[threadId]/route.test.ts new file mode 100644 index 00000000..bbdbb4c8 --- /dev/null +++ b/apps/web-ui/app/api/chat/subagents/[threadId]/route.test.ts @@ -0,0 +1,57 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +vi.mock('@/lib/rbac/authorize', () => ({ authorize: vi.fn() })); +vi.mock('@/lib/auth-session', () => ({ getSessionTenantId: vi.fn() })); +vi.mock('@/lib/db/repository-factory', () => ({ + getSubagentRunRepository: vi.fn(), +})); + +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getSubagentRunRepository } from '@/lib/db/repository-factory'; +import { GET } from './route'; + +const params = (threadId: string) => ({ params: Promise.resolve({ threadId }) }); +const listByThread = vi.fn(); + +beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(authorize).mockResolvedValue(null as never); + vi.mocked(getSessionTenantId).mockResolvedValue('t1' as never); + vi.mocked(getSubagentRunRepository).mockReturnValue({ save: vi.fn(), listByThread } as never); + listByThread.mockResolvedValue([{ subagentId: 'a', role: 'A', transcript: [] }]); +}); + +describe('GET /api/chat/subagents/[threadId]', () => { + it('returns the thread\'s sub-agent runs', async () => { + const body = await (await GET({} as never, params('thread-1'))).json(); + + expect(body.success).toBe(true); + expect(body.data).toHaveLength(1); + expect(listByThread).toHaveBeenCalledWith('t1', 'thread-1'); + }); + + it('propagates an RBAC denial', async () => { + const denied = { status: 403 }; + vi.mocked(authorize).mockResolvedValue(denied as never); + expect(await GET({} as never, params('thread-1'))).toBe(denied); + }); + + it('403s with no tenant context', async () => { + vi.mocked(getSessionTenantId).mockResolvedValue(null as never); + expect((await GET({} as never, params('thread-1'))).status).toBe(403); + }); + + // getSessionTenantId THROWS on an unauthenticated session (it does not return + // null), so the 403 above is only reachable if the throw is caught and mapped — + // otherwise a signed-out caller would get a misleading 500. + it('403s when the session lookup throws', async () => { + vi.mocked(getSessionTenantId).mockRejectedValue(new Error('Unauthenticated: no valid session')); + expect((await GET({} as never, params('thread-1'))).status).toBe(403); + }); + + it('500s when the repository throws', async () => { + listByThread.mockRejectedValue(new Error('db down')); + expect((await GET({} as never, params('thread-1'))).status).toBe(500); + }); +}); diff --git a/apps/web-ui/app/api/chat/subagents/[threadId]/route.ts b/apps/web-ui/app/api/chat/subagents/[threadId]/route.ts new file mode 100644 index 00000000..08c4b2e3 --- /dev/null +++ b/apps/web-ui/app/api/chat/subagents/[threadId]/route.ts @@ -0,0 +1,42 @@ +import { NextRequest, NextResponse } from 'next/server'; +import { authorize } from '@/lib/rbac/authorize'; +import { getSessionTenantId } from '@/lib/auth-session'; +import { getSubagentRunRepository } from '@/lib/db/repository-factory'; + +export async function GET( + _request: NextRequest, + { params }: { params: Promise<{ threadId: string }> }, +) { + console.log('API - GET /api/chat/subagents/[threadId] - Fetching sub-agent runs'); + + const authError = await authorize('read', 'Agent'); + if (authError) return authError; + + try { + // getSessionTenantId throws on a missing session/tenant rather than returning + // null, so catch it here — a signed-out caller must see 403, not 500. + let tenantId: string | null = null; + try { + tenantId = await getSessionTenantId(); + } catch { + tenantId = null; + } + if (!tenantId) { + return NextResponse.json({ success: false, error: 'No tenant context' }, { status: 403 }); + } + + const { threadId } = await params; + // listByThread goes through getTenantClient and AgentSubagentRun is in + // TENANT_SCOPED_MODELS, so the query is tenant-scoped regardless of what + // threadId the caller supplies. + const runs = await getSubagentRunRepository().listByThread(tenantId, threadId); + + return NextResponse.json({ success: true, data: runs }); + } catch (error) { + console.error('API - Error fetching sub-agent runs:', error); + return NextResponse.json( + { success: false, error: error instanceof Error ? error.message : 'Failed to fetch sub-agent runs' }, + { status: 500 }, + ); + } +} diff --git a/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx b/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx index 432d8f8f..fa5fe14b 100644 --- a/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx +++ b/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx @@ -1,9 +1,27 @@ // @vitest-environment jsdom -import { describe, it, expect } from 'vitest' -import { render, screen } from '@testing-library/react' +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { render as rtlRender, screen, waitFor } from '@testing-library/react' +import { QueryClient, QueryClientProvider } from '@tanstack/react-query' import { RunRail, deriveStepTitle } from '../run-rail' import type { RunState } from '../run-state' +// SubagentCard reads persisted transcripts through TanStack Query, so every rail +// render needs a client. Fetch is stubbed per-test; the default is an empty thread. +const fetchMock = vi.fn() + +const render = (ui: React.ReactElement) => { + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }) + return rtlRender({ui}) +} + +beforeEach(() => { + fetchMock.mockReset() + fetchMock.mockResolvedValue({ ok: true, json: async () => ({ success: true, data: [] }) }) + vi.stubGlobal('fetch', fetchMock) +}) + +afterEach(() => vi.unstubAllGlobals()) + const EMPTY_RUN_STATE: RunState = { plan: [], planUpdatedBy: null, @@ -15,6 +33,7 @@ const EMPTY_RUN_STATE: RunState = { hasApprovalData: false, tokenUsage: { input: 0, output: 0 }, subagents: [], + usedSubagents: false, } const CONTEXT = { accountNames: [], modelLabel: '', skillName: null, toolCount: null, kbLabel: 'No knowledge base' } @@ -108,6 +127,7 @@ describe('RunRail', () => { it('renders one card per sub-agent and counts only the running ones in the heading', () => { const runState: RunState = { ...EMPTY_RUN_STATE, + usedSubagents: true, subagents: [ { id: 's1', role: 'EC2 scanner', task: 'Scan EC2', status: 'running', toolCount: 2, tokensIn: 100, tokensOut: 20 }, { id: 's2', role: 'RDS scanner', task: 'Scan RDS', status: 'done', toolCount: 5, tokensIn: 900, tokensOut: 80, summary: 'found it' }, @@ -125,6 +145,46 @@ describe('RunRail', () => { expect(screen.queryByText(/^Sub-agents/)).toBeNull() }) + // data-subagent parts are not persisted, so a reloaded thread arrives with an + // empty `subagents` list. Without this reconstruction the cards — and the + // transcripts Task 11 persists — would be unreachable after a refresh. + it('rebuilds the sub-agent cards from persisted runs after a reload', async () => { + fetchMock.mockResolvedValue({ + ok: true, + json: async () => ({ + success: true, + data: [{ + subagentId: 's1', role: 'EC2 scanner', task: 'Scan EC2', status: 'done', + toolCount: 4, tokensIn: 800, tokensOut: 60, summary: 'found it', + transcript: [{ kind: 'ai', text: 'looked around' }], + }], + }), + }) + + render() + + await waitFor(() => expect(fetchMock).toHaveBeenCalledWith('/api/chat/subagents/thread-1')) + expect(await screen.findByText('EC2 scanner')).toBeTruthy() + expect(screen.getByText('Sub-agents (0 running)')).toBeTruthy() + }) + + it('does not fetch persisted runs for a thread that never dispatched', () => { + render() + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('prefers live sub-agent state over a persisted fetch while streaming', () => { + const runState: RunState = { + ...EMPTY_RUN_STATE, + usedSubagents: true, + subagents: [{ id: 's1', role: 'EC2 scanner', task: 'Scan EC2', status: 'running', toolCount: 1, tokensIn: 10, tokensOut: 1 }], + } + render() + + expect(screen.getByText('Sub-agents (1 running)')).toBeTruthy() + expect(fetchMock).not.toHaveBeenCalled() + }) + it('still shows the pending-approval status row', () => { const runState: RunState = { ...EMPTY_RUN_STATE, diff --git a/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts b/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts index 6eb829e9..e252d225 100644 --- a/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts +++ b/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts @@ -296,3 +296,47 @@ describe('deriveRunState — subagents', () => { expect(deriveRunState([msg([{ type: 'text', text: 'hi' }])] as any, new Set()).subagents).toEqual([]); }); }); + +// data-subagent parts are NOT persisted (history is rebuilt from the LangGraph +// checkpointer), so after a reload `subagents` is empty even for a thread that +// fanned out. The dispatch_agent tool call IS persisted, and it is what tells the +// rail to reconstruct the cards from agent_subagent_runs instead of showing nothing. +describe('deriveRunState — usedSubagents', () => { + it('is false for a thread that never dispatched', () => { + expect(deriveRunState([msg([{ type: 'text', text: 'hi' }])] as any, new Set()).usedSubagents).toBe(false); + }); + + it('is true from a live data-subagent part', () => { + const state = deriveRunState([msg([ + subagentPart({ id: 'a', role: 'A', task: 't', status: 'done', toolCount: 0, tokensIn: 0, tokensOut: 0 }), + ])] as any, new Set()); + + expect(state.usedSubagents).toBe(true); + }); + + it('is true from a reloaded dispatch_agent tool part, whose data parts are gone', () => { + // Shape emitted by /api/threads/[threadId]/history on reload. + const state = deriveRunState([msg([ + { type: 'tool-invocation', toolCallId: 'c1', toolName: 'dispatch_agent', state: 'call' } as any, + ])] as any, new Set()); + + expect(state.subagents).toEqual([]); + expect(state.usedSubagents).toBe(true); + }); + + it('is true from the streamed typed tool part name', () => { + const state = deriveRunState([msg([ + { type: 'tool-dispatch_agent', toolCallId: 'c1', state: 'output-available', output: 'report' } as any, + ])] as any, new Set()); + + expect(state.usedSubagents).toBe(true); + }); + + it('is not tripped by an unrelated tool', () => { + const state = deriveRunState([msg([ + { type: 'tool-invocation', toolCallId: 'c1', toolName: 'aws_read', state: 'call' } as any, + ])] as any, new Set()); + + expect(state.usedSubagents).toBe(false); + }); +}); diff --git a/apps/web-ui/components/agent/chat/__tests__/subagent-card.test.tsx b/apps/web-ui/components/agent/chat/__tests__/subagent-card.test.tsx new file mode 100644 index 00000000..10575edf --- /dev/null +++ b/apps/web-ui/components/agent/chat/__tests__/subagent-card.test.tsx @@ -0,0 +1,93 @@ +// @vitest-environment jsdom +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { render, screen, waitFor, fireEvent } from '@testing-library/react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; + +import { SubagentCard } from '../subagent-card'; +import type { SubagentState } from '../run-state'; + +const DONE: SubagentState = { + id: 'sub-1', + role: 'Cost auditor', + task: 'Audit lambda config', + status: 'done', + toolCount: 3, + tokensIn: 900, + tokensOut: 120, + summary: 'Two functions are oversized.', +}; + +const RUNS = [{ + subagentId: 'sub-1', + role: 'Cost auditor', + task: 'Audit lambda config', + status: 'done', + toolCount: 3, + tokensIn: 900, + tokensOut: 120, + summary: 'Two functions are oversized.', + transcript: [ + { kind: 'ai', text: 'Listing functions first.' }, + { kind: 'tool', name: 'aws_read', text: '{"FunctionName":"api-worker"}' }, + ], +}]; + +const fetchMock = vi.fn(); + +function renderCard(ui: React.ReactElement) { + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render({ui}); +} + +beforeEach(() => { + fetchMock.mockReset(); + fetchMock.mockResolvedValue({ ok: true, json: async () => ({ success: true, data: RUNS }) }); + vi.stubGlobal('fetch', fetchMock); +}); + +afterEach(() => vi.unstubAllGlobals()); + +describe('SubagentCard transcript loading', () => { + it('does no network work until the card is expanded', () => { + renderCard(); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('fetches the thread\'s persisted runs on expand and renders this sub-agent\'s transcript', async () => { + renderCard(); + + fireEvent.click(screen.getByRole('button')); + + await waitFor(() => expect(fetchMock).toHaveBeenCalledWith('/api/chat/subagents/thread-1')); + expect(await screen.findByText('Listing functions first.')).toBeTruthy(); + expect(screen.getByText('{"FunctionName":"api-worker"}')).toBeTruthy(); + expect(screen.getByText('Transcript')).toBeTruthy(); + }); + + it('still shows task and findings when no threadId is available, without fetching', async () => { + renderCard(); + + fireEvent.click(screen.getByRole('button')); + + expect(await screen.findByText('Two functions are oversized.')).toBeTruthy(); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('renders task and findings even when the transcript request fails', async () => { + fetchMock.mockResolvedValue({ ok: false, json: async () => ({ success: false, error: 'db down' }) }); + renderCard(); + + fireEvent.click(screen.getByRole('button')); + + expect(await screen.findByText('Two functions are oversized.')).toBeTruthy(); + await waitFor(() => expect(fetchMock).toHaveBeenCalled()); + expect(screen.queryByText('Transcript')).toBeNull(); + }); + + it('cannot be expanded while the sub-agent is still running', async () => { + renderCard(); + + expect(screen.getByRole('button')).toHaveProperty('disabled', true); + expect(fetchMock).not.toHaveBeenCalled(); + }); +}); diff --git a/apps/web-ui/components/agent/chat/run-rail.tsx b/apps/web-ui/components/agent/chat/run-rail.tsx index 22ad4c9b..1879437a 100644 --- a/apps/web-ui/components/agent/chat/run-rail.tsx +++ b/apps/web-ui/components/agent/chat/run-rail.tsx @@ -6,7 +6,8 @@ import { Activity, AlertTriangle, Bot, ChevronsDownUp, ChevronsUpDown, Cloud, Cp import { Plan, PlanContent, PlanStep } from "@/components/ai-elements/plan"; import { Spinner } from "@/components/ui/spinner"; import { SubagentCard } from "./subagent-card"; -import type { RunState } from "./run-state"; +import { useSubagentRuns } from "@/lib/queries/subagents"; +import type { RunState, SubagentState } from "./run-state"; const PHASE_LABELS: Record = { planning: "Planning", execution: "Executing", reflection: "Reflecting", @@ -57,6 +58,25 @@ export function RunRail({ threadId?: string; }) { const { plan, currentPhase, pendingApproval, pendingClarifications } = runState; + + // A reloaded thread has no data-subagent parts (they are not persisted), so the + // cards are rebuilt from agent_subagent_runs. Only fetched when the thread is + // known to have fanned out — a persisted dispatch_agent tool call — and only when + // there is no live state to prefer. + const needsPersistedSubagents = runState.usedSubagents && runState.subagents.length === 0; + const { data: persistedRuns } = useSubagentRuns(threadId, needsPersistedSubagents); + const subagents: SubagentState[] = needsPersistedSubagents + ? (persistedRuns ?? []).map((run) => ({ + id: run.subagentId, + role: run.role, + task: run.task, + status: run.status === "done" || run.status === "failed" ? run.status : "running", + toolCount: run.toolCount, + tokensIn: run.tokensIn, + tokensOut: run.tokensOut, + ...(run.summary ? { summary: run.summary } : {}), + })) + : runState.subagents; const done = plan.filter((s) => s.status === "completed").length; const mutativePending = pendingApproval?.tools.some((t) => t.guard?.isMutative) ?? false; @@ -150,13 +170,13 @@ export function RunRail({ {/* Dispatched sub-agents — collapsed cards so their prose never lands in the transcript; each one expands to its own task + findings. */} - {runState.subagents.length > 0 && ( + {subagents.length > 0 && ( s.status === "running").length} running)`} + title={`Sub-agents (${subagents.filter((s) => s.status === "running").length} running)`} >
- {runState.subagents.map((subagent) => ( + {subagents.map((subagent) => ( ))}
diff --git a/apps/web-ui/components/agent/chat/run-state.ts b/apps/web-ui/components/agent/chat/run-state.ts index 34e99f58..ff8447b1 100644 --- a/apps/web-ui/components/agent/chat/run-state.ts +++ b/apps/web-ui/components/agent/chat/run-state.ts @@ -40,6 +40,11 @@ export interface RunState { /** Sub-agents dispatched in this thread, in first-seen order. Later parts for * the same id replace the earlier entry in place. */ subagents: SubagentState[]; + /** True when this thread fanned out at all — set by a live data-subagent part + * OR by a persisted dispatch_agent tool call. data-subagent parts are not + * persisted, so after a reload this is the ONLY signal that the rail should + * rebuild the cards from agent_subagent_runs rather than render nothing. */ + usedSubagents: boolean; } interface LoosePart { type: string; data?: any; text?: string } @@ -59,6 +64,7 @@ export function deriveRunState( let tokenIn = 0; let tokenOut = 0; const subagents = new Map(); + let usedSubagents = false; // Tool calls with an output-bearing tool part ANYWHERE in the thread. An // executed / rejected / answered tool can never be pending again, no matter // what a stale data-approval or data-clarification part claims (defense @@ -77,6 +83,11 @@ export function deriveRunState( p.state === 'output-available' || p.state === 'output-error'; if (hasOutput) executedIds.add(String(p.toolCallId)); + // `tool-invocation` + toolName is the reloaded history shape; + // `tool-dispatch_agent` is the streamed typed-part shape. + if (p.toolName === 'dispatch_agent' || p.type === 'tool-dispatch_agent') { + usedSubagents = true; + } } switch (part.type) { case 'data-plan': { @@ -118,6 +129,7 @@ export function deriveRunState( } case 'data-subagent': { hasStructuredData = true; + usedSubagents = true; const id = String(part.data?.id ?? ''); if (!id) break; // Map.set on an existing key preserves insertion order, so the @@ -157,6 +169,7 @@ export function deriveRunState( hasApprovalData, tokenUsage: { input: tokenIn, output: tokenOut }, subagents: Array.from(subagents.values()), + usedSubagents, }; } diff --git a/apps/web-ui/components/agent/chat/subagent-card.tsx b/apps/web-ui/components/agent/chat/subagent-card.tsx index 50b385bb..b9819459 100644 --- a/apps/web-ui/components/agent/chat/subagent-card.tsx +++ b/apps/web-ui/components/agent/chat/subagent-card.tsx @@ -4,6 +4,7 @@ import { useState } from 'react'; import { Bot, CheckCircle2, ChevronDown, ChevronRight, Loader2, XCircle } from 'lucide-react'; import { cn } from '@/lib/utils'; +import { useSubagentRuns } from '@/lib/queries/subagents'; import type { SubagentState } from './run-state'; function formatTokens(n: number): string { @@ -21,14 +22,20 @@ export function SubagentCard({ threadId, }: { subagent: SubagentState; - /** Reserved for fetching a persisted sub-agent transcript when a card is - * expanded after a reload — not used yet; the findings panel below reads - * purely from the live `subagent.summary`. */ + /** Thread whose persisted sub-agent runs hold the transcript. Optional: without + * it the card still renders the live task + findings, just no transcript. */ threadId?: string; }) { const [expanded, setExpanded] = useState(false); const canExpand = subagent.status !== 'running' && !!subagent.summary; + // Loaded only on expand — the transcript is never streamed (three interleaved + // sub-agent narrations are unreadable) and never persisted in the message + // history, so it comes from agent_subagent_runs on demand. Keyed per thread, so + // expanding a second card in the same thread is served from cache. + const { data: runs } = useSubagentRuns(threadId, expanded); + const transcript = runs?.find(r => r.subagentId === subagent.id)?.transcript ?? []; + return (
)} diff --git a/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx b/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx index 22ed3fed..ce381c3b 100644 --- a/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx +++ b/apps/web-ui/components/agent/workspace/__tests__/agent-turn.test.tsx @@ -20,6 +20,7 @@ const EMPTY_RUN_STATE: RunState = { hasApprovalData: false, tokenUsage: { input: 0, output: 0 }, subagents: [], + usedSubagents: false, } function baseProps(events: TranscriptEvent[]) { diff --git a/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx b/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx index 7b5e24d4..aa2e7e25 100644 --- a/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx +++ b/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx @@ -31,6 +31,7 @@ const EMPTY_RUN_STATE: RunState = { hasApprovalData: false, tokenUsage: { input: 0, output: 0 }, subagents: [], + usedSubagents: false, } function planOf(doneCount: number, total: number): RunState['plan'] { diff --git a/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts b/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts new file mode 100644 index 00000000..fb22b1c2 --- /dev/null +++ b/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts @@ -0,0 +1,148 @@ +/** + * Redaction at the persistence boundary. + * + * `aws_read` permits `lambda get-function-configuration`, which returns + * Environment.Variables in PLAINTEXT (see the NOTE in aws-read-tool.ts). Task 11 + * persists sub-agent transcripts to Postgres for 30 days, so those values must be + * scrubbed before the write — not at the tool, and not at the call site. + */ +import { describe, it, expect } from 'vitest'; + +import { REDACTED, redactTranscript } from '../subagent-redact'; + +type Entry = { kind: 'ai' | 'tool'; name?: string; text: string }; + +const entry = (text: string, name = 'aws_read'): Entry => ({ kind: 'tool', name, text }); + +/** Everything the redactor produced, flattened, so a leak anywhere is caught. */ +const flatten = (value: unknown) => JSON.stringify(value); + +describe('redactTranscript — Environment.Variables (redact by location)', () => { + const lambdaConfig = JSON.stringify({ + FunctionName: 'api-worker', + Environment: { Variables: { DB_PASSWORD: 'hunter2', LOG_LEVEL: 'debug' } }, + }); + + it('replaces every value under Environment.Variables but keeps the keys', () => { + const [out] = redactTranscript([entry(lambdaConfig)])!; + const parsed = JSON.parse(out.text); + + expect(parsed.Environment.Variables.DB_PASSWORD).toBe(REDACTED); + // Redacted by LOCATION: we cannot know which arbitrary env var name is a secret. + expect(parsed.Environment.Variables.LOG_LEVEL).toBe(REDACTED); + // The operator still needs to see that the variable exists. + expect(Object.keys(parsed.Environment.Variables)).toEqual(['DB_PASSWORD', 'LOG_LEVEL']); + expect(parsed.FunctionName).toBe('api-worker'); + }); + + it('leaks the plaintext value nowhere in the output', () => { + expect(flatten(redactTranscript([entry(lambdaConfig)]))).not.toContain('hunter2'); + }); +}); + +describe('redactTranscript — secret-shaped keys anywhere in the document', () => { + it.each([ + ['apiKey', '{"credentials":{"apiKey":"sk_live_abc"}}', 'sk_live_abc'], + ['api_key', '{"api_key":"sk_live_abc"}', 'sk_live_abc'], + ['api-key', '{"api-key":"sk_live_abc"}', 'sk_live_abc'], + ['PASSWORD', '{"PASSWORD":"hunter2"}', 'hunter2'], + ['passwd', '{"passwd":"hunter2"}', 'hunter2'], + ['clientSecret', '{"clientSecret":"shh"}', 'shh'], + ['authToken', '{"authToken":"abc123"}', 'abc123'], + ['Credential', '{"Credential":"abc123"}', 'abc123'], + ['private_key', '{"private_key":"-----BEGIN RSA-----"}', 'BEGIN RSA'], + ['SessionToken', '{"SessionToken":"IQoJb3JpZ2lu"}', 'IQoJb3JpZ2lu'], + ])('redacts %s case-insensitively', (_label, json, plaintext) => { + expect(flatten(redactTranscript([entry(json)]))).not.toContain(plaintext); + }); + + it('walks nested arrays and objects', () => { + const [out] = redactTranscript([entry('{"items":[{"token":"abc"}]}')])!; + expect(JSON.parse(out.text).items[0].token).toBe(REDACTED); + }); +}); + +describe('redactTranscript — ordinary values survive', () => { + it('leaves a realistic non-secret payload byte-identical', () => { + const payload = JSON.stringify({ + FunctionName: 'api-worker', + FunctionArn: 'arn:aws:lambda:us-east-1:123456789012:function:api-worker', + Runtime: 'nodejs20.x', + MemorySize: 512, + Timeout: 30, + LastModified: '2026-07-26T10:15:00.000+0000', + Instances: [{ InstanceId: 'i-0abc123def4567890', State: 'running' }], + Tags: { Environment: 'prod', Owner: 'platform' }, + }); + + const input: Entry[] = [{ kind: 'tool', name: 'aws_read', text: payload }]; + const [out] = redactTranscript(input)!; + + expect(out.text).toBe(payload); + expect(out).toEqual(input[0]); + }); + + it('does not touch an `Environment` that has no `Variables`', () => { + const payload = JSON.stringify({ Environment: { Region: 'us-east-1' } }); + expect(redactTranscript([entry(payload)])![0].text).toBe(payload); + }); + + it('leaves plain narration alone', () => { + const text = 'Checked 3 accounts; 2 instances are oversized in us-east-1.'; + expect(redactTranscript([{ kind: 'ai', text }])![0].text).toBe(text); + }); +}); + +describe('redactTranscript — non-JSON text', () => { + it('scrubs shell-style KEY=value assignments', () => { + const out = redactTranscript([entry('export DB_PASSWORD=hunter2 && ./run.sh')])!; + expect(out[0].text).not.toContain('hunter2'); + expect(out[0].text).toContain('DB_PASSWORD'); + }); + + it('scrubs `key: value` prose', () => { + expect(flatten(redactTranscript([entry('api_key: sk_live_abc')]))).not.toContain('sk_live_abc'); + }); + + it('scrubs quoted JSON-ish fragments that never parsed', () => { + expect(flatten(redactTranscript([entry('trailing junk "secret": "shh" more')]))) + .not.toContain('shh'); + }); +}); + +describe('redactTranscript — malformed and hostile input', () => { + it('does not throw and does not pass a secret through truncated JSON', () => { + let out: Entry[] | null = null; + expect(() => { out = redactTranscript([entry('{"password":"hunter2"')]); }).not.toThrow(); + expect(flatten(out)).not.toContain('hunter2'); + }); + + it('returns null for null and undefined for undefined', () => { + expect(redactTranscript(null)).toBeNull(); + expect(redactTranscript(undefined)).toBeUndefined(); + }); + + it('returns an empty transcript unchanged', () => { + expect(redactTranscript([])).toEqual([]); + }); + + it('survives a self-referential object without hanging', () => { + const cyclic: Record = { password: 'hunter2' }; + cyclic.self = cyclic; + + let out: unknown; + expect(() => { out = redactTranscript(cyclic); }).not.toThrow(); + // The cycle must have been broken, so the result is serialisable... + expect(() => JSON.stringify(out)).not.toThrow(); + // ...and the secret must not be in it. + expect(JSON.stringify(out)).not.toContain('hunter2'); + }); + + it('survives pathologically deep nesting', () => { + let deep: Record = { token: 'abc' }; + for (let i = 0; i < 500; i++) deep = { nested: deep }; + + expect(() => redactTranscript([entry(JSON.stringify(deep))])).not.toThrow(); + expect(flatten(redactTranscript([entry(JSON.stringify(deep))]))).not.toContain('"abc"'); + }); +}); diff --git a/apps/web-ui/lib/agent/subagent-redact.ts b/apps/web-ui/lib/agent/subagent-redact.ts new file mode 100644 index 00000000..def57389 --- /dev/null +++ b/apps/web-ui/lib/agent/subagent-redact.ts @@ -0,0 +1,167 @@ +/** + * Secret redaction for sub-agent transcripts, applied at the persistence boundary. + * + * Why this exists: the `aws_read` allowlist permits `lambda + * get-function-configuration`, which returns `Environment.Variables` in PLAINTEXT — + * DB passwords, API keys, Stripe secrets (see the NOTE in aws-read-tool.ts). A + * sub-agent transcript is persisted for 30 days in a multi-tenant database that also + * backs chat history, so those values must never reach at-rest storage. The sub-agent + * still sees the value live in its own reasoning; that is accepted and out of scope. + * + * Two complementary strategies: + * - by LOCATION — everything under an `Environment.Variables` object is redacted, + * because an arbitrary env var name tells us nothing about its sensitivity; + * - by KEY NAME — a secret-shaped key is redacted wherever it appears. + * + * Only the VALUE is replaced. Keys are always preserved: an operator expanding a card + * needs to see that `DB_PASSWORD` exists. + * + * Pure, synchronous, no I/O, no imports from the agent runtime — and total. It runs + * inside a database write, so throwing would fail the persistence it exists to protect. + */ + +export const REDACTED = '[REDACTED]'; + +/** Depth ceiling. Beyond it we fail closed rather than emit an unwalked subtree. */ +const MAX_DEPTH = 64; + +/** Case-insensitive substring match against a key name. */ +const SECRET_KEY_PATTERNS = [ + 'password', + 'passwd', + 'secret', + 'token', + 'apikey', + 'api_key', + 'api-key', + 'credential', + 'privatekey', + 'private_key', + 'sessiontoken', +]; + +function isSecretKey(key: string): boolean { + const lower = key.toLowerCase(); + return SECRET_KEY_PATTERNS.some(pattern => lower.includes(pattern)); +} + +/** Alternation used by the non-JSON fallback. Ordered longest-first so `api_key` wins over `key`. */ +const SECRET_KEY_ALTERNATION = '[A-Za-z0-9_.\\-]*(?:password|passwd|secret|token|api[_-]?key|credential|private[_-]?key)[A-Za-z0-9_.\\-]*'; + +/** + * Redact a parsed JSON value in place-free fashion. + * + * @param redactAll true when the caller is already inside an `Environment.Variables` + * object, where every value is redacted regardless of its key. + */ +function redactValue(value: unknown, depth: number, seen: WeakSet, redactAll: boolean): unknown { + if (depth > MAX_DEPTH) return REDACTED; + if (value === null || typeof value !== 'object') { + return redactAll ? REDACTED : value; + } + + const asObject = value as object; + // A cycle can only come from a caller handing us a live object graph (parsed JSON + // is always acyclic). Break it rather than recursing forever. + if (seen.has(asObject)) return '[CYCLE]'; + seen.add(asObject); + + if (Array.isArray(value)) { + return value.map(item => redactValue(item, depth + 1, seen, redactAll)); + } + + const out: Record = {}; + for (const [key, child] of Object.entries(value as Record)) { + if (redactAll || isSecretKey(key)) { + // Redact the whole subtree: a secret-named key may hold an object. + out[key] = redactValue(child, depth + 1, seen, true); + continue; + } + // Redact by LOCATION: everything under `Environment: { Variables: {...} }`. + // Keyed on `Variables` alone rather than the full path — an object named + // `Variables` in an AWS response is env vars whatever its parent, and erring + // wide here costs an operator nothing but erring narrow leaks a secret. + out[key] = redactValue(child, depth + 1, seen, /^variables$/i.test(key)); + } + return out; +} + +/** Regex scrubbing for text that is not JSON (shell commands, prose, truncated output). */ +function redactPlainText(text: string): string { + let out = text; + // "key": "value" / 'key': 'value' + out = out.replace( + new RegExp(`(["']?${SECRET_KEY_ALTERNATION}["']?\\s*:\\s*)(["'])(?:\\\\.|(?!\\2).)*\\2`, 'gi'), + (_m, prefix: string, quote: string) => `${prefix}${quote}${REDACTED}${quote}`, + ); + // key: value (unquoted, to end of line / delimiter) + out = out.replace( + new RegExp(`(\\b${SECRET_KEY_ALTERNATION}\\s*:\\s*)([^\\s,;}"']+)`, 'gi'), + (_m, prefix: string) => `${prefix}${REDACTED}`, + ); + // KEY=value (shell assignment) + out = out.replace( + new RegExp(`(\\b${SECRET_KEY_ALTERNATION}\\s*=\\s*)("[^"]*"|'[^']*'|[^\\s&;|]+)`, 'gi'), + (_m, prefix: string) => `${prefix}${REDACTED}`, + ); + return out; +} + +/** Redact one transcript entry's `text`, preferring a structural walk over regex. */ +function redactText(text: string): string { + const trimmed = text.trim(); + const looksStructural = trimmed.startsWith('{') || trimmed.startsWith('['); + + if (looksStructural) { + try { + const parsed: unknown = JSON.parse(trimmed); + return JSON.stringify(redactValue(parsed, 0, new WeakSet(), false)); + } catch { + // Truncated or otherwise unparseable JSON — fall through to regex. Never + // return the raw input here: that is exactly the leak path. + } + } + return redactPlainText(text); +} + +function isTranscriptEntry(value: unknown): value is { text: string } { + return typeof value === 'object' + && value !== null + && typeof (value as { text?: unknown }).text === 'string'; +} + +/** + * Redact secrets from a sub-agent transcript (or any value on its way to storage). + * + * `null`/`undefined` pass through unchanged. An array of transcript entries has each + * entry's `text` scrubbed; anything else is walked structurally. Never throws — on an + * internal error the input is replaced with a safe placeholder rather than persisted raw. + */ +export function redactTranscript(transcript: T): T { + if (transcript === null || transcript === undefined) return transcript; + + try { + if (typeof transcript === 'string') { + return redactText(transcript) as unknown as T; + } + + if (Array.isArray(transcript)) { + return transcript.map(item => + isTranscriptEntry(item) + ? { ...item, text: redactText(item.text) } + : redactValue(item, 0, new WeakSet(), false), + ) as unknown as T; + } + + if (isTranscriptEntry(transcript)) { + return { ...transcript, text: redactText(transcript.text) } as unknown as T; + } + + return redactValue(transcript, 0, new WeakSet(), false) as unknown as T; + } catch (error) { + console.error('[subagent-redact] redaction failed; storing a placeholder instead:', error); + // Fail closed: a redactor that cannot prove the payload is clean must not + // hand the raw payload to the database. + return REDACTED as unknown as T; + } +} diff --git a/apps/web-ui/lib/db/pg-config.ts b/apps/web-ui/lib/db/pg-config.ts index 711ca90f..a83f8738 100644 --- a/apps/web-ui/lib/db/pg-config.ts +++ b/apps/web-ui/lib/db/pg-config.ts @@ -77,6 +77,7 @@ export const TENANT_SCOPED_MODELS = new Set([ 'CertificateDeployment', 'CertificateExecution', 'ChatMessage', + 'AgentSubagentRun', 'CustomRole', 'UserTenantRole', 'TenantConfig', diff --git a/apps/web-ui/lib/db/repositories/subagent/interface.ts b/apps/web-ui/lib/db/repositories/subagent/interface.ts new file mode 100644 index 00000000..4d1aa319 --- /dev/null +++ b/apps/web-ui/lib/db/repositories/subagent/interface.ts @@ -0,0 +1,36 @@ +/** + * Sub-agent run persistence — one row per dispatch_agent sub-agent, so a collapsed + * card in the chat run rail can still be expanded after a page reload. + * + * The transcript deliberately lives here and NEVER in the dispatch_agent ToolMessage: + * that message enters the orchestrator's context and would defeat the very isolation + * sub-agents exist to provide. + */ + +export interface SubagentTranscriptEntry { + kind: 'ai' | 'tool'; + name?: string; + text: string; +} + +export interface SubagentRunRecord { + tenantId: string; + threadId: string; + subagentId: string; + role: string; + task: string; + status: string; + toolCount: number; + tokensIn: number; + tokensOut: number; + summary?: string | null; + transcript?: SubagentTranscriptEntry[] | null; +} + +export interface SubagentRunRepository { + /** Upsert by (tenantId, threadId, subagentId) — a sub-agent is written once + * on completion, but a retried write must not create a duplicate row. + * Implementations MUST redact secrets before persisting. */ + save(record: SubagentRunRecord): Promise; + listByThread(tenantId: string, threadId: string): Promise; +} diff --git a/apps/web-ui/lib/db/repositories/subagent/postgres.test.ts b/apps/web-ui/lib/db/repositories/subagent/postgres.test.ts new file mode 100644 index 00000000..c315c32b --- /dev/null +++ b/apps/web-ui/lib/db/repositories/subagent/postgres.test.ts @@ -0,0 +1,131 @@ +/** + * The redaction guarantee is asserted HERE, at the repository, because that is where + * it is enforced — every present and future caller of save() is covered and none can + * forget. These tests inspect the argument actually handed to Prisma's upsert rather + * than asserting the redactor was called, so deleting the redactTranscript(...) wrapper + * in postgres.ts fails them. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const upsert = vi.fn(); +const findMany = vi.fn(); +const getTenantClient = vi.fn(() => ({ agentSubagentRun: { upsert, findMany } })); + +vi.mock('@/lib/db/pg-config', () => ({ getTenantClient: (...args: unknown[]) => getTenantClient(...(args as [])) })); + +import { SubagentRunPostgresRepository } from './postgres'; +import { REDACTED } from '@/lib/agent/subagent-redact'; +import type { SubagentRunRecord } from './interface'; + +const repo = new SubagentRunPostgresRepository(); + +const LAMBDA_CONFIG = JSON.stringify({ + FunctionName: 'api-worker', + Environment: { Variables: { DB_PASSWORD: 'hunter2' } }, +}); + +const record = (overrides: Partial = {}): SubagentRunRecord => ({ + tenantId: 't1', + threadId: 'thread-1', + subagentId: 'sub-1', + role: 'Cost auditor', + task: 'Audit lambda config', + status: 'done', + toolCount: 2, + tokensIn: 100, + tokensOut: 50, + summary: 'Nothing unusual.', + transcript: [{ kind: 'tool', name: 'aws_read', text: LAMBDA_CONFIG }], + ...overrides, +}); + +beforeEach(() => { + vi.clearAllMocks(); + upsert.mockResolvedValue({}); + findMany.mockResolvedValue([]); +}); + +describe('SubagentRunPostgresRepository.save', () => { + it('redacts Environment.Variables out of the persisted transcript', async () => { + await repo.save(record()); + + const args = upsert.mock.calls[0][0]; + for (const payload of [args.create, args.update]) { + expect(JSON.stringify(payload.transcript)).toContain(REDACTED); + expect(JSON.stringify(payload.transcript)).not.toContain('hunter2'); + } + // Nothing anywhere in the write may carry the plaintext. + expect(JSON.stringify(args)).not.toContain('hunter2'); + }); + + it('redacts the LLM-authored summary too — it is shown even when the transcript is empty', async () => { + await repo.save(record({ + transcript: [], + summary: 'The function is misconfigured; DB_PASSWORD=hunter2 is set in plaintext.', + })); + + const args = upsert.mock.calls[0][0]; + expect(args.create.summary).toContain(REDACTED); + expect(JSON.stringify(args)).not.toContain('hunter2'); + }); + + it('upserts on (tenantId, threadId, subagentId) so a retried write cannot duplicate', async () => { + await repo.save(record()); + + const args = upsert.mock.calls[0][0]; + expect(args.where.tenantId_threadId_subagentId).toEqual({ + tenantId: 't1', threadId: 'thread-1', subagentId: 'sub-1', + }); + expect(getTenantClient).toHaveBeenCalledWith('t1'); + }); + + it('preserves non-secret fields and sets a 30-day TTL', async () => { + const before = Date.now(); + await repo.save(record({ transcript: [{ kind: 'ai', text: 'Checked 3 accounts.' }] })); + + const { create } = upsert.mock.calls[0][0]; + expect(create.role).toBe('Cost auditor'); + expect(create.toolCount).toBe(2); + expect(create.tokensIn).toBe(100); + expect(create.transcript[0].text).toBe('Checked 3 accounts.'); + expect(create.expiresAt.getTime()).toBeGreaterThanOrEqual(before + 29 * 24 * 60 * 60 * 1000); + }); + + it('writes null rather than undefined for an absent transcript', async () => { + await repo.save(record({ transcript: null, summary: null })); + + const { create } = upsert.mock.calls[0][0]; + expect(create.transcript).toBeNull(); + expect(create.summary).toBeNull(); + }); +}); + +describe('SubagentRunPostgresRepository.listByThread', () => { + it('reads through the tenant-scoped client', async () => { + findMany.mockResolvedValue([{ + tenantId: 't1', threadId: 'thread-1', subagentId: 'sub-1', + role: 'R', task: 'T', status: 'done', toolCount: 1, tokensIn: 2, tokensOut: 3, + summary: 's', transcript: [{ kind: 'ai', text: 'x' }], + }]); + + const rows = await repo.listByThread('t1', 'thread-1'); + + expect(getTenantClient).toHaveBeenCalledWith('t1'); + expect(findMany).toHaveBeenCalledWith({ + where: { threadId: 'thread-1' }, + orderBy: { createdAt: 'asc' }, + }); + expect(rows).toHaveLength(1); + expect(rows[0].transcript).toEqual([{ kind: 'ai', text: 'x' }]); + }); + + it('maps a null transcript to null', async () => { + findMany.mockResolvedValue([{ + tenantId: 't1', threadId: 'thread-1', subagentId: 'sub-1', + role: 'R', task: 'T', status: 'failed', toolCount: 0, tokensIn: 0, tokensOut: 0, + summary: null, transcript: null, + }]); + + expect((await repo.listByThread('t1', 'thread-1'))[0].transcript).toBeNull(); + }); +}); diff --git a/apps/web-ui/lib/db/repositories/subagent/postgres.ts b/apps/web-ui/lib/db/repositories/subagent/postgres.ts new file mode 100644 index 00000000..7013caa7 --- /dev/null +++ b/apps/web-ui/lib/db/repositories/subagent/postgres.ts @@ -0,0 +1,75 @@ +/** + * SubagentRunPostgresRepository + * + * PostgreSQL implementation of SubagentRunRepository using Prisma ORM. + * Reads/writes the `agent_subagent_runs` table (defined in libs/prisma/schema.prisma). + * + * AgentSubagentRun is listed in TENANT_SCOPED_MODELS, so getTenantClient() injects + * tenantId into every read and write here — a caller cannot reach another tenant's + * transcript by supplying its threadId. + */ +import { getTenantClient } from '@/lib/db/pg-config'; +import { redactTranscript } from '@/lib/agent/subagent-redact'; +import type { SubagentRunRecord, SubagentRunRepository, SubagentTranscriptEntry } from './interface'; + +const TTL_30_DAYS_MS = 30 * 24 * 60 * 60 * 1000; + +export class SubagentRunPostgresRepository implements SubagentRunRepository { + async save(record: SubagentRunRecord): Promise { + const db = getTenantClient(record.tenantId); + const data = { + tenantId: record.tenantId, + threadId: record.threadId, + subagentId: record.subagentId, + role: record.role, + task: record.task, + status: record.status, + toolCount: record.toolCount, + tokensIn: record.tokensIn, + tokensOut: record.tokensOut, + // Redacted HERE rather than at the call site so no caller can bypass it. + // aws_read permits `lambda get-function-configuration`, which returns + // Environment.Variables in plaintext; those must never reach at-rest storage. + // The summary is redacted too: it is LLM-authored prose that can quote a + // secret the sub-agent saw, and it is the one field still shown after a + // reload when the transcript is empty. + summary: redactTranscript(record.summary ?? null), + transcript: redactTranscript(record.transcript ?? null) as never, + expiresAt: new Date(Date.now() + TTL_30_DAYS_MS), + }; + + await db.agentSubagentRun.upsert({ + where: { + tenantId_threadId_subagentId: { + tenantId: record.tenantId, + threadId: record.threadId, + subagentId: record.subagentId, + }, + }, + create: data, + update: data, + }); + } + + async listByThread(tenantId: string, threadId: string): Promise { + const db = getTenantClient(tenantId); + const rows = await db.agentSubagentRun.findMany({ + where: { threadId }, + orderBy: { createdAt: 'asc' }, + }); + + return rows.map(row => ({ + tenantId: row.tenantId, + threadId: row.threadId, + subagentId: row.subagentId, + role: row.role, + task: row.task, + status: row.status, + toolCount: row.toolCount, + tokensIn: row.tokensIn, + tokensOut: row.tokensOut, + summary: row.summary, + transcript: (row.transcript ?? null) as SubagentTranscriptEntry[] | null, + })); + } +} diff --git a/apps/web-ui/lib/db/repository-factory.ts b/apps/web-ui/lib/db/repository-factory.ts index 7a65e477..62596ea1 100644 --- a/apps/web-ui/lib/db/repository-factory.ts +++ b/apps/web-ui/lib/db/repository-factory.ts @@ -30,6 +30,7 @@ import type { ISkillRepository } from './repositories/skill/interface'; import type { ISlackWorkspaceLinkRepository } from './repositories/slack-workspace-link/interface'; import type { IDashboardRepository } from './repositories/dashboard/interface'; import type { ITelegramBotLinkRepository } from './repositories/telegram-bot-link/interface'; +import type { SubagentRunRepository } from './repositories/subagent/interface'; export function getTenantConfigRepository(): ITenantConfigRepository { // eslint-disable-next-line @typescript-eslint/no-require-imports @@ -151,6 +152,12 @@ export function getTelegramBotLinkRepository(): ITelegramBotLinkRepository { return new TelegramBotLinkPostgresRepository(); } +export function getSubagentRunRepository(): SubagentRunRepository { + // eslint-disable-next-line @typescript-eslint/no-require-imports + const { SubagentRunPostgresRepository } = require('./repositories/subagent/postgres'); + return new SubagentRunPostgresRepository(); +} + /** * Feature flag helper — kept for backward compat with existing callers. * Always returns true since all entities now use PostgreSQL. diff --git a/apps/web-ui/lib/queries/query-keys.ts b/apps/web-ui/lib/queries/query-keys.ts index 1845262b..a1dae06b 100644 --- a/apps/web-ui/lib/queries/query-keys.ts +++ b/apps/web-ui/lib/queries/query-keys.ts @@ -59,6 +59,10 @@ export const queryKeys = { content: (id: string, versionId?: string) => [...queryKeys.certificates.detail(id), 'content', versionId ?? 'active'] as const, }, + subagents: { + all: ['subagents'] as const, + byThread: (threadId: string) => [...queryKeys.subagents.all, 'thread', threadId] as const, + }, kbChat: { all: ['kb-chat'] as const, sessions: () => [...queryKeys.kbChat.all, 'sessions'] as const, diff --git a/apps/web-ui/lib/queries/subagents.ts b/apps/web-ui/lib/queries/subagents.ts new file mode 100644 index 00000000..04f80082 --- /dev/null +++ b/apps/web-ui/lib/queries/subagents.ts @@ -0,0 +1,52 @@ +'use client'; + +/** + * TanStack Query hook for persisted sub-agent runs. + * + * Stream data parts are not persisted — chat history is rebuilt from the LangGraph + * checkpointer — so live expansion works from client state but a reload loses it. + * A collapsed card fetches its transcript from `agent_subagent_runs` on demand. + * + * Keyed per thread, not per sub-agent: one response carries every sub-agent in the + * thread, so expanding a second card is served from cache. + */ +import { useQuery } from '@tanstack/react-query'; +import { queryKeys } from '@/lib/queries/query-keys'; + +export interface PersistedSubagentTranscriptEntry { + kind: 'ai' | 'tool'; + name?: string; + text: string; +} + +export interface PersistedSubagentRun { + subagentId: string; + role: string; + task: string; + status: string; + toolCount: number; + tokensIn: number; + tokensOut: number; + summary?: string | null; + transcript?: PersistedSubagentTranscriptEntry[] | null; +} + +/** + * @param threadId thread whose sub-agent runs to load; the query stays idle without one + * @param enabled pass the card's expanded state — keeps the default render free of + * network work + */ +export function useSubagentRuns(threadId: string | undefined, enabled: boolean) { + return useQuery({ + queryKey: queryKeys.subagents.byThread(threadId ?? 'none'), + enabled: enabled && !!threadId, + queryFn: async (): Promise => { + const res = await fetch(`/api/chat/subagents/${encodeURIComponent(threadId!)}`); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) { + throw new Error(json.error || 'Failed to load sub-agent transcript'); + } + return json.data as PersistedSubagentRun[]; + }, + }); +} diff --git a/libs/prisma/migrations/20260726124500_add_agent_subagent_runs/migration.sql b/libs/prisma/migrations/20260726124500_add_agent_subagent_runs/migration.sql new file mode 100644 index 00000000..9ae0188b --- /dev/null +++ b/libs/prisma/migrations/20260726124500_add_agent_subagent_runs/migration.sql @@ -0,0 +1,42 @@ +-- AgentSubagentRun — one row per dispatch_agent sub-agent so a collapsed card can be +-- expanded after a page reload. Secrets are redacted at the persistence boundary +-- (apps/web-ui/lib/agent/subagent-redact.ts) before any transcript reaches this table. +-- +-- AUDIT NOTE: `prisma migrate diff` emitted three unrelated statements ahead of this +-- table and they were deleted by hand, as the repo's history requires: +-- DROP INDEX "agent_memories_embedding_hnsw"; +-- DROP INDEX "idx_inventory_search_vector"; +-- DROP INDEX "idx_kb_document_chunks_embedding"; +-- Those indexes are created by raw SQL (HNSW / GIN) and are invisible to the Prisma +-- datamodel, so every generated migration treats them as drift. Dropping them would +-- silently destroy vector search and inventory full-text search — the same damage +-- 20260725120000_restore_raw_sql_indexes exists to repair. Do not re-add them. + +-- CreateTable +CREATE TABLE "agent_subagent_runs" ( + "id" TEXT NOT NULL, + "tenantId" TEXT NOT NULL, + "threadId" TEXT NOT NULL, + "subagentId" TEXT NOT NULL, + "role" TEXT NOT NULL, + "task" TEXT NOT NULL, + "status" TEXT NOT NULL, + "toolCount" INTEGER NOT NULL DEFAULT 0, + "tokensIn" INTEGER NOT NULL DEFAULT 0, + "tokensOut" INTEGER NOT NULL DEFAULT 0, + "summary" TEXT, + "transcript" JSONB, + "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + "expiresAt" TIMESTAMP(3) NOT NULL, + + CONSTRAINT "agent_subagent_runs_pkey" PRIMARY KEY ("id") +); + +-- CreateIndex +CREATE INDEX "agent_subagent_runs_tenantId_threadId_createdAt_idx" ON "agent_subagent_runs"("tenantId", "threadId", "createdAt"); + +-- CreateIndex +CREATE INDEX "agent_subagent_runs_expiresAt_idx" ON "agent_subagent_runs"("expiresAt"); + +-- CreateIndex +CREATE UNIQUE INDEX "agent_subagent_runs_tenantId_threadId_subagentId_key" ON "agent_subagent_runs"("tenantId", "threadId", "subagentId"); diff --git a/libs/prisma/schema.prisma b/libs/prisma/schema.prisma index fdb7f526..e875e392 100644 --- a/libs/prisma/schema.prisma +++ b/libs/prisma/schema.prisma @@ -627,6 +627,36 @@ model ChatMessage { @@map("chat_messages") } +// AgentSubagentRun — one row per dispatch_agent sub-agent, so a collapsed card +// can be expanded after a page reload. The transcript lives here and NEVER in +// the orchestrator's message list: putting it there would defeat the context +// isolation that makes sub-agents worthwhile. +// +// Written through SubagentRunPostgresRepository, which redacts secrets before +// the row is persisted (see lib/agent/subagent-redact.ts) — aws_read permits +// `lambda get-function-configuration`, whose Environment.Variables are plaintext. +model AgentSubagentRun { + id String @id @default(cuid()) + tenantId String + threadId String + subagentId String + role String + task String + status String // running|done|failed + toolCount Int @default(0) + tokensIn Int @default(0) + tokensOut Int @default(0) + summary String? + transcript Json? + createdAt DateTime @default(now()) + expiresAt DateTime // 30-day TTL, matching ChatMessage + + @@unique([tenantId, threadId, subagentId]) + @@index([tenantId, threadId, createdAt]) + @@index([expiresAt]) + @@map("agent_subagent_runs") +} + // ChatSession — thread metadata for AI agent chat sessions // Tracks thread title, owner, timestamps for sidebar listing model ChatSession { From 5daa14b44782c265983d49cb970d58f9a03396b7 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 26 Jul 2026 18:41:29 +0530 Subject: [PATCH 37/42] fix(chat): keep the heartbeat writing when no sub-agent is in flight MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The heartbeat tick walked `liveSubagents` only, and `emitSubagent` deletes each sub-agent from that map on its terminal event. So the tick wrote zero bytes exactly during the orchestrator's post-fan-out call — the longest silence and the largest context in the run — which is the connection CloudFront's 60s originReadTimeout drops. It protected the cheap window and abandoned the expensive one. Extract the tick body into `buildHeartbeatChunks` in stream-parts.ts (pure, exported, covered) and make it always yield at least one chunk: a transient `data-keepalive` part when nothing is in flight. Transient parts reach the client but are never appended to `message.parts`, and `deriveRunState` has no case for `data-keepalive`, so the keep-alive moves no UI state. The tick body had to leave route.ts to be testable at all: nothing in the suite imports route.ts, so deleting `clearInterval(heartbeat)` outright still left 95/95 green. Co-Authored-By: Claude Opus 5 (1M context) --- .../api/chat/__tests__/stream-parts.test.ts | 55 ++++++++++++++++++- apps/web-ui/app/api/chat/route.ts | 14 +++-- apps/web-ui/app/api/chat/stream-parts.ts | 33 +++++++++++ .../agent/chat/__tests__/run-state.test.ts | 16 ++++++ 4 files changed, 113 insertions(+), 5 deletions(-) diff --git a/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts b/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts index b2022559..fddbeba3 100644 --- a/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts +++ b/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts @@ -1,7 +1,7 @@ import { describe, it, expect } from 'vitest'; import { buildMemoryPart, humanizeReflection, humanizePlanning, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from '@/app/api/chat/stream-parts'; import { buildUsagePart } from '../stream-parts'; -import { buildSubagentPart } from '../stream-parts'; +import { buildSubagentPart, buildHeartbeatChunks } from '../stream-parts'; const WM_JSON = JSON.stringify({ summary: 'User asked to connect Jira; site resource retrieved.', @@ -264,3 +264,56 @@ describe('buildSubagentPart', () => { expect((part.data as { summary?: string }).summary).toBe('found things'); }); }); + +// The heartbeat exists solely to keep bytes flowing so CloudFront's 60s +// originReadTimeout (infra/compute/index.ts:962) can't kill a long run. Its tick +// body lives here — not inline in route.ts — because nothing in the suite imports +// route.ts, so an inline tick is untested by construction. +describe('buildHeartbeatChunks', () => { + const base = { + id: 'sa-1', role: 'EC2 auditor', task: 'audit account 1', + toolCount: 3, tokensIn: 900, tokensOut: 120, + }; + + it('re-emits one data-subagent part per in-flight sub-agent and no keep-alive', () => { + const chunks = buildHeartbeatChunks([ + { ...base, id: 'sa-1', status: 'running' }, + { ...base, id: 'sa-2', status: 'running' }, + ]); + expect(chunks).toHaveLength(2); + expect(chunks.map((c) => c.type)).toEqual(['data-subagent', 'data-subagent']); + expect(chunks.map((c: any) => c.id)).toEqual(['subagent-sa-1', 'subagent-sa-2']); + expect(chunks.some((c) => c.type === 'data-keepalive')).toBe(false); + }); + + // REGRESSION GUARD for F1. `liveSubagents` empties as sub-agents reach their + // terminal event, so a tick that only walks that map writes ZERO bytes during + // the orchestrator's post-fan-out call — the longest silence and the largest + // context in the run, exactly when the 60s origin timeout bites. A tick MUST + // write something. This test fails if the empty-case branch is removed. + it('emits exactly one keep-alive chunk when no sub-agent is in flight', () => { + const chunks = buildHeartbeatChunks([]); + expect(chunks).toHaveLength(1); + expect(chunks[0].type).toBe('data-keepalive'); + }); + + it('marks the keep-alive transient and gives it no sub-agent fields', () => { + const [chunk] = buildHeartbeatChunks([]) as any[]; + // transient parts are delivered to the client but never appended to + // message.parts, so a keep-alive cannot bloat the message or move the reducer. + expect(chunk.transient).toBe(true); + expect(chunk.id).toBeUndefined(); + expect(chunk.data).toEqual({}); + expect(JSON.stringify(chunk)).not.toMatch(/sa-1|EC2 auditor|audit account|transcript/); + }); + + it('still omits the transcript for a terminal event passed through the tick', () => { + const chunks = buildHeartbeatChunks([{ + ...base, status: 'done', summary: 'found things', + transcript: [{ kind: 'ai', text: 'internal reasoning' }], + }]); + expect(chunks).toHaveLength(1); + expect(JSON.stringify(chunks[0])).not.toContain('internal reasoning'); + expect(JSON.stringify(chunks[0])).not.toContain('transcript'); + }); +}); diff --git a/apps/web-ui/app/api/chat/route.ts b/apps/web-ui/app/api/chat/route.ts index 6c7a6bf3..f4da8582 100644 --- a/apps/web-ui/app/api/chat/route.ts +++ b/apps/web-ui/app/api/chat/route.ts @@ -9,7 +9,7 @@ import { triageChatMessage } from '@/lib/agent/triage'; import { respondDirect } from './direct-chat'; import { resolveKnowledgeBaseIds } from '@/lib/agent/auto-kb-select'; import { acquireThreadLock, releaseThreadLock as releaseThreadLockDb } from '@/lib/agent/thread-lock'; -import { buildPlanPart, buildPhasePart, buildInterruptParts, buildMemoryPart, buildUsagePart, buildSubagentPart, humanizeReflection, humanizePlanning, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from './stream-parts'; +import { buildPlanPart, buildPhasePart, buildInterruptParts, buildMemoryPart, buildUsagePart, buildSubagentPart, buildHeartbeatChunks, humanizeReflection, humanizePlanning, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from './stream-parts'; import { resolveResumedToolCallId, type ResumedPendingCall } from './resume-tool-id'; import type { PlanStep } from '@/lib/agent/agent-shared'; import { logRunSummary } from '@/lib/agent/run-timings'; @@ -852,11 +852,17 @@ function processStream( // CloudFront's originReadTimeout is 60s (infra/compute/index.ts:962). A // sub-agent can think for 90s without producing a byte, so re-emit the - // in-flight cards on a timer to keep the connection alive. + // in-flight cards on a timer to keep the connection alive. The tick body + // is buildHeartbeatChunks (stream-parts.ts) — pure and tested — and it + // ALWAYS yields at least one chunk: `liveSubagents` empties as sub-agents + // finish, and the orchestrator's post-fan-out call is the longest silence + // in the run, so a map-dependent tick would go quiet exactly when the + // timeout bites. Nothing may be inserted between this setInterval and the + // `try` below — the matching clearInterval lives in that try's finally. const HEARTBEAT_MS = 15_000; const heartbeat = setInterval(() => { - for (const event of liveSubagents.values()) { - safeEnqueue(buildSubagentPart(event) as UIMessageChunk); + for (const chunk of buildHeartbeatChunks(liveSubagents.values())) { + safeEnqueue(chunk); } }, HEARTBEAT_MS); diff --git a/apps/web-ui/app/api/chat/stream-parts.ts b/apps/web-ui/app/api/chat/stream-parts.ts index 5e9bf0b1..710dea3e 100644 --- a/apps/web-ui/app/api/chat/stream-parts.ts +++ b/apps/web-ui/app/api/chat/stream-parts.ts @@ -1,3 +1,4 @@ +import type { UIMessageChunk } from 'ai'; import { pendingToolCallsOf } from '@/lib/agent/guard'; import { extractJsonObject } from '@/lib/agent/llm-json'; import type { GuardVerdict, PlanStep, ReflectionState } from '@/lib/agent/agent-shared'; @@ -54,6 +55,38 @@ export function buildSubagentPart(event: SubagentEvent): DataPart { }; } +/** + * The body of the sub-agent heartbeat tick, as a pure function so it is testable + * (nothing in the suite imports route.ts, so an inline tick body has no coverage + * at all). + * + * Two jobs, and the second one is the whole reason the heartbeat exists: + * - re-emit a card for every sub-agent still in flight, so a sub-agent that + * thinks for 90s without producing a byte still refreshes its card; + * - guarantee the tick writes SOMETHING even when nothing is in flight. The + * caller's live map empties as sub-agents reach their terminal event, and the + * orchestrator's post-fan-out call — every sub-agent's findings in the prompt — + * is both the longest silence and the largest context in the run. A + * map-dependent tick would protect the cheap window and go silent through the + * expensive one, which is exactly the connection CloudFront's 60s + * originReadTimeout (infra/compute/index.ts:962) drops. + * + * The keep-alive is `transient`, so the client receives the bytes but never + * appends the part to `message.parts`: it cannot bloat the message and + * `deriveRunState` (components/agent/chat/run-state.ts) has no case for + * `data-keepalive`, so it touches no UI state. + */ +export function buildHeartbeatChunks(live: Iterable): UIMessageChunk[] { + const chunks: UIMessageChunk[] = []; + for (const event of live) { + chunks.push(buildSubagentPart(event)); + } + if (chunks.length === 0) { + chunks.push({ type: 'data-keepalive', data: {}, transient: true }); + } + return chunks; +} + // Exact markers from planning-agent.ts's reflectNode `feedback` template (do not // change independently of that template): // `🔍 **Reflection Analysis:**\n${analysis}\n\n` + diff --git a/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts b/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts index e252d225..8ce93896 100644 --- a/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts +++ b/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts @@ -54,6 +54,22 @@ describe('deriveRunState', () => { expect(rs.pendingApproval).toBeNull(); }); + // The heartbeat's keep-alive (buildHeartbeatChunks in app/api/chat/stream-parts.ts) + // is a transient part, so it should never land in message.parts at all — but if it + // ever does, it must move NOTHING: no UI affordance may light up because the + // connection was merely kept open. + it('ignores a data-keepalive part entirely — no structured data, no phase, no subagents', () => { + const rs = deriveRunState([msg([{ type: 'data-keepalive', data: {} }])] as any, new Set()); + expect(rs.hasStructuredData).toBe(false); + expect(rs.usedSubagents).toBe(false); + expect(rs.subagents).toEqual([]); + expect(rs.phases).toEqual([]); + expect(rs.currentPhase).toBe('text'); + expect(rs.pendingApproval).toBeNull(); + expect(rs.pendingClarifications).toEqual([]); + expect(rs.tokenUsage).toEqual({ input: 0, output: 0 }); + }); + it('legacy thread (no data parts) → hasStructuredData false, empty plan', () => { const rs = deriveRunState([msg([{ type: 'reasoning', text: 'PLANNING_PHASE_START\nsteps…' }])] as any, new Set()); expect(rs.hasStructuredData).toBe(false); From d6db4d49ae0eab9466bae91f3a94a180c9c459f5 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Mon, 27 Jul 2026 10:38:32 +0530 Subject: [PATCH 38/42] fix(agent): close the three secret-at-rest leaks found by the branch review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit B1 — dispatch_agent returned the sub-agent report unredacted. That value becomes a ToolMessage, and the chat route persists every new message verbatim, so it reached two sinks the subagent repository never sees: chat_messages (30-day TTL, and replayed to the browser by /api/threads/[threadId]/history) and the LangGraph checkpoint tables, which have no TTL at all. Redacting at the source covers all three sinks and the orchestrator's own context; the repository call stays as defence-in-depth. B2 — `cloudformation describe-stacks` is allowlisted and returns {ParameterKey, ParameterValue} pairs, where the secret sits under a key that is not secret-shaped and is named by a SIBLING field. Key-name matching was structurally blind to it. Adds a sibling name/value rule (covering ParameterKey/ Value, OutputKey/Value, {name,value} and {Key,Value}) plus credential-in-URL rewriting for the connection strings that carry no key at all. Both rules run on the structural walk and in the regex fallback. Tags and ordinary parameters are left byte-identical. F3 — tool output was truncated BEFORE the transcript was built, so truncated JSON no longer parsed and redaction silently fell back to its weaker regex path, which has no Environment.Variables location rule. A real get-function-configuration exceeds the 4000-char cap routinely, so the weak path was the common path. The transcript is now redacted then truncated. The model-visible copy stays raw: the location rule blanks every value under Variables, which is right for storage and wrong for a live agent that must still reason about what it just read. Co-Authored-By: Claude Opus 5 (1M context) --- .../agent/__tests__/subagent-redact.test.ts | 104 ++++++++++++++++++ .../lib/agent/dispatch-agent-tool.test.ts | 29 +++++ apps/web-ui/lib/agent/dispatch-agent-tool.ts | 10 +- apps/web-ui/lib/agent/subagent-redact.ts | 78 ++++++++++++- apps/web-ui/lib/agent/subagent.test.ts | 61 ++++++++++ apps/web-ui/lib/agent/subagent.ts | 28 ++++- 6 files changed, 302 insertions(+), 8 deletions(-) diff --git a/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts b/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts index fb22b1c2..8d8b0b3d 100644 --- a/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts +++ b/apps/web-ui/lib/agent/__tests__/subagent-redact.test.ts @@ -62,6 +62,110 @@ describe('redactTranscript — secret-shaped keys anywhere in the document', () }); }); +describe('redactTranscript — sibling name/value pairs', () => { + // `cloudformation describe-stacks` is allowlisted in aws-read-tool.ts and is + // exactly what an audit sub-agent gets pointed at. Its secret lives in a + // `ParameterValue` — a key that is not secret-shaped — named by a SIBLING + // field, so key-name matching is structurally blind to it. CloudFormation + // masks `NoEcho: true` parameters, but a non-NoEcho `DBPassword` is common + // and is the very misconfiguration the audit is looking for. + const describeStacks = JSON.stringify({ + Stacks: [{ + StackName: 'payments-prod', + Parameters: [ + { ParameterKey: 'DBPassword', ParameterValue: 'hunter2' }, + { ParameterKey: 'InstanceType', ParameterValue: 't3.medium' }, + ], + Outputs: [{ OutputKey: 'DbUrl', OutputValue: 'postgres://u:s3cr3t!@db.internal:5432/x' }], + }], + }); + + const stack = () => JSON.parse(redactTranscript([entry(describeStacks)])![0].text).Stacks[0]; + + it('redacts ParameterValue when the sibling ParameterKey names a secret', () => { + expect(stack().Parameters[0].ParameterValue).toBe(REDACTED); + }); + + it('keeps the sibling name so the operator sees WHICH parameter was withheld', () => { + expect(stack().Parameters.map((p: { ParameterKey: string }) => p.ParameterKey)) + .toEqual(['DBPassword', 'InstanceType']); + expect(stack().Outputs[0].OutputKey).toBe('DbUrl'); + expect(stack().StackName).toBe('payments-prod'); + }); + + it('leaves an ordinary parameter untouched', () => { + // A redactor that eats `t3.medium` fails the operator as badly as one that leaks. + expect(stack().Parameters[1].ParameterValue).toBe('t3.medium'); + }); + + it('leaks neither secret anywhere in the output', () => { + const flat = flatten(redactTranscript([entry(describeStacks)])); + expect(flat).not.toContain('hunter2'); + expect(flat).not.toContain('s3cr3t!'); + }); + + it('handles the lowercase {name,value} form used by container env definitions', () => { + const payload = JSON.stringify({ + environment: [ + { name: 'DB_PASSWORD', value: 'hunter2' }, + { name: 'LOG_LEVEL', value: 'debug' }, + ], + }); + const out = JSON.parse(redactTranscript([entry(payload)])![0].text); + + expect(out.environment[0].value).toBe(REDACTED); + expect(out.environment[0].name).toBe('DB_PASSWORD'); + // Named by the operator, not secret-shaped: it must survive. + expect(out.environment[1].value).toBe('debug'); + }); + + it('handles the bare {Key,Value} form', () => { + const out = JSON.parse(redactTranscript([entry('{"Key":"ApiToken","Value":"abc123"}')])![0].text); + expect(out.Value).toBe(REDACTED); + expect(out.Key).toBe('ApiToken'); + }); + + it('does not redact a Value whose sibling name is innocuous', () => { + // Tags are the highest-traffic {Key,Value} shape in AWS responses. + const payload = JSON.stringify({ Tags: [{ Key: 'Name', Value: 'web-1' }, { Key: 'Owner', Value: 'platform' }] }); + expect(redactTranscript([entry(payload)])![0].text).toBe(payload); + }); + + it('applies the sibling rule inside the regex fallback path', () => { + // Truncated JSON never parses, so the structural walk cannot run. + const truncated = '{"Parameters":[{"ParameterKey":"DBPassword","ParameterValue":"hunter2"},{"Param'; + expect(flatten(redactTranscript([entry(truncated)]))).not.toContain('hunter2'); + }); +}); + +describe('redactTranscript — credentials embedded in URLs', () => { + it('rewrites scheme://user:password@host, keeping user and host', () => { + const out = JSON.parse(redactTranscript([entry('{"DbUrl":"postgres://admin:letmein@db.internal:5432/app"}')])![0].text); + expect(out.DbUrl).toBe(`postgres://admin:${REDACTED}@db.internal:5432/app`); + }); + + it('rewrites a credential URL in prose that never parsed as JSON', () => { + // This is the report path: LLM-authored prose, no key to match on. + const out = redactTranscript('The worker uses mongodb://svc:p4ssw0rd@cluster0.mongodb.net/prod'); + expect(out).not.toContain('p4ssw0rd'); + expect(out).toContain('cluster0.mongodb.net/prod'); + }); + + it('handles an empty user segment', () => { + expect(redactTranscript('redis://:t0psecret@cache.internal:6379')).not.toContain('t0psecret'); + }); + + it('leaves credential-free URLs and ARNs alone', () => { + const payload = JSON.stringify({ + Endpoint: 'https://console.aws.amazon.com/ec2/home?region=us-east-1', + Repo: 'ssh://git@github.com/acme/infra.git', + Bucket: 's3://nucleus-artifacts/builds/1.2.3', + Arn: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db-AbCdEf', + }); + expect(redactTranscript([entry(payload)])![0].text).toBe(payload); + }); +}); + describe('redactTranscript — ordinary values survive', () => { it('leaves a realistic non-secret payload byte-identical', () => { const payload = JSON.stringify({ diff --git a/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts b/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts index b66c06f4..7ac6c8f0 100644 --- a/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts +++ b/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts @@ -88,6 +88,35 @@ describe('dispatch_agent tool', () => { expect(result).not.toContain('INTERNAL_REASONING_MARKER'); }); + it('redacts the report before it reaches the orchestrator', async () => { + // The return value becomes a ToolMessage in the orchestrator's message list, + // and route.ts persists every new message VERBATIM. That reaches two at-rest + // sinks the subagent repository's redaction does not cover: `chat_messages` + // (30-day TTL, and replayed to the browser by /api/threads/[id]/history) and + // the LangGraph PostgresSaver checkpoint tables, which have NO TTL at all. + vi.mocked(runSubagent).mockResolvedValue({ + ...okResult, + report: 'api-worker connects to postgres://admin:letmein@db.internal:5432/app', + }); + + const result = await makeTool().invoke({ role: 'a', task: 't', expectedOutput: 'e' }); + + expect(result).not.toContain('letmein'); + // Redacting must not change the type — LangGraph requires a string. + expect(typeof result).toBe('string'); + // The finding stays actionable: only the credential is withheld. + expect(result).toContain('db.internal'); + }); + + it('leaves a report with no secrets in it byte-identical', async () => { + // Over-redaction fails the operator too: a report the model worked hard to + // compose must survive the boundary unchanged when it carries nothing secret. + const clean = 'Findings: i-0abc123 (t3.medium) idle at 2% CPU; arn:aws:lambda:us-east-1:123456789012:function:api-worker last modified 2026-07-26.'; + vi.mocked(runSubagent).mockResolvedValue({ ...okResult, report: clean }); + + expect(await makeTool().invoke({ role: 'a', task: 't', expectedOutput: 'e' })).toBe(clean); + }); + it('degrades gracefully instead of throwing when the budget is exhausted', async () => { const tool = makeTool({ ...BUDGET, maxSubagentsPerRun: 1 }); await tool.invoke({ role: 'a', task: 't', expectedOutput: 'e' }); diff --git a/apps/web-ui/lib/agent/dispatch-agent-tool.ts b/apps/web-ui/lib/agent/dispatch-agent-tool.ts index d6c86bdc..8e319ae9 100644 --- a/apps/web-ui/lib/agent/dispatch-agent-tool.ts +++ b/apps/web-ui/lib/agent/dispatch-agent-tool.ts @@ -15,6 +15,7 @@ import { randomUUID } from 'crypto'; import { Semaphore } from './concurrency'; import { runSubagent, type SubagentTranscriptEntry } from './subagent'; import type { SubagentBudgetConfig } from './subagent-budget'; +import { redactTranscript } from './subagent-redact'; export interface SubagentEvent { id: string; @@ -123,7 +124,14 @@ export function createDispatchAgentTool(deps: DispatchAgentDeps) { transcript: result.transcript, }); - return result.report; + // Redact HERE, not only in the repository. This return value becomes a + // ToolMessage in the orchestrator's message list, and the chat route + // persists every new message verbatim — reaching two at-rest sinks the + // repository never sees: `chat_messages` (30-day TTL, and replayed to + // the browser by /api/threads/[threadId]/history) and the LangGraph + // checkpoint tables, which have no TTL at all. Redacting at the source + // covers all three sinks and the orchestrator's own context. + return redactTranscript(result.report); } catch (error) { // runSubagent is already total, but a semaphore or emit failure must // not propagate into ToolNode and abort the orchestrator's turn. diff --git a/apps/web-ui/lib/agent/subagent-redact.ts b/apps/web-ui/lib/agent/subagent-redact.ts index def57389..6334e665 100644 --- a/apps/web-ui/lib/agent/subagent-redact.ts +++ b/apps/web-ui/lib/agent/subagent-redact.ts @@ -48,6 +48,57 @@ function isSecretKey(key: string): boolean { /** Alternation used by the non-JSON fallback. Ordered longest-first so `api_key` wins over `key`. */ const SECRET_KEY_ALTERNATION = '[A-Za-z0-9_.\\-]*(?:password|passwd|secret|token|api[_-]?key|credential|private[_-]?key)[A-Za-z0-9_.\\-]*'; +/** + * Credentials embedded in a URL: `scheme://user:password@host` → the password is + * replaced, the user and host kept. No key name is involved, so neither the key-name + * nor the location rule can see these — and an `OutputValue` holding a connection + * string is a realistic `describe-stacks` response. + * + * The userinfo segment forbids `/` so a path containing `@` (an S3 key, an ARN) can + * never be mistaken for credentials, and it requires a `:` so `ssh://git@github.com` + * is left alone. + */ +const URL_CREDENTIAL_RE = /\b([A-Za-z][A-Za-z0-9+.\-]*:\/\/)([^\s:/@]*):([^\s/@]+)@/g; + +function redactUrlCredentials(text: string): string { + return text.replace(URL_CREDENTIAL_RE, (_m, scheme: string, user: string) => `${scheme}${user}:${REDACTED}@`); +} + +/** + * Sibling name/value pairs: `{ParameterKey: 'DBPassword', ParameterValue: 'hunter2'}`. + * + * The secret sits under a key that is NOT secret-shaped (`ParameterValue`); what names + * it is a *sibling* field. Key-name matching is structurally blind to this, and it is + * the shape `cloudformation describe-stacks` returns — the exact call an audit + * sub-agent makes. The same shape covers container env definitions + * (`{name, value}`) and bare `{Key, Value}` tags. + * + * Only the value is redacted, never the name, so an operator still sees WHICH + * parameter was withheld. + * + * @returns the field names in this object whose values must be redacted. + */ +function siblingSecretFields(obj: Record): Set { + const redact = new Set(); + const byLower = new Map(); + for (const key of Object.keys(obj)) byLower.set(key.toLowerCase(), key); + + for (const [lower, actual] of byLower) { + const name = obj[actual]; + // The NAME must itself be a string, and it is the string we test for + // secret-ness — not the key it is stored under. + if (typeof name !== 'string' || !isSecretKey(name)) continue; + + for (const suffix of ['key', 'name']) { + if (!lower.endsWith(suffix)) continue; + // `ParameterKey` pairs with `ParameterValue`; `Key` with `Value`. + const partner = byLower.get(`${lower.slice(0, -suffix.length)}value`); + if (partner !== undefined && partner !== actual) redact.add(partner); + } + } + return redact; +} + /** * Redact a parsed JSON value in place-free fashion. * @@ -57,7 +108,9 @@ const SECRET_KEY_ALTERNATION = '[A-Za-z0-9_.\\-]*(?:password|passwd|secret|token function redactValue(value: unknown, depth: number, seen: WeakSet, redactAll: boolean): unknown { if (depth > MAX_DEPTH) return REDACTED; if (value === null || typeof value !== 'object') { - return redactAll ? REDACTED : value; + if (redactAll) return REDACTED; + // A credential can hide inside an otherwise innocuous string value. + return typeof value === 'string' ? redactUrlCredentials(value) : value; } const asObject = value as object; @@ -70,9 +123,12 @@ function redactValue(value: unknown, depth: number, seen: WeakSet, redac return value.map(item => redactValue(item, depth + 1, seen, redactAll)); } + const record = value as Record; + const siblingSecrets = redactAll ? null : siblingSecretFields(record); + const out: Record = {}; - for (const [key, child] of Object.entries(value as Record)) { - if (redactAll || isSecretKey(key)) { + for (const [key, child] of Object.entries(record)) { + if (redactAll || isSecretKey(key) || siblingSecrets?.has(key)) { // Redact the whole subtree: a secret-named key may hold an object. out[key] = redactValue(child, depth + 1, seen, true); continue; @@ -86,9 +142,25 @@ function redactValue(value: unknown, depth: number, seen: WeakSet, redac return out; } +/** + * Sibling name/value pairs in text the structural walk could not parse. Truncated JSON + * is the common case, not the exotic one: a sub-agent's tool output is capped, so a + * large `describe-stacks` response arrives here with its tail cut off. + */ +const SIBLING_PAIR_TEXT_RE = + /("[A-Za-z0-9_.\-]*(?:Key|Name)"\s*:\s*")((?:\\.|[^"])*)("\s*,\s*"[A-Za-z0-9_.\-]*Value"\s*:\s*")(?:\\.|[^"])*(")/gi; + /** Regex scrubbing for text that is not JSON (shell commands, prose, truncated output). */ function redactPlainText(text: string): string { let out = text; + // Credentials inside a URL — no key name to match on, so this must run here too. + out = redactUrlCredentials(out); + // {"ParameterKey":"DBPassword","ParameterValue":"…"} — the name is a sibling field. + out = out.replace( + SIBLING_PAIR_TEXT_RE, + (whole, open: string, name: string, mid: string, close: string) => + isSecretKey(name) ? `${open}${name}${mid}${REDACTED}${close}` : whole, + ); // "key": "value" / 'key': 'value' out = out.replace( new RegExp(`(["']?${SECRET_KEY_ALTERNATION}["']?\\s*:\\s*)(["'])(?:\\\\.|(?!\\2).)*\\2`, 'gi'), diff --git a/apps/web-ui/lib/agent/subagent.test.ts b/apps/web-ui/lib/agent/subagent.test.ts index da2b03f8..c72fa973 100644 --- a/apps/web-ui/lib/agent/subagent.test.ts +++ b/apps/web-ui/lib/agent/subagent.test.ts @@ -181,6 +181,67 @@ describe('runSubagent', () => { }); }); +describe('transcript redaction runs before truncation', () => { + /** Per-tool output cap inside a sub-agent (subagent.ts). */ + const CAP = 4000; + + /** + * A `lambda get-function-configuration` response comfortably over the cap — + * which a real one is, routinely. `BOOTSTRAP` is deliberately NOT secret-shaped: + * only the `Environment.Variables` LOCATION rule catches it, and that rule needs + * JSON that still parses. Truncate first and the redactor falls through to its + * regex path, where the strongest rule does not exist. + */ + function bigLambdaConfig(): string { + const vars: Record = { BOOTSTRAP: 'hunter2', DB_PASSWORD: 'letmein' }; + for (let i = 0; i < 120; i++) vars[`PLAIN_SETTING_${i}`] = 'x'.repeat(30); + return JSON.stringify({ FunctionName: 'api-worker', Environment: { Variables: vars } }); + } + + const lambdaRun = async (payload: string) => { + const tool = { name: 'get_function_configuration', invoke: vi.fn(async () => payload) }; + const model = scriptedModel([ + { content: 'reading the configuration', tool_calls: [{ id: 't1', name: 'get_function_configuration', args: {} }] }, + { content: 'BOOTSTRAP is set in plaintext.' }, + ]); + const result = await runSubagent(SPEC, { model, tools: [tool], budget: BUDGET }); + return { result, model }; + }; + + it('keeps the location rule alive on an over-cap payload', async () => { + const payload = bigLambdaConfig(); + expect(payload.length).toBeGreaterThan(CAP); + + const { result } = await lambdaRun(payload); + const toolEntry = result.transcript.find(e => e.kind === 'tool')!; + + expect(toolEntry.text).not.toContain('hunter2'); + // The key survives: an operator must still see that BOOTSTRAP exists. + expect(toolEntry.text).toContain('BOOTSTRAP'); + }); + + it('still caps the transcript entry at the per-tool limit', async () => { + const { result } = await lambdaRun(bigLambdaConfig()); + const toolEntry = result.transcript.find(e => e.kind === 'tool')!; + + expect(toolEntry.text.length).toBeLessThanOrEqual(CAP + 3); + }); + + it('does not change what the sub-agent model sees', async () => { + // Redaction is a PERSISTENCE-boundary concern. The location rule redacts + // EVERY value under `Variables`, benign ones included — correct for at-rest + // storage, wrong for a live agent that has to answer "which function still + // points at the deprecated table". The model-visible copy stays raw. + const payload = bigLambdaConfig(); + const { model } = await lambdaRun(payload); + + const messages = model.invoke.mock.calls[0][0] as Array<{ content?: unknown }>; + const toolMessage = messages.find(m => typeof m.content === 'string' && (m.content as string).includes('FunctionName'))!; + + expect(toolMessage.content).toBe(payload.slice(0, CAP) + '...'); + }); +}); + describe('shell is unavailable to sub-agents', () => { it('refuses every bash-like tool name in any case', () => { for (const name of ['bash', 'shell', 'run_command', 'execute_command', 'Execute_Command', 'EXECUTE_COMMAND']) { diff --git a/apps/web-ui/lib/agent/subagent.ts b/apps/web-ui/lib/agent/subagent.ts index 30e4dc99..d40af5af 100644 --- a/apps/web-ui/lib/agent/subagent.ts +++ b/apps/web-ui/lib/agent/subagent.ts @@ -17,6 +17,7 @@ import { classifyTool } from './tool-classifier'; import { hasSubagentReadOnlyMarker } from './subagent-tool-marker'; import { contentToText, truncateOutput } from './agent-shared'; import type { SubagentBudgetConfig } from './subagent-budget'; +import { redactTranscript } from './subagent-redact'; /** ~1500 tokens. Enforced in characters so the bound is deterministic and testable. */ export const SUBAGENT_REPORT_MAX_CHARS = 6000; @@ -275,19 +276,38 @@ async function runSubagentLoop( try { const raw = await tool.invoke(call.args ?? {}); - return { call, executed: true, output: truncateOutput(typeof raw === 'string' ? raw : JSON.stringify(raw), SUBAGENT_TOOL_OUTPUT_MAX_CHARS) }; + const text = typeof raw === 'string' ? raw : JSON.stringify(raw); + return { + call, + executed: true, + output: truncateOutput(text, SUBAGENT_TOOL_OUTPUT_MAX_CHARS), + // Redact BEFORE truncating. Truncated JSON no longer parses, which + // silently downgrades redaction to its weaker regex path — and that + // path has no location-based `Environment.Variables` rule. A real + // `get-function-configuration` exceeds the cap routinely, so + // truncate-first would make the weak path the COMMON path. + // + // Only the persisted transcript is redacted. The model-visible copy + // above stays raw: the location rule blanks every value under + // `Variables`, benign ones included, which is right for at-rest + // storage and wrong for a live agent that must still answer + // questions about the configuration it just read. + transcriptText: truncateOutput(redactTranscript(text), SUBAGENT_TOOL_OUTPUT_MAX_CHARS), + }; } catch (error) { // A failing tool is data, not a crash — the sub-agent should report it. - return { call, executed: true, output: `ERROR: ${error instanceof Error ? error.message : String(error)}` }; + // The message can echo the arguments, so it is redacted like any output. + const message = `ERROR: ${error instanceof Error ? error.message : String(error)}`; + return { call, executed: true, output: message, transcriptText: redactTranscript(message) }; } })); - for (const { call, executed, output } of results) { + for (const { call, executed, output, transcriptText } of results) { // Counted from the verdict, not from the output text: a real tool whose // output happens to begin with "REFUSED:" (an echoed IAM denial) is a // call that actually ran and must be billed as one. if (executed) toolCount++; - transcript.push({ kind: 'tool', name: call.name, text: output }); + transcript.push({ kind: 'tool', name: call.name, text: transcriptText ?? output }); messages.push(new ToolMessage({ content: output, tool_call_id: call.id ?? `${call.name}-${toolCount}` })); } From 96d293c838b9ae4c2af032724823fb377d795ef0 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Mon, 27 Jul 2026 22:15:01 +0530 Subject: [PATCH 39/42] fix(agent): make the sub-agent token budget an actual ceiling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `tryReserve` gated on settled spend, but a sub-agent only reports usage when it finishes — and ToolNode dispatches a turn's `dispatch_agent` calls concurrently. Every reservation in a fan-out therefore saw `tokensSpent === 0`, so the token budget was never consulted for the fan-out it exists to bound. Only the count limit ever bit: 16 sub-agents were admitted against a 50,000-token budget, with a true worst case of `maxSubagentsPerRun x subagentMaxIterations` model calls and no token brake at all. The field is a user-facing slider, so it over-promised. Reserve an estimate at dispatch and reconcile on completion. The ledger now keeps `tokensSpent` (reconciled) alongside `tokensReserved` (in flight) and admits only when both plus the new estimate fit. `tryReserve` returns an idempotent `reservation` handle; `recordSpend` is gone, so nothing can spend without first reserving. The release runs in a `finally`, covering success, internal failure, timeout and the catch path -- a reservation stranded by a failed sub-agent would otherwise starve the run of capacity it never used. The estimate is quadratic in `subagentMaxIterations`, because a sub-agent re-sends its transcript on every model call; a linear estimate under-reserves by 40% at the maximum iteration count, exactly where a ceiling must hold. At the shipped defaults the token gate admits 8 while `maxConcurrentSubagents` is 3, so the concurrency cap still binds first and parallelism is unchanged. Refusals now distinguish the count limit, an exhausted budget, and the misconfiguration where one sub-agent cannot fit in the whole budget. Co-Authored-By: Claude Opus 5 (1M context) --- .../lib/agent/dispatch-agent-tool.test.ts | 165 +++++++++++++++++- apps/web-ui/lib/agent/dispatch-agent-tool.ts | 100 ++++++++++- 2 files changed, 253 insertions(+), 12 deletions(-) diff --git a/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts b/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts index 7ac6c8f0..0800799e 100644 --- a/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts +++ b/apps/web-ui/lib/agent/dispatch-agent-tool.test.ts @@ -6,17 +6,27 @@ vi.mock('./subagent', async (importOriginal) => { }); import { runSubagent } from './subagent'; -import { createRunBudgetLedger, createDispatchAgentTool } from './dispatch-agent-tool'; +import { createRunBudgetLedger, createDispatchAgentTool, estimateSubagentTokens } from './dispatch-agent-tool'; const BUDGET = { enabled: true, maxConcurrentSubagents: 2, maxSubagentsPerRun: 3, - maxSubagentTokensPerRun: 1000, + // Large enough that the COUNT limit is what bites in the tests below that are + // about the count. Token-ceiling behaviour gets its own describe block, where + // the budget is sized deliberately against `estimateSubagentTokens`. + maxSubagentTokensPerRun: 200_000, subagentMaxIterations: 4, subagentTimeoutMs: 5000, }; +/** Reserve and immediately settle, i.e. "this much has already been spent". */ +function spend(ledger: ReturnType, tokensIn: number, tokensOut: number) { + const verdict = ledger.tryReserve(); + if (!verdict.ok) throw new Error(`expected a reservation, got: ${verdict.reason}`); + verdict.reservation.settle(tokensIn, tokensOut); +} + const okResult = { report: 'Findings: i-123 idle', toolCount: 2, tokensIn: 300, tokensOut: 50, status: 'done' as const, transcript: [], @@ -41,7 +51,7 @@ describe('createRunBudgetLedger', () => { it('refuses once the token budget is spent', () => { const ledger = createRunBudgetLedger(BUDGET); - ledger.recordSpend(900, 200); + spend(ledger, 190_000, 5_000); const verdict = ledger.tryReserve(); expect(verdict.ok).toBe(false); @@ -54,6 +64,111 @@ describe('createRunBudgetLedger', () => { }); }); +describe('createRunBudgetLedger — the token ceiling actually binds (F4)', () => { + it('bounds a concurrent fan-out by the TOKEN budget, not only the count', () => { + // THE DEFECT. ToolNode dispatches a turn's dispatch_agent calls concurrently, + // so every tryReserve runs BEFORE any sub-agent finishes and reports usage. + // A ledger that gates on settled spend alone therefore consults the token + // budget only after the fan-out it exists to bound has already been granted. + const budget = { ...BUDGET, maxSubagentsPerRun: 16, maxSubagentTokensPerRun: 50_000 }; + const ledger = createRunBudgetLedger(budget); + const estimate = estimateSubagentTokens(budget); + + // No settle() calls in between — this is the real fan-out ordering. + const verdicts = Array.from({ length: 16 }, () => ledger.tryReserve()); + const granted = verdicts.filter(v => v.ok).length; + + expect(granted).toBe(Math.floor(50_000 / estimate)); + expect(granted).toBeLessThan(16); + const refusal = verdicts.find(v => !v.ok) as { reason: string }; + expect(refusal.reason).toMatch(/token budget/i); + }); + + it('reconciles: a sub-agent that came in under estimate frees the difference', () => { + const budget = { ...BUDGET, maxSubagentsPerRun: 16, maxSubagentTokensPerRun: 50_000 }; + const ledger = createRunBudgetLedger(budget); + const estimate = estimateSubagentTokens(budget); + const capacity = Math.floor(50_000 / estimate); + + const held = Array.from({ length: capacity }, () => ledger.tryReserve()); + expect(held.every(v => v.ok)).toBe(true); + expect(ledger.tryReserve().ok).toBe(false); + + // Each one actually cost a fraction of its reservation. Releasing the + // difference must let the run keep fanning out. + for (const v of held) { + if (v.ok) v.reservation.settle(100, 20); + } + expect(ledger.tryReserve().ok).toBe(true); + }); + + it('settle is idempotent — a double release cannot mint budget', () => { + const budget = { ...BUDGET, maxSubagentsPerRun: 16, maxSubagentTokensPerRun: 50_000 }; + const ledger = createRunBudgetLedger(budget); + const estimate = estimateSubagentTokens(budget); + + const first = ledger.tryReserve(); + if (!first.ok) throw new Error('expected a reservation'); + first.reservation.settle(estimate, 0); + first.reservation.settle(estimate, 0); + first.reservation.settle(estimate, 0); + + // One sub-agent that spent exactly its estimate leaves room for the rest — + // no more, no less. Triple-settling must not have triple-counted the spend + // (which would refuse too early) nor triple-released (which would overspend). + const remaining = Array.from({ length: 16 }, () => ledger.tryReserve()).filter(v => v.ok).length; + expect(remaining).toBe(Math.floor(50_000 / estimate) - 1); + }); + + it('distinguishes the count refusal from the token refusal', () => { + const countBound = createRunBudgetLedger({ ...BUDGET, maxSubagentsPerRun: 1, maxSubagentTokensPerRun: 1_000_000 }); + countBound.tryReserve(); + const byCount = countBound.tryReserve() as { reason: string }; + + const tokenBound = createRunBudgetLedger({ ...BUDGET, maxSubagentsPerRun: 16, maxSubagentTokensPerRun: 50_000 }); + let byToken = tokenBound.tryReserve(); + while (byToken.ok) byToken = tokenBound.tryReserve(); + + expect(byCount.reason).toMatch(/sub-agent limit/i); + expect(byCount.reason).not.toMatch(/token/i); + expect((byToken as { reason: string }).reason).toMatch(/token budget/i); + }); + + it('says so plainly when one sub-agent cannot fit in the whole budget', () => { + // A tenant who sets the minimum token budget alongside the maximum iteration + // count has configured a run that can afford zero sub-agents. Refusing with + // "token budget exhausted" before anything ran would be baffling; the message + // has to name the misconfiguration. + const ledger = createRunBudgetLedger({ + ...BUDGET, subagentMaxIterations: 16, maxSubagentTokensPerRun: 50_000, + }); + const verdict = ledger.tryReserve(); + + expect(verdict.ok).toBe(false); + expect((verdict as { reason: string }).reason).toMatch(/exceeds the entire/i); + }); + + it('holds no state across runs', () => { + const budget = { ...BUDGET, maxSubagentsPerRun: 16, maxSubagentTokensPerRun: 50_000 }; + const exhausted = createRunBudgetLedger(budget); + let v = exhausted.tryReserve(); + while (v.ok) v = exhausted.tryReserve(); + + expect(createRunBudgetLedger(budget).tryReserve().ok).toBe(true); + }); + + it('checks and increments synchronously, with no await in between', () => { + // Two concurrent callers must never both observe the same free capacity. + // An `await` anywhere between the check and the increment would reintroduce + // exactly that race, so the call must not be thenable at all. + const ledger = createRunBudgetLedger(BUDGET); + const verdict = ledger.tryReserve(); + + expect(verdict).not.toBeInstanceOf(Promise); + expect((verdict as unknown as { then?: unknown }).then).toBeUndefined(); + }); +}); + describe('dispatch_agent tool', () => { const makeTool = (budget = BUDGET) => createDispatchAgentTool({ model: {}, @@ -167,6 +282,50 @@ describe('dispatch_agent tool', () => { expect(peak).toBeLessThanOrEqual(2); }); + describe('reservation release (F4)', () => { + // A stranded reservation is worse than no ceiling at all: it silently starves + // the rest of the run. Every exit path must hand its capacity back. + const budget = { ...BUDGET, maxSubagentsPerRun: 16, maxSubagentTokensPerRun: 200_000 }; + const estimate = estimateSubagentTokens(budget); + const capacity = Math.floor(200_000 / estimate); + + type ResultOverride = Partial> & { status?: 'done' | 'failed' }; + + const runOnceWith = async (result: ResultOverride | Error) => { + const ledger = createRunBudgetLedger(budget); + const tool = createDispatchAgentTool({ model: {}, subagentTools: [], ledger, budget }); + if (result instanceof Error) vi.mocked(runSubagent).mockRejectedValue(result); + else vi.mocked(runSubagent).mockResolvedValue({ ...okResult, ...result }); + + await tool.invoke({ role: 'a', task: 't', expectedOutput: 'e' }); + return ledger; + }; + + /** How many more sub-agents this ledger will admit. */ + const remainingCapacity = (ledger: ReturnType) => + Array.from({ length: 16 }, () => ledger.tryReserve()).filter(v => v.ok).length; + + it('releases on success, charging only what was actually used', async () => { + const ledger = await runOnceWith({ tokensIn: 300, tokensOut: 50 }); + // 350 real tokens is a rounding error against the budget, so the run keeps + // essentially all of its capacity — it must NOT still be holding a full + // estimate for a sub-agent that has already finished. + expect(remainingCapacity(ledger)).toBe(Math.floor((200_000 - 350) / estimate)); + }); + + it('releases when the sub-agent times out or fails internally', async () => { + const ledger = await runOnceWith({ status: 'failed', tokensIn: 100, tokensOut: 0 }); + expect(remainingCapacity(ledger)).toBe(Math.floor((200_000 - 100) / estimate)); + }); + + it('releases when the sub-agent runtime rejects outright', async () => { + // The catch path: a semaphore or sink failure spent no tokens, so the + // reservation must come back whole. + const ledger = await runOnceWith(new Error('semaphore exploded')); + expect(remainingCapacity(ledger)).toBe(capacity); + }); + }); + it('does not reject when the progress sink throws', async () => { // The chat route's sink enqueues onto a ReadableStream, which throws once // the client disconnects — so a throwing sink is the normal client-abort diff --git a/apps/web-ui/lib/agent/dispatch-agent-tool.ts b/apps/web-ui/lib/agent/dispatch-agent-tool.ts index 8e319ae9..a143aeec 100644 --- a/apps/web-ui/lib/agent/dispatch-agent-tool.ts +++ b/apps/web-ui/lib/agent/dispatch-agent-tool.ts @@ -29,9 +29,48 @@ export interface SubagentEvent { transcript?: SubagentTranscriptEntry[]; } +/** + * Tokens a single model call carries beyond the transcript: the sub-agent system + * prompt, the standalone task brief, and the report instructions. + */ +const PROMPT_BASE_TOKENS = 2_000; + +/** + * Tokens one tool result contributes. Derived, not guessed: `subagent.ts` caps + * every tool output at SUBAGENT_TOOL_OUTPUT_MAX_CHARS (4000) before it enters the + * message list, and ~4 characters per token is the standard approximation. + */ +const TOOL_RESULT_TOKENS = 1_000; + +/** + * Worst-case token cost of one sub-agent, used to reserve capacity at dispatch. + * + * A sub-agent re-sends its whole transcript on every model call, so cost is + * QUADRATIC in the iteration count, not linear: iteration i carries i-1 prior tool + * results. Summing over N iterations gives N·base + (N²/2)·result. A linear + * estimate looks safe at the default N=8 and badly under-reserves at the maximum + * N=16 — which is precisely where a ceiling needs to hold. + * + * Over-estimating costs only CONCURRENT admission, never throughput: the + * reservation is released and replaced by actual usage the moment the sub-agent + * finishes, so later dispatches in the same run reserve against real numbers. + */ +export function estimateSubagentTokens(budget: SubagentBudgetConfig): number { + const n = Math.max(1, budget.subagentMaxIterations); + return n * PROMPT_BASE_TOKENS + Math.ceil((n * n) / 2) * TOOL_RESULT_TOKENS; +} + +/** Capacity held by one in-flight sub-agent, returned when it settles. */ +export interface BudgetReservation { + /** + * Release this reservation and charge what the sub-agent actually used. + * Idempotent — a double settle can neither mint budget nor double-charge. + */ + settle(tokensIn: number, tokensOut: number): void; +} + export interface RunBudgetLedger { - tryReserve(): { ok: true } | { ok: false; reason: string }; - recordSpend(tokensIn: number, tokensOut: number): void; + tryReserve(): { ok: true; reservation: BudgetReservation } | { ok: false; reason: string }; semaphore: Semaphore; } @@ -41,7 +80,11 @@ export interface RunBudgetLedger { */ export function createRunBudgetLedger(budget: SubagentBudgetConfig): RunBudgetLedger { let dispatched = 0; + /** Reconciled usage, from sub-agents that have finished. */ let tokensSpent = 0; + /** Estimated capacity held by sub-agents still in flight. */ + let tokensReserved = 0; + const estimate = estimateSubagentTokens(budget); const semaphore = new Semaphore(budget.maxConcurrentSubagents); return { @@ -52,14 +95,41 @@ export function createRunBudgetLedger(budget: SubagentBudgetConfig): RunBudgetLe if (dispatched >= budget.maxSubagentsPerRun) { return { ok: false, reason: `per-run sub-agent limit reached (${budget.maxSubagentsPerRun})` }; } - if (tokensSpent >= budget.maxSubagentTokensPerRun) { - return { ok: false, reason: `sub-agent token budget exhausted (${budget.maxSubagentTokensPerRun})` }; + // Gate on spent + IN-FLIGHT + this one. ToolNode dispatches a turn's + // dispatch_agent calls concurrently, so gating on settled spend alone + // consults the budget only after the fan-out it exists to bound has + // already been granted — every reservation would see tokensSpent === 0. + if (tokensSpent + tokensReserved + estimate > budget.maxSubagentTokensPerRun) { + const reason = estimate > budget.maxSubagentTokensPerRun + // Not exhaustion — a misconfiguration. Nothing has run, and + // nothing ever will, so name the two settings involved rather + // than reporting an empty budget as spent. + ? `one sub-agent's estimated cost (${estimate} tokens at ${budget.subagentMaxIterations} iterations) exceeds the entire run token budget (${budget.maxSubagentTokensPerRun}) — raise the token budget or lower the iteration limit` + : `sub-agent token budget exhausted (${budget.maxSubagentTokensPerRun})`; + return { ok: false, reason }; } + + // Check and increment stay in one synchronous block. An `await` + // anywhere between them would let two concurrent callers observe the + // same free capacity and both be admitted. dispatched++; - return { ok: true }; - }, - recordSpend(tokensIn: number, tokensOut: number) { - tokensSpent += tokensIn + tokensOut; + tokensReserved += estimate; + + let settled = false; + return { + ok: true, + reservation: { + settle(tokensIn: number, tokensOut: number) { + // Guard against a double release: the dispatch tool settles + // in a `finally`, and an unusual path could reach it twice. + // Releasing twice would mint capacity that was never held. + if (settled) return; + settled = true; + tokensReserved -= estimate; + tokensSpent += tokensIn + tokensOut; + }, + }, + }; }, semaphore, }; @@ -102,6 +172,12 @@ export function createDispatchAgentTool(deps: DispatchAgentDeps) { emit({ status: 'running' }); + // Actual usage, reconciled against the reservation in the `finally` + // below. A sub-agent that throws before reporting usage spent nothing + // we can attribute, so its reservation comes back whole. + let actualIn = 0; + let actualOut = 0; + try { const result = await deps.ledger.semaphore.run(() => runSubagent( { role, task, expectedOutput }, @@ -113,7 +189,8 @@ export function createDispatchAgentTool(deps: DispatchAgentDeps) { }, )); - deps.ledger.recordSpend(result.tokensIn, result.tokensOut); + actualIn = result.tokensIn; + actualOut = result.tokensOut; emit({ status: result.status === 'failed' ? 'failed' : 'done', @@ -139,6 +216,11 @@ export function createDispatchAgentTool(deps: DispatchAgentDeps) { console.error(`[dispatch_agent] "${role}" failed: ${message}`); emit({ status: 'failed', summary: message }); return `The sub-agent "${role}" failed: ${message}. Continue with the information you already have, or perform this work yourself.`; + } finally { + // Must run on EVERY exit path — success, internal failure, timeout, + // and the catch above. A reservation stranded by a failed sub-agent + // starves the rest of the run of capacity it never actually used. + reservation.reservation.settle(actualIn, actualOut); } }, { From 32b87c4e0ea1f1ee8f571f1a697d42f554b21733 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 2 Aug 2026 22:27:43 +0530 Subject: [PATCH 40/42] feat(aiops): make all agent behavior settings tenant/UI-driven, no env vars Chat triage, the four memory subsystems, autonomous skill creation, the skill-synthesis thresholds and the run iteration cap move from env kill-switches to the 'aiops-features' tenant-config key, edited from the AI Ops console. Defaults reproduce the old env-unset behavior exactly; a per-tenant 30s cache keeps the hot gates synchronous. SUBAGENTS_ENABLED inverts to an emergency-off (unset = enabled; the tenant toggle governs). Triage/auto-select now treat skill matching as the DEFAULT, returning null only when nothing in the catalog is even loosely related. Co-Authored-By: Claude Fable 5 --- .env.example | 21 +-- apps/web-ui/app/api/settings/aiops/route.ts | 43 +++++ .../app/agent-ops/settings/subagents/page.tsx | 4 +- .../settings/aiops-feature-settings.tsx | 161 ++++++++++++++++++ .../settings/aiops-subagent-settings.tsx | 6 +- apps/web-ui/env.ts | 7 - apps/web-ui/lib/agent/aiops-features.test.ts | 62 +++++++ apps/web-ui/lib/agent/aiops-features.ts | 154 +++++++++++++++++ apps/web-ui/lib/agent/memory-nodes.ts | 12 +- apps/web-ui/lib/agent/memory/episode.test.ts | 11 +- apps/web-ui/lib/agent/memory/episode.ts | 9 +- .../lib/agent/memory/procedural.test.ts | 8 +- apps/web-ui/lib/agent/memory/procedural.ts | 7 +- .../web-ui/lib/agent/memory/reconcile.test.ts | 8 +- apps/web-ui/lib/agent/memory/reconcile.ts | 7 +- .../lib/agent/memory/skill-synthesis.test.ts | 10 +- .../lib/agent/memory/skill-synthesis.ts | 23 ++- .../lib/agent/memory/working-memory.test.ts | 12 +- .../web-ui/lib/agent/memory/working-memory.ts | 9 +- apps/web-ui/lib/agent/subagent-budget.test.ts | 6 +- apps/web-ui/lib/agent/subagent-budget.ts | 14 +- apps/web-ui/lib/agent/triage.test.ts | 16 +- apps/web-ui/lib/agent/triage.ts | 22 ++- apps/web-ui/lib/queries/aiops-settings.ts | 33 ++++ 24 files changed, 563 insertions(+), 102 deletions(-) create mode 100644 apps/web-ui/components/settings/aiops-feature-settings.tsx create mode 100644 apps/web-ui/lib/agent/aiops-features.test.ts create mode 100644 apps/web-ui/lib/agent/aiops-features.ts diff --git a/.env.example b/.env.example index 75dbf9a7..6242027d 100644 --- a/.env.example +++ b/.env.example @@ -90,35 +90,22 @@ DEFAULT_TENANT_ID=org-default # Working memory (Phase 1) — in-session context compaction for long-running agents. # When over WORKING_MEMORY_TOKEN_BUDGET estimated tokens, older turns are folded into # a rolling summary + scratchpad (via the reflector model) so multi-hour runs survive -# the context window. Set WORKING_MEMORY_ENABLED=false for identical legacy behavior. -WORKING_MEMORY_ENABLED=true WORKING_MEMORY_TOKEN_BUDGET=60000 WORKING_MEMORY_KEEP_RECENT=8 # Save-time memory reconciliation (Phase 2) — dedup/contradiction resolution via an # LLM judge before writing. Set false to restore blind upserts (legacy behavior). -MEMORY_RECONCILE_ENABLED=true # Episodic memory (Phase 3) — capture one distilled episode per tool-using run and # replay similar past episodes as few-shot experience at task start. -EPISODIC_MEMORY_ENABLED=true # Procedural memory (Phase 4) — learn operating rules from corrections/failures and # inject them as '### Operating rules (learned)'; promote matured rules to Skills (human-approved). -PROCEDURAL_MEMORY_ENABLED=true # Hermes autonomy + observability # MEMORY_LOG_VERBOSE: print full injected memoryContext blocks (summary lines are always on). -# CHAT_TRIAGE_ENABLED: chat-layer router — conversational messages get a direct reply instead of the agent workflow; the same call auto-picks the skill. false = always run the workflow (legacy behavior). # (Auto skill selection/loading has no env flag — it is always available and toggled per-session via the "Auto skills" switch in the AIOps console.) -# AUTO_SKILL_CREATION_ENABLED: matured procedural domains auto-become/refresh enabled system skills (read-only tier). -# AUTO_SKILL_MATURITY_THRESHOLD: accessCount a rule needs before promoting. MEMORY_LOG_VERBOSE=true -CHAT_TRIAGE_ENABLED=true -AUTO_SKILL_CREATION_ENABLED=true -AUTO_SKILL_MATURITY_THRESHOLD=3 -# SKILL_SYNTHESIS_MIN_RULES: matured rules a domain needs before it earns a synthesized skill. -SKILL_SYNTHESIS_MIN_RULES=3 # ============================================================================ # Optional integrations @@ -148,11 +135,13 @@ LANGFUSE_HOST=https://cloud.langfuse.com # Skip audit DB writes in local dev # SKIP_AUDIT_LOGGING=true -# AI Ops sub-agents (Layer B). SUBAGENTS_ENABLED is an absolute platform -# kill-switch — tenant settings cannot re-enable it when false. +# AI Ops sub-agents (Layer B). The feature is UI-driven: each tenant enables it +# and tunes its budget from the AI Ops console settings — no env var needed. +# SUBAGENTS_ENABLED=false is an EMERGENCY deployment-wide kill-switch (tenant +# settings cannot re-enable while it is set); leave it unset in normal operation. # The SUBAGENT_MAX_* vars are CEILINGS that lower (never raise) the built-in # maximums a tenant may configure. -SUBAGENTS_ENABLED=false +# SUBAGENTS_ENABLED=false # SUBAGENT_MAX_CONCURRENCY=6 # SUBAGENT_MAX_PER_RUN=16 # SUBAGENT_MAX_TOKENS_PER_RUN=1000000 diff --git a/apps/web-ui/app/api/settings/aiops/route.ts b/apps/web-ui/app/api/settings/aiops/route.ts index aac85abf..b89ae25b 100644 --- a/apps/web-ui/app/api/settings/aiops/route.ts +++ b/apps/web-ui/app/api/settings/aiops/route.ts @@ -13,6 +13,14 @@ import { resolveSubagentBudget, validateBudgetInput, } from '@/lib/agent/subagent-budget'; +import { + AIOPS_FEATURES_KEY, + FEATURE_BOUNDS, + clampFeatures, + resolveAiopsFeatures, + validateFeaturesInput, + primeAiopsFeaturesCache, +} from '@/lib/agent/aiops-features'; export async function GET() { console.log('API - GET /api/settings/aiops - Fetching sub-agent budget'); @@ -27,6 +35,7 @@ export async function GET() { } const budget = await resolveSubagentBudget(tenantId); + const features = await resolveAiopsFeatures(tenantId); return NextResponse.json({ success: true, @@ -36,6 +45,8 @@ export async function GET() { // ceiling is what it is rather than silently clipping input. bounds: BUDGET_BOUNDS, platformEnabled: platformSubagentsEnabled(), + features, + featureBounds: FEATURE_BOUNDS, }, }); } catch (error) { @@ -60,6 +71,38 @@ export async function PUT(request: NextRequest) { } const body = await request.json(); + + // { features: {...} } saves the feature flags; a budget-shaped body keeps + // the original sub-agent semantics (backward compatible). + if (body && typeof body === 'object' && 'features' in body) { + const featError = validateFeaturesInput(body.features); + if (featError) { + return NextResponse.json({ success: false, error: featError }, { status: 400 }); + } + const features = clampFeatures(body.features); + const session = await getServerSession(authOptions); + const updatedBy = session?.user?.email || 'api-user'; + await TenantConfigService.saveConfig(AIOPS_FEATURES_KEY, features, tenantId, updatedBy); + // Apply immediately on this replica; others converge within the cache TTL. + primeAiopsFeaturesCache(tenantId, features); + await AuditService.logUserAction({ + eventType: 'aiops.features.settings.updated', + severity: 'medium', + apiRoute: 'PUT /api/settings/aiops', + httpMethod: 'PUT', + action: 'Update AI Ops Feature Settings', + resourceType: 'settings', + resourceId: AIOPS_FEATURES_KEY, + resourceName: 'AI Ops Features', + user: updatedBy, + userType: 'user', + status: 'success', + details: `triage=${features.chatTriageEnabled} workingMem=${features.workingMemoryEnabled} episodic=${features.episodicMemoryEnabled} procedural=${features.proceduralMemoryEnabled} reconcile=${features.memoryReconcileEnabled} autoSkill=${features.autoSkillCreationEnabled} minRules=${features.skillSynthesisMinRules} maxIterations=${features.maxIterations}`, + tenantId, + }); + return NextResponse.json({ success: true, data: { features } }); + } + const validationError = validateBudgetInput(body); if (validationError) { return NextResponse.json({ success: false, error: validationError }, { status: 400 }); diff --git a/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx b/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx index 6b184a5b..218adf00 100644 --- a/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx +++ b/apps/web-ui/app/app/agent-ops/settings/subagents/page.tsx @@ -1,10 +1,12 @@ 'use client'; import { AiopsSubagentSettings } from '@/components/settings/aiops-subagent-settings'; +import { AiopsFeatureSettings } from '@/components/settings/aiops-feature-settings'; export default function AiopsSubagentSettingsPage() { return ( -
+
+
); diff --git a/apps/web-ui/components/settings/aiops-feature-settings.tsx b/apps/web-ui/components/settings/aiops-feature-settings.tsx new file mode 100644 index 00000000..0e15dbf2 --- /dev/null +++ b/apps/web-ui/components/settings/aiops-feature-settings.tsx @@ -0,0 +1,161 @@ +'use client'; + +/** + * AI Ops feature settings card — the tenant-facing switchboard for behaviors + * that used to be env-var kill-switches (chat triage, the memory subsystems, + * autonomous skill creation, the run iteration cap). Entirely UI-driven; the + * server clamps numeric values against FEATURE_BOUNDS on read and write. + * + * Rendered next to AiopsSubagentSettings in the AI Ops console settings dialog + * and on the /app/agent-ops/settings/subagents page. + */ +import { useEffect, useState } from 'react'; +import { toast } from 'sonner'; + +import { Button } from '@/components/ui/button'; +import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card'; +import { Input } from '@/components/ui/input'; +import { Label } from '@/components/ui/label'; +import { Switch } from '@/components/ui/switch'; +import { Spinner } from '@/components/ui/spinner'; +import { + useAiopsSubagentSettings, + useSaveAiopsFeatureSettings, + type AiopsFeatureConfig, +} from '@/lib/queries/aiops-settings'; + +const TOGGLES: Array<{ key: keyof AiopsFeatureConfig; label: string; help: string }> = [ + { + key: 'chatTriageEnabled', + label: 'Smart routing (chat triage)', + help: 'One quick classifier call answers greetings/small talk directly and routes real work to the full agent — it also auto-picks the matching skill. Off: every message runs the whole planning graph.', + }, + { + key: 'workingMemoryEnabled', + label: 'Working memory', + help: 'Compacts long runs in-session (summary folding) so the agent keeps its train of thought within the context budget.', + }, + { + key: 'episodicMemoryEnabled', + label: 'Episodic memory', + help: 'Distills one episode (context → actions → outcome) per tool-using run and replays similar past sessions as experience.', + }, + { + key: 'proceduralMemoryEnabled', + label: 'Procedural memory', + help: 'Learns operating rules from corrections and failures, injected as "Operating rules (learned)" on later runs.', + }, + { + key: 'memoryReconcileEnabled', + label: 'Memory reconciliation', + help: 'An LLM judge merges new facts into existing memories (update/supersede) instead of piling up near-duplicates.', + }, + { + key: 'autoSkillCreationEnabled', + label: 'Autonomous skill creation', + help: 'When a domain accumulates enough matured rules, the system authors a read-only sys- skill from them. Disabling a created skill vetoes it.', + }, +]; + +const NUMBERS: Array<{ key: 'autoSkillMaturityThreshold' | 'skillSynthesisMinRules' | 'maxIterations'; label: string; help: string }> = [ + { + key: 'maxIterations', + label: 'Max iterations per run', + help: 'Executor loop cap for a single run (planning and fast agents). Higher = more thorough and more expensive.', + }, + { + key: 'autoSkillMaturityThreshold', + label: 'Rule maturity threshold', + help: 'Times a learned rule must be recalled before it counts as matured.', + }, + { + key: 'skillSynthesisMinRules', + label: 'Matured rules per auto-skill', + help: 'Matured rules a domain needs before an autonomous skill is synthesized.', + }, +]; + +export function AiopsFeatureSettings() { + const { data, isLoading } = useAiopsSubagentSettings(); + const saveMutation = useSaveAiopsFeatureSettings(); + const [draft, setDraft] = useState(null); + + // Initialize the draft once from the server value; later refetches must not + // clobber unsaved edits. + useEffect(() => { + if (data?.features && !draft) setDraft(data.features); + }, [data, draft]); + + if (isLoading || !draft) { + return ( +
+ Loading… +
+ ); + } + + const bounds = data?.featureBounds; + + const save = () => { + saveMutation.mutate(draft, { + onSuccess: (saved) => { + setDraft(saved); + toast.success('AI Ops settings saved', { + description: 'Applied to new runs immediately on this instance; within ~30s everywhere.', + }); + }, + onError: (err) => toast.error('Could not save AI Ops settings', { description: err.message }), + }); + }; + + return ( + + + Agent behavior + + Routing, memory, and skill-learning features for this organization. All settings live + here — no server configuration involved. + + + + {TOGGLES.map(({ key, label, help }) => ( +
+
+ +

{help}

+
+ setDraft({ ...draft, [key]: checked })} + /> +
+ ))} + +
+ {NUMBERS.map(({ key, label, help }) => ( +
+ + setDraft({ ...draft, [key]: Number(e.target.value) })} + /> +

+ {help} + {bounds?.[key] ? ` (${bounds[key].min}–${bounds[key].max})` : ''} +

+
+ ))} +
+ + + + + ); +} diff --git a/apps/web-ui/components/settings/aiops-subagent-settings.tsx b/apps/web-ui/components/settings/aiops-subagent-settings.tsx index 6e292695..e1605747 100644 --- a/apps/web-ui/components/settings/aiops-subagent-settings.tsx +++ b/apps/web-ui/components/settings/aiops-subagent-settings.tsx @@ -140,8 +140,10 @@ export function AiopsSubagentSettings() { {!platformEnabled && ( - Sub-agents are disabled for this deployment, so these settings are read-only. An - administrator must enable the feature before they can be changed. + Sub-agents are switched off for this whole deployment by the emergency kill-switch + (SUBAGENTS_ENABLED=false), so these settings are read-only until an administrator + removes it. In normal operation no environment variable is needed — the toggle + below is the only control. )} diff --git a/apps/web-ui/env.ts b/apps/web-ui/env.ts index 94f9a422..f8fefa96 100644 --- a/apps/web-ui/env.ts +++ b/apps/web-ui/env.ts @@ -86,17 +86,10 @@ export const env = createEnv({ WEBHOOK_SECRET: z.string().optional(), // Feature flags / misc - WORKING_MEMORY_ENABLED: z.string().optional(), WORKING_MEMORY_TOKEN_BUDGET: z.string().optional(), WORKING_MEMORY_KEEP_RECENT: z.string().optional(), - MEMORY_RECONCILE_ENABLED: z.string().optional(), - EPISODIC_MEMORY_ENABLED: z.string().optional(), - PROCEDURAL_MEMORY_ENABLED: z.string().optional(), MEMORY_LOG_VERBOSE: z.string().optional(), AUTO_SKILL_SELECTION_ENABLED: z.string().optional(), - AUTO_SKILL_CREATION_ENABLED: z.string().optional(), - AUTO_SKILL_MATURITY_THRESHOLD: z.string().optional(), - SKILL_SYNTHESIS_MIN_RULES: z.string().optional(), USE_PG_SCHEDULES: z.string().optional(), DUAL_WRITE_SCHEDULES: z.string().optional(), INTERNAL_API_KEY: z.string().optional(), diff --git a/apps/web-ui/lib/agent/aiops-features.test.ts b/apps/web-ui/lib/agent/aiops-features.test.ts new file mode 100644 index 00000000..48d70340 --- /dev/null +++ b/apps/web-ui/lib/agent/aiops-features.test.ts @@ -0,0 +1,62 @@ +import { describe, it, expect, vi } from 'vitest'; + +vi.mock('@/lib/tenant-config-service', () => ({ TenantConfigService: { getConfig: vi.fn() } })); +import { TenantConfigService } from '@/lib/tenant-config-service'; +import { + DEFAULT_FEATURES, FEATURE_BOUNDS, clampFeatures, validateFeaturesInput, + getAiopsFeatures, resolveAiopsFeatures, primeAiopsFeaturesCache, +} from './aiops-features'; + +describe('clampFeatures', () => { + it('null input → defaults (identical to the historical env-unset behavior)', () => { + expect(clampFeatures(null)).toEqual(DEFAULT_FEATURES); + }); + + it('clamps numerics into bounds and ignores junk types', () => { + const out = clampFeatures({ + chatTriageEnabled: false, + maxIterations: 9999, + skillSynthesisMinRules: 0, + episodicMemoryEnabled: 'yes' as never, + }); + expect(out.chatTriageEnabled).toBe(false); + expect(out.maxIterations).toBe(FEATURE_BOUNDS.maxIterations.max); + expect(out.skillSynthesisMinRules).toBe(FEATURE_BOUNDS.skillSynthesisMinRules.min); + expect(out.episodicMemoryEnabled).toBe(true); // junk → default + }); +}); + +describe('validateFeaturesInput', () => { + it('accepts a partial valid payload', () => { + expect(validateFeaturesInput({ chatTriageEnabled: true, maxIterations: 20 })).toBeNull(); + }); + it('rejects non-objects, wrong types, and out-of-bounds values', () => { + expect(validateFeaturesInput(null)).toMatch(/object/); + expect(validateFeaturesInput({ chatTriageEnabled: 'true' })).toMatch(/boolean/); + expect(validateFeaturesInput({ maxIterations: 3.5 })).toMatch(/integer/); + expect(validateFeaturesInput({ maxIterations: 999 })).toMatch(/between/); + }); +}); + +describe('cache', () => { + it('sync read returns defaults cold, primed value after, and no tenant → defaults', () => { + vi.mocked(TenantConfigService.getConfig).mockResolvedValue(null as never); + expect(getAiopsFeatures()).toEqual(DEFAULT_FEATURES); + expect(getAiopsFeatures('t-cold')).toEqual(DEFAULT_FEATURES); + primeAiopsFeaturesCache('t-primed', { ...DEFAULT_FEATURES, chatTriageEnabled: false }); + expect(getAiopsFeatures('t-primed').chatTriageEnabled).toBe(false); + }); + + it('resolveAiopsFeatures reads the store fresh and clamps', async () => { + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ maxIterations: 9999 } as never); + const out = await resolveAiopsFeatures('t-fresh'); + expect(out.maxIterations).toBe(FEATURE_BOUNDS.maxIterations.max); + // and the sync cache is primed by the resolve + expect(getAiopsFeatures('t-fresh').maxIterations).toBe(FEATURE_BOUNDS.maxIterations.max); + }); + + it('a store failure degrades to defaults instead of throwing', async () => { + vi.mocked(TenantConfigService.getConfig).mockRejectedValue(new Error('db down')); + await expect(resolveAiopsFeatures('t-err')).resolves.toEqual(DEFAULT_FEATURES); + }); +}); diff --git a/apps/web-ui/lib/agent/aiops-features.ts b/apps/web-ui/lib/agent/aiops-features.ts new file mode 100644 index 00000000..0ed94bcc --- /dev/null +++ b/apps/web-ui/lib/agent/aiops-features.ts @@ -0,0 +1,154 @@ +/** + * AI Ops feature settings — tenant-configured from the AI Ops console + * ("AI Ops settings" dialog), with NO environment-variable dependency. + * + * Every flag here used to be an env kill-switch (CHAT_TRIAGE_ENABLED, + * *_MEMORY_ENABLED, AUTO_SKILL_CREATION_ENABLED, …). They are now rows on the + * 'aiops-features' tenant-config key; the defaults below reproduce the old + * env-unset behavior exactly (everything on, minRules 3, 30 iterations), so a + * tenant that never opens the settings dialog sees no change. + * + * READ PATH: the gates that consume these flags (triage, memory nodes, the + * agent loops) are hot and often synchronous, so reads go through a small + * per-tenant TTL cache: + * - getAiopsFeatures(tenantId) — SYNC. Returns the cached value (or the + * defaults on a cold cache) and kicks off a background refresh when stale. + * A toggle flipped in the UI therefore applies within CACHE_TTL_MS on other + * replicas / later runs — good enough for feature flags, and it keeps the + * gates free of async plumbing. + * - resolveAiopsFeatures(tenantId) — async, always-fresh; used by the + * settings API and at graph construction (which is already async). + */ +import { TenantConfigService } from '@/lib/tenant-config-service'; + +export const AIOPS_FEATURES_KEY = 'aiops-features'; + +export interface AiopsFeatureConfig { + /** One cheap classifier call routes chat messages direct-reply vs full agent graph (and auto-picks the skill). */ + chatTriageEnabled: boolean; + /** In-session compaction for long runs (summary folding + budget-aware window). */ + workingMemoryEnabled: boolean; + /** Distill one episode per tool-using run; replayed as few-shot experience. */ + episodicMemoryEnabled: boolean; + /** Learn operating rules from corrections/failures; injected as "Operating rules (learned)". */ + proceduralMemoryEnabled: boolean; + /** LLM judge (ADD/UPDATE/SUPERSEDE/…) applied when saving memories. */ + memoryReconcileEnabled: boolean; + /** Autonomous synthesis of sys- skills from matured procedural rules. */ + autoSkillCreationEnabled: boolean; + /** Rules whose accessCount reaches this are "matured". */ + autoSkillMaturityThreshold: number; + /** Matured rules a domain needs before a sys- skill is synthesized. */ + skillSynthesisMinRules: number; + /** Executor/generator loop cap for both the planning and fast agents. */ + maxIterations: number; +} + +type NumericKey = 'autoSkillMaturityThreshold' | 'skillSynthesisMinRules' | 'maxIterations'; + +export const FEATURE_BOUNDS: Record = { + autoSkillMaturityThreshold: { min: 1, max: 10, default: 3 }, + skillSynthesisMinRules: { min: 1, max: 10, default: 3 }, + maxIterations: { min: 5, max: 50, default: 30 }, +}; + +const BOOLEAN_KEYS = [ + 'chatTriageEnabled', 'workingMemoryEnabled', 'episodicMemoryEnabled', + 'proceduralMemoryEnabled', 'memoryReconcileEnabled', 'autoSkillCreationEnabled', +] as const; + +export const DEFAULT_FEATURES: AiopsFeatureConfig = Object.freeze({ + chatTriageEnabled: true, + workingMemoryEnabled: true, + episodicMemoryEnabled: true, + proceduralMemoryEnabled: true, + memoryReconcileEnabled: true, + autoSkillCreationEnabled: true, + autoSkillMaturityThreshold: FEATURE_BOUNDS.autoSkillMaturityThreshold.default, + skillSynthesisMinRules: FEATURE_BOUNDS.skillSynthesisMinRules.default, + maxIterations: FEATURE_BOUNDS.maxIterations.default, +}); + +/** Clamp on READ, like the sub-agent budget: a stored row never widens bounds. */ +export function clampFeatures(input: Partial | null): AiopsFeatureConfig { + const out: Record = {}; + for (const key of BOOLEAN_KEYS) { + out[key] = typeof input?.[key] === 'boolean' ? input[key] : DEFAULT_FEATURES[key]; + } + for (const key of Object.keys(FEATURE_BOUNDS) as NumericKey[]) { + const spec = FEATURE_BOUNDS[key]; + const raw = Number(input?.[key]); + const value = Number.isFinite(raw) ? Math.round(raw) : spec.default; + out[key] = Math.min(spec.max, Math.max(spec.min, value)); + } + return out as unknown as AiopsFeatureConfig; +} + +/** Validate a PUT payload. Returns an error string for a 400, or null. */ +export function validateFeaturesInput(input: unknown): string | null { + if (!input || typeof input !== 'object' || Array.isArray(input)) { + return 'features must be an object'; + } + const body = input as Record; + for (const key of BOOLEAN_KEYS) { + if (body[key] !== undefined && typeof body[key] !== 'boolean') { + return `${key} must be a boolean`; + } + } + for (const key of Object.keys(FEATURE_BOUNDS) as NumericKey[]) { + if (body[key] === undefined) continue; + if (typeof body[key] !== 'number' || !Number.isInteger(body[key] as number)) { + return `${key} must be an integer`; + } + const spec = FEATURE_BOUNDS[key]; + const value = body[key] as number; + if (value < spec.min || value > spec.max) { + return `${key} must be between ${spec.min} and ${spec.max}`; + } + } + return null; +} + +// --------------------------------------------------------------------------- +// Per-tenant cache +// --------------------------------------------------------------------------- + +const CACHE_TTL_MS = 30_000; +const cache = new Map(); +const inflight = new Set(); + +async function refresh(tenantId: string): Promise { + const stored = await TenantConfigService + .getConfig>(AIOPS_FEATURES_KEY, tenantId) + .catch(() => null); + const value = clampFeatures(stored); + cache.set(tenantId, { value, at: Date.now() }); + return value; +} + +/** + * SYNC read for hot gates. Cold cache returns DEFAULT_FEATURES (identical to + * the historical env-unset behavior) while a background refresh populates it. + * Never throws: a config-store failure degrades to defaults. + */ +export function getAiopsFeatures(tenantId?: string | null): AiopsFeatureConfig { + if (!tenantId) return DEFAULT_FEATURES; + const hit = cache.get(tenantId); + if ((!hit || Date.now() - hit.at > CACHE_TTL_MS) && !inflight.has(tenantId)) { + inflight.add(tenantId); + void refresh(tenantId) + .catch(() => { /* degraded to defaults */ }) + .finally(() => inflight.delete(tenantId)); + } + return hit?.value ?? DEFAULT_FEATURES; +} + +/** Always-fresh read; primes the sync cache. For the settings API + graph factories. */ +export async function resolveAiopsFeatures(tenantId: string): Promise { + return refresh(tenantId); +} + +/** Prime the cache after a PUT so the saving replica applies the change immediately. */ +export function primeAiopsFeaturesCache(tenantId: string, value: AiopsFeatureConfig): void { + cache.set(tenantId, { value, at: Date.now() }); +} diff --git a/apps/web-ui/lib/agent/memory-nodes.ts b/apps/web-ui/lib/agent/memory-nodes.ts index 1f2d422f..24e1ecaf 100644 --- a/apps/web-ui/lib/agent/memory-nodes.ts +++ b/apps/web-ui/lib/agent/memory-nodes.ts @@ -104,7 +104,7 @@ Return only the relevant memories.` // ── Learned operating rules — distance-gated, no LLM filter ───────── let proceduresSection = ""; - if (proceduralMemoryEnabled()) { + if (proceduralMemoryEnabled(tenantId)) { try { const rules = await getMemoryService().recall({ tenantId, userId, query, kinds: ["PROCEDURAL"], limit: PROCEDURE_RECALL_LIMIT, @@ -129,7 +129,7 @@ Return only the relevant memories.` // ── Episodic few-shot replay — distance-gated, no LLM filter ──────── let episodesSection = ""; - if (episodicMemoryEnabled()) { + if (episodicMemoryEnabled(tenantId)) { try { const eps = await getMemoryService().recall({ tenantId, userId, query, kinds: ["EPISODIC"], limit: EPISODE_RECALL_LIMIT, @@ -209,7 +209,7 @@ export function createMemorySaveNode(deps: MemoryNodeDeps) { Examples: how a scaling issue was resolved, successful deployment patterns - Error resolutions → namespace: ["errors", ""] Examples: how an OOM was fixed, what caused a timeout, permission error workarounds -` + (proceduralMemoryEnabled() ? `- Operating rules → add "kind": "PROCEDURAL", namespace: ["procedures", ""] +` + (proceduralMemoryEnabled(tenantId) ? `- Operating rules → add "kind": "PROCEDURAL", namespace: ["procedures", ""] A rule for HOW the agent should behave in this environment, learned from this run. Extract a rule ONLY from a correction, a failure the run recovered from, or an explicit user preference about behavior. Shape: { "kind": "PROCEDURAL", "namespace": ["procedures", "aws-cli"], "key": "paginate-list-calls", "value": { "instruction": "Always paginate list/describe calls", "trigger": "any AWS CLI list operation", "evidence": "run truncated results and missed the target resource", "confidence": "high" } } @@ -274,7 +274,7 @@ Extract memories to save.` return { memoryStats: { phase: 'save', savedFacts: 0, savedRules: 0, episodeCaptured: false } }; } - if (reconcileEnabled()) { + if (reconcileEnabled(tenantId)) { console.log(`🧠 [MEMORY SAVE] Reconciling ${toSave.length} extracted facts...`); const threadId = runtimeConfig?.configurable?.thread_id as string | undefined; const summary = await reconcileMemories({ @@ -321,7 +321,7 @@ Extract memories to save.` // ── Episodic capture — independent of fact extraction; never blocks END ── const { plan, toolResults, errors, reflection, isComplete, iterationCount } = state; const threadIdForEpisode = runtimeConfig?.configurable?.thread_id as string | undefined; - const shouldCapture = episodicMemoryEnabled() && !!threadIdForEpisode && toolResults.length > 0; + const shouldCapture = episodicMemoryEnabled(tenantId) && !!threadIdForEpisode && toolResults.length > 0; if (shouldCapture) { await captureEpisode({ tenantId, userId, threadId: threadIdForEpisode, @@ -331,7 +331,7 @@ Extract memories to save.` } // Autonomous skill synthesis — matured domains become/refresh enabled system skills (full Hermes). - if (proceduralMemoryEnabled()) { + if (proceduralMemoryEnabled(tenantId)) { await synthesizeDomainSkills({ tenantId, threadId: threadIdForEpisode, distillerModel: reflectorModel }); } diff --git a/apps/web-ui/lib/agent/memory/episode.test.ts b/apps/web-ui/lib/agent/memory/episode.test.ts index 07d4621c..7e902791 100644 --- a/apps/web-ui/lib/agent/memory/episode.test.ts +++ b/apps/web-ui/lib/agent/memory/episode.test.ts @@ -35,13 +35,13 @@ beforeEach(() => { mockSvc.remember.mockResolvedValue('ep-row-id'); vi.mocked(getMemoryService).mockReturnValue(mockSvc as any); }); -afterEach(() => { delete process.env.EPISODIC_MEMORY_ENABLED; }); +import { DEFAULT_FEATURES, primeAiopsFeaturesCache } from '../aiops-features'; describe('episodicMemoryEnabled', () => { - it('defaults true; false/0 disable', () => { + it('defaults true; tenant setting false disables', () => { expect(episodicMemoryEnabled()).toBe(true); - process.env.EPISODIC_MEMORY_ENABLED = 'false'; - expect(episodicMemoryEnabled()).toBe(false); + primeAiopsFeaturesCache('t-epi-off', { ...DEFAULT_FEATURES, episodicMemoryEnabled: false }); + expect(episodicMemoryEnabled('t-epi-off')).toBe(false); }); }); @@ -76,9 +76,10 @@ describe('captureEpisode', () => { }); it('flag off → short-circuits before invoking the distiller', async () => { - process.env.EPISODIC_MEMORY_ENABLED = 'false'; + primeAiopsFeaturesCache('t1', { ...DEFAULT_FEATURES, episodicMemoryEnabled: false }); const distiller = distillerReturning(JSON.stringify(goodEpisode)); const saved = await captureEpisode(baseParams({ distillerModel: distiller }) as any); + primeAiopsFeaturesCache('t1', { ...DEFAULT_FEATURES }); expect(saved).toBe(false); expect(distiller.invoke).not.toHaveBeenCalled(); }); diff --git a/apps/web-ui/lib/agent/memory/episode.ts b/apps/web-ui/lib/agent/memory/episode.ts index 0bddef78..a3505a40 100644 --- a/apps/web-ui/lib/agent/memory/episode.ts +++ b/apps/web-ui/lib/agent/memory/episode.ts @@ -16,15 +16,16 @@ import { contentToText } from '../agent-shared'; import { getMemoryService } from './memory-service'; import { compressToolOutput } from './working-memory'; import type { EpisodicValue } from './types'; +import { getAiopsFeatures } from '../aiops-features'; export const EPISODE_RECALL_LIMIT = 2; // Looser than reconcile's 0.55 — an ANALOGOUS past experience is useful even // when not near-identical. Initial guess — tune from recall logs. export const EPISODE_DISTANCE_THRESHOLD = 0.65; -export function episodicMemoryEnabled(): boolean { - const v = process.env.EPISODIC_MEMORY_ENABLED?.toLowerCase(); - return !(v === 'false' || v === '0'); +export function episodicMemoryEnabled(tenantId?: string): boolean { + // Tenant setting (AI Ops console -> settings), default on. No env dependency. + return getAiopsFeatures(tenantId).episodicMemoryEnabled; } export interface CaptureEpisodeParams { @@ -57,7 +58,7 @@ If the run was routine or unremarkable (simple lookups, trivial queries), return ); export async function captureEpisode(p: CaptureEpisodeParams): Promise { - if (!episodicMemoryEnabled()) return false; + if (!episodicMemoryEnabled(p.tenantId)) return false; try { const toolSummary = p.toolResults .map((t) => `- ${t.toolName} [${t.isError ? 'ERROR' : 'OK'}]: ${compressToolOutput(t.output, 400)}`) diff --git a/apps/web-ui/lib/agent/memory/procedural.test.ts b/apps/web-ui/lib/agent/memory/procedural.test.ts index 9c30699a..9cf8035c 100644 --- a/apps/web-ui/lib/agent/memory/procedural.test.ts +++ b/apps/web-ui/lib/agent/memory/procedural.test.ts @@ -4,13 +4,13 @@ import { PROCEDURE_RECALL_LIMIT, PROCEDURE_DISTANCE_THRESHOLD, } from './procedural'; -afterEach(() => { delete process.env.PROCEDURAL_MEMORY_ENABLED; }); +import { DEFAULT_FEATURES, primeAiopsFeaturesCache } from '../aiops-features'; describe('proceduralMemoryEnabled', () => { - it('defaults true; false/0 disable', () => { + it('defaults true; tenant setting false disables', () => { expect(proceduralMemoryEnabled()).toBe(true); - process.env.PROCEDURAL_MEMORY_ENABLED = 'false'; - expect(proceduralMemoryEnabled()).toBe(false); + primeAiopsFeaturesCache('t-proc-off', { ...DEFAULT_FEATURES, proceduralMemoryEnabled: false }); + expect(proceduralMemoryEnabled('t-proc-off')).toBe(false); }); }); diff --git a/apps/web-ui/lib/agent/memory/procedural.ts b/apps/web-ui/lib/agent/memory/procedural.ts index 09b1c1f4..a8c85280 100644 --- a/apps/web-ui/lib/agent/memory/procedural.ts +++ b/apps/web-ui/lib/agent/memory/procedural.ts @@ -7,14 +7,15 @@ */ import type { ProceduralValue } from './types'; +import { getAiopsFeatures } from '../aiops-features'; export const PROCEDURE_RECALL_LIMIT = 3; // Shared value with EPISODE_DISTANCE_THRESHOLD — one knob until logs say otherwise. export const PROCEDURE_DISTANCE_THRESHOLD = 0.65; -export function proceduralMemoryEnabled(): boolean { - const v = process.env.PROCEDURAL_MEMORY_ENABLED?.toLowerCase(); - return !(v === 'false' || v === '0'); +export function proceduralMemoryEnabled(tenantId?: string): boolean { + // Tenant setting (AI Ops console -> settings), default on. No env dependency. + return getAiopsFeatures(tenantId).proceduralMemoryEnabled; } function isNonEmptyString(v: unknown): v is string { diff --git a/apps/web-ui/lib/agent/memory/reconcile.test.ts b/apps/web-ui/lib/agent/memory/reconcile.test.ts index 502f3257..ad53e4ae 100644 --- a/apps/web-ui/lib/agent/memory/reconcile.test.ts +++ b/apps/web-ui/lib/agent/memory/reconcile.test.ts @@ -35,13 +35,13 @@ beforeEach(() => { mockSvc.reinforce.mockResolvedValue(undefined); vi.mocked(getMemoryService).mockReturnValue(mockSvc as any); }); -afterEach(() => { delete process.env.MEMORY_RECONCILE_ENABLED; }); +import { DEFAULT_FEATURES, primeAiopsFeaturesCache } from '../aiops-features'; describe('reconcileEnabled', () => { - it('defaults true; false/0 disable', () => { + it('defaults true; tenant setting false disables', () => { expect(reconcileEnabled()).toBe(true); - process.env.MEMORY_RECONCILE_ENABLED = 'false'; - expect(reconcileEnabled()).toBe(false); + primeAiopsFeaturesCache('t-rec-off', { ...DEFAULT_FEATURES, memoryReconcileEnabled: false }); + expect(reconcileEnabled('t-rec-off')).toBe(false); }); }); diff --git a/apps/web-ui/lib/agent/memory/reconcile.ts b/apps/web-ui/lib/agent/memory/reconcile.ts index ae503646..44c485dd 100644 --- a/apps/web-ui/lib/agent/memory/reconcile.ts +++ b/apps/web-ui/lib/agent/memory/reconcile.ts @@ -12,15 +12,16 @@ import type { BaseChatModel } from '@langchain/core/language_models/chat_models' import { contentToText } from '../agent-shared'; import { getMemoryService } from './memory-service'; import type { ExtractedFact, MemoryHit, ReconcileDecision, ReconcileSummary } from './types'; +import { getAiopsFeatures } from '../aiops-features'; export const RECONCILE_TOP_K = 5; // Cosine distance (0 = identical). Neighbors farther than this are treated as // unrelated and the fact goes straight to ADD. Initial guess — tune from logs. export const RECONCILE_DISTANCE_THRESHOLD = 0.55; -export function reconcileEnabled(): boolean { - const v = process.env.MEMORY_RECONCILE_ENABLED?.toLowerCase(); - return !(v === 'false' || v === '0'); +export function reconcileEnabled(tenantId?: string): boolean { + // Tenant setting (AI Ops console -> settings), default on. No env dependency. + return getAiopsFeatures(tenantId).memoryReconcileEnabled; } interface FactWithNeighbors { diff --git a/apps/web-ui/lib/agent/memory/skill-synthesis.test.ts b/apps/web-ui/lib/agent/memory/skill-synthesis.test.ts index 59b27665..1466cbf8 100644 --- a/apps/web-ui/lib/agent/memory/skill-synthesis.test.ts +++ b/apps/web-ui/lib/agent/memory/skill-synthesis.test.ts @@ -10,6 +10,7 @@ import { getMemoryService } from './memory-service'; import { synthesizeDomainSkills, autoSkillCreationEnabled, autoSkillMaturityThreshold, skillSynthesisMinRules, } from './skill-synthesis'; +import { DEFAULT_FEATURES, primeAiopsFeaturesCache } from '../aiops-features'; const mockRepo = { getBySlug: vi.fn(), create: vi.fn(), update: vi.fn() }; const mockSvc = { update: vi.fn() }; @@ -57,12 +58,12 @@ afterEach(() => { }); describe('flags', () => { - it('defaults + env overrides', () => { + it('defaults + tenant-setting overrides', () => { expect(autoSkillCreationEnabled()).toBe(true); expect(autoSkillMaturityThreshold()).toBe(3); expect(skillSynthesisMinRules()).toBe(3); - process.env.SKILL_SYNTHESIS_MIN_RULES = '5'; - expect(skillSynthesisMinRules()).toBe(5); + primeAiopsFeaturesCache('t-syn', { ...DEFAULT_FEATURES, skillSynthesisMinRules: 5 }); + expect(skillSynthesisMinRules('t-syn')).toBe(5); }); }); @@ -147,8 +148,9 @@ describe('synthesizeDomainSkills', () => { }); it('flag off → 0 without any query', async () => { - process.env.AUTO_SKILL_CREATION_ENABLED = 'false'; + primeAiopsFeaturesCache('t1', { ...DEFAULT_FEATURES, autoSkillCreationEnabled: false }); const n = await synthesizeDomainSkills({ ...base, distillerModel: distillerReturning(goodDistill) }); + primeAiopsFeaturesCache('t1', { ...DEFAULT_FEATURES }); expect(n).toBe(0); expect(mockQueryRaw).not.toHaveBeenCalled(); }); diff --git a/apps/web-ui/lib/agent/memory/skill-synthesis.ts b/apps/web-ui/lib/agent/memory/skill-synthesis.ts index 1ece9918..b0a0d115 100644 --- a/apps/web-ui/lib/agent/memory/skill-synthesis.ts +++ b/apps/web-ui/lib/agent/memory/skill-synthesis.ts @@ -16,20 +16,19 @@ import { contentToText } from '../agent-shared'; import { getPrismaClient } from '@/lib/db/pg-config'; import { getSkillRepository } from '@/lib/db/repository-factory'; import { getMemoryService } from './memory-service'; +import { getAiopsFeatures } from '../aiops-features'; -export function autoSkillCreationEnabled(): boolean { - const v = process.env.AUTO_SKILL_CREATION_ENABLED?.toLowerCase(); - return !(v === 'false' || v === '0'); +export function autoSkillCreationEnabled(tenantId?: string): boolean { + // Tenant setting (AI Ops console -> settings), default on. No env dependency. + return getAiopsFeatures(tenantId).autoSkillCreationEnabled; } -export function autoSkillMaturityThreshold(): number { - const n = Number(process.env.AUTO_SKILL_MATURITY_THRESHOLD); - return Number.isFinite(n) && n > 0 ? n : 3; +export function autoSkillMaturityThreshold(tenantId?: string): number { + return getAiopsFeatures(tenantId).autoSkillMaturityThreshold; } -export function skillSynthesisMinRules(): number { - const n = Number(process.env.SKILL_SYNTHESIS_MIN_RULES); - return Number.isFinite(n) && n > 0 ? n : 3; +export function skillSynthesisMinRules(tenantId?: string): number { + return getAiopsFeatures(tenantId).skillSynthesisMinRules; } const MAX_EPISODES = 3; @@ -62,11 +61,11 @@ export async function synthesizeDomainSkills(params: { threadId?: string; distillerModel: BaseChatModel; }): Promise { - if (!autoSkillCreationEnabled()) return 0; + if (!autoSkillCreationEnabled(params.tenantId)) return 0; try { const prisma = getPrismaClient(); - const threshold = autoSkillMaturityThreshold(); - const minRules = skillSynthesisMinRules(); + const threshold = autoSkillMaturityThreshold(params.tenantId); + const minRules = skillSynthesisMinRules(params.tenantId); // 1. Best candidate domain (tenant-bound; bare `procedures` namespaces excluded). const candidates = await prisma.$queryRaw>` diff --git a/apps/web-ui/lib/agent/memory/working-memory.test.ts b/apps/web-ui/lib/agent/memory/working-memory.test.ts index ba6f05e7..95751bc8 100644 --- a/apps/web-ui/lib/agent/memory/working-memory.test.ts +++ b/apps/web-ui/lib/agent/memory/working-memory.test.ts @@ -1,12 +1,12 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { workingMemoryEnabled, tokenBudget, keepRecent } from './working-memory'; +import { DEFAULT_FEATURES, primeAiopsFeaturesCache } from '../aiops-features'; describe('working-memory config', () => { const saved = { ...process.env }; afterEach(() => { process.env = { ...saved }; }); it('defaults: enabled=true, budget=60000, keep=8', () => { - delete process.env.WORKING_MEMORY_ENABLED; delete process.env.WORKING_MEMORY_TOKEN_BUDGET; delete process.env.WORKING_MEMORY_KEEP_RECENT; expect(workingMemoryEnabled()).toBe(true); @@ -14,9 +14,9 @@ describe('working-memory config', () => { expect(keepRecent()).toBe(8); }); - it('WORKING_MEMORY_ENABLED=false disables', () => { - process.env.WORKING_MEMORY_ENABLED = 'false'; - expect(workingMemoryEnabled()).toBe(false); + it('tenant setting false disables', () => { + primeAiopsFeaturesCache('t-wm-off', { ...DEFAULT_FEATURES, workingMemoryEnabled: false }); + expect(workingMemoryEnabled('t-wm-off')).toBe(false); }); it('reads numeric overrides', () => { @@ -141,7 +141,7 @@ describe('prepareContext', () => { afterEach(() => { delete process.env.WORKING_MEMORY_ENABLED; delete process.env.WORKING_MEMORY_TOKEN_BUDGET; }); it('disabled → falls back to getRecentMessages(fallbackWindow), no WM section, no LLM call', async () => { - process.env.WORKING_MEMORY_ENABLED = 'false'; + primeAiopsFeaturesCache('t-wm-flow', { ...DEFAULT_FEATURES, workingMemoryEnabled: false }); // Alternate roles so getRecentMessages passes the window through unchanged // (adjacent same-role messages make it inject synthetic "Acknowledged." AIMessages, // which would push the returned length above the fallbackWindow — a getRecentMessages @@ -149,7 +149,7 @@ describe('prepareContext', () => { const msgs = Array.from({ length: 30 }, (_, i) => i % 2 === 0 ? new HumanMessage(`m${i}`) : new AIMessage(`m${i}`), ); - const res = await prepareContext(baseState(msgs), { reflectorModel: fakeReflector }, 20); + const res = await prepareContext(baseState(msgs), { reflectorModel: fakeReflector, tenantId: 't-wm-flow' }, 20); expect(res.workingMemorySection).toBe(''); expect(res.stateUpdate).toEqual({}); // Disabled path returns EXACTLY the legacy getRecentMessages(fallbackWindow) window diff --git a/apps/web-ui/lib/agent/memory/working-memory.ts b/apps/web-ui/lib/agent/memory/working-memory.ts index 3456528a..1ff9ddf9 100644 --- a/apps/web-ui/lib/agent/memory/working-memory.ts +++ b/apps/web-ui/lib/agent/memory/working-memory.ts @@ -5,13 +5,14 @@ import { getRecentMessages, contentToText } from '../agent-shared'; import type { ReflectionState } from '../agent-shared'; import type { Scratchpad, WorkingMemory } from './types'; import { getMemoryService } from './memory-service'; +import { getAiopsFeatures } from '../aiops-features'; // Working-memory configuration. Read process.env directly (not the typed `env` // object) so Vitest can mutate values per-test — env.ts skips validation under // NODE_ENV==='test' but caches at import time. -export function workingMemoryEnabled(): boolean { - const v = process.env.WORKING_MEMORY_ENABLED?.toLowerCase(); - return !(v === 'false' || v === '0'); +export function workingMemoryEnabled(tenantId?: string): boolean { + // Tenant setting (AI Ops console -> settings), default on. No env dependency. + return getAiopsFeatures(tenantId).workingMemoryEnabled; } // Legacy default when neither an env override nor a model-derived budget is available. @@ -179,7 +180,7 @@ export async function prepareContext( const { messages } = state; // Disabled → identical to legacy behavior. - if (!workingMemoryEnabled()) { + if (!workingMemoryEnabled(deps.tenantId)) { return { windowMessages: getRecentMessages(messages, fallbackWindow), workingMemorySection: '', diff --git a/apps/web-ui/lib/agent/subagent-budget.test.ts b/apps/web-ui/lib/agent/subagent-budget.test.ts index 9d057960..b39f0863 100644 --- a/apps/web-ui/lib/agent/subagent-budget.test.ts +++ b/apps/web-ui/lib/agent/subagent-budget.test.ts @@ -23,9 +23,11 @@ afterEach(() => { }); describe('platformSubagentsEnabled', () => { - it('is false unless SUBAGENTS_ENABLED is exactly "true"', () => { + // The feature is UI-driven: no env var is required to turn it on. The env + // var survives only as an emergency deployment-wide OFF. + it('is true by default and false only when SUBAGENTS_ENABLED is exactly "false"', () => { delete process.env.SUBAGENTS_ENABLED; - expect(platformSubagentsEnabled()).toBe(false); + expect(platformSubagentsEnabled()).toBe(true); process.env.SUBAGENTS_ENABLED = 'false'; expect(platformSubagentsEnabled()).toBe(false); process.env.SUBAGENTS_ENABLED = 'true'; diff --git a/apps/web-ui/lib/agent/subagent-budget.ts b/apps/web-ui/lib/agent/subagent-budget.ts index 959e9d96..36712f20 100644 --- a/apps/web-ui/lib/agent/subagent-budget.ts +++ b/apps/web-ui/lib/agent/subagent-budget.ts @@ -11,8 +11,9 @@ * Clamping runs on READ (not only on write) so lowering a ceiling retroactively * binds config rows written while it was higher. * - * SUBAGENTS_ENABLED is an absolute platform kill-switch: tenant config can - * never re-enable the feature when it is off. + * SUBAGENTS_ENABLED=false is an emergency platform kill-switch: tenant config + * can never re-enable the feature while it is set. Unset means enabled — the + * per-tenant UI toggle is the normal control. */ import { TenantConfigService } from '@/lib/tenant-config-service'; @@ -53,9 +54,14 @@ function ceilingFor(key: NumericKey): number { return Math.min(spec.max, Math.round(fromEnv)); } -/** Absolute platform kill-switch. Defaults to OFF. */ +/** + * Emergency platform kill-switch. Sub-agents are UI-driven: each tenant's + * toggle in the AI Ops settings decides, and no env var is required. Setting + * SUBAGENTS_ENABLED=false disables the feature deployment-wide regardless of + * tenant config — an ops brake for a shared-ECS incident, not a launch gate. + */ export function platformSubagentsEnabled(): boolean { - return process.env.SUBAGENTS_ENABLED === 'true'; + return process.env.SUBAGENTS_ENABLED !== 'false'; } export function clampBudget(input: Partial | null): SubagentBudgetConfig { diff --git a/apps/web-ui/lib/agent/triage.test.ts b/apps/web-ui/lib/agent/triage.test.ts index 86724028..c10be2c7 100644 --- a/apps/web-ui/lib/agent/triage.test.ts +++ b/apps/web-ui/lib/agent/triage.test.ts @@ -6,6 +6,10 @@ vi.mock('@/lib/skill-service', () => ({ })); vi.mock('./model-factory', () => ({ createAgentModels: vi.fn() })); vi.mock('./auto-skill-select', () => ({ autoSelectSkill: vi.fn() })); +// triageChatMessage resolves the tenant's AI Ops feature settings fresh on every +// call; stub the store so tests control the chat-triage toggle. +vi.mock('@/lib/tenant-config-service', () => ({ TenantConfigService: { getConfig: vi.fn() } })); +import { TenantConfigService } from '@/lib/tenant-config-service'; import { getSkillSummaries, getSkillById } from '@/lib/skill-service'; import { createAgentModels } from './model-factory'; @@ -21,13 +25,13 @@ beforeEach(() => { vi.mocked(getSkillSummaries).mockResolvedValue(CATALOG); vi.mocked(getSkillById).mockResolvedValue({ id: 'cost-analyser', name: 'Cost Analyser', description: 'analyses spend', tier: 'read-only' } as any); }); -afterEach(() => { delete process.env.CHAT_TRIAGE_ENABLED; }); +import { DEFAULT_FEATURES, primeAiopsFeaturesCache } from './aiops-features'; describe('chatTriageEnabled', () => { - it('defaults true; false disables', () => { + it('defaults true; tenant setting false disables', () => { expect(chatTriageEnabled()).toBe(true); - process.env.CHAT_TRIAGE_ENABLED = 'false'; - expect(chatTriageEnabled()).toBe(false); + primeAiopsFeaturesCache('t-triage-off', { ...DEFAULT_FEATURES, chatTriageEnabled: false }); + expect(chatTriageEnabled('t-triage-off')).toBe(false); }); }); @@ -97,7 +101,7 @@ describe('triageChatMessage', () => { }); it('kill-switch: falls back to legacy autoSelectSkill with route task', async () => { - process.env.CHAT_TRIAGE_ENABLED = 'false'; + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ chatTriageEnabled: false } as never); vi.mocked(autoSelectSkill).mockResolvedValue({ slug: 'cost-analyser', reasoning: 'legacy' }); const res = await triageChatMessage(base); expect(res).toEqual({ route: 'task', skillId: 'cost-analyser', reasoning: 'legacy' }); @@ -105,7 +109,7 @@ describe('triageChatMessage', () => { }); it('kill-switch + preselected skill: pure task fallback, no LLM calls', async () => { - process.env.CHAT_TRIAGE_ENABLED = 'false'; + vi.mocked(TenantConfigService.getConfig).mockResolvedValue({ chatTriageEnabled: false } as never); const res = await triageChatMessage({ ...base, skillAlreadySelected: true }); expect(res.route).toBe('task'); expect(vi.mocked(autoSelectSkill)).not.toHaveBeenCalled(); diff --git a/apps/web-ui/lib/agent/triage.ts b/apps/web-ui/lib/agent/triage.ts index b4b9d630..424b6291 100644 --- a/apps/web-ui/lib/agent/triage.ts +++ b/apps/web-ui/lib/agent/triage.ts @@ -13,8 +13,9 @@ * * Fail-open: any error, parse failure, or disabled flag routes to 'task' * (a wasted planner pass is cheaper than a wrong "I can't do that" reply). - * Kill-switch: CHAT_TRIAGE_ENABLED=false reverts routing to always-'task' - * while keeping skill auto-selection (via the legacy autoSelectSkill path). + * Tenant toggle (AI Ops console settings → 'Smart routing'): off reverts + * routing to always-'task' while keeping skill auto-selection (via the legacy + * autoSelectSkill path). No environment variable involved. */ import { SystemMessage, HumanMessage } from '@langchain/core/messages'; @@ -22,10 +23,11 @@ import { contentToText, type ResolvedModelConfig } from './agent-shared'; import { createAgentModels } from './model-factory'; import { getSkillSummaries, getSkillById } from '@/lib/skill-service'; import { autoSelectSkill } from './auto-skill-select'; +import { getAiopsFeatures, resolveAiopsFeatures } from './aiops-features'; -export function chatTriageEnabled(): boolean { - const v = process.env.CHAT_TRIAGE_ENABLED?.toLowerCase(); - return !(v === 'false' || v === '0'); +export function chatTriageEnabled(tenantId?: string): boolean { + // Tenant setting (AI Ops console → settings), default on. No env dependency. + return getAiopsFeatures(tenantId).chatTriageEnabled; } export interface TriageResult { @@ -64,9 +66,11 @@ export async function triageChatMessage(params: { /** True when the user manually pinned a skill — skill matching is skipped. */ skillAlreadySelected?: boolean; }): Promise { - // Kill-switch: preserve pre-triage behavior exactly (always workflow, - // legacy standalone skill auto-selection). - if (!chatTriageEnabled()) { + // Tenant toggle: preserve pre-triage behavior exactly (always workflow, + // legacy standalone skill auto-selection). Resolved fresh — this runs once + // per message and must honor a just-saved settings change. + const features = await resolveAiopsFeatures(params.tenantId).catch(() => getAiopsFeatures(params.tenantId)); + if (!features.chatTriageEnabled) { if (params.skillAlreadySelected) return TASK_FALLBACK; const auto = await autoSelectSkill({ tenantId: params.tenantId, message: params.message, model: params.model }); return { route: 'task', skillId: auto?.slug ?? null, reasoning: auto?.reasoning ?? 'triage disabled' }; @@ -80,7 +84,7 @@ export async function triageChatMessage(params: { .catch(() => null); const skillInstruction = catalog - ? `\n${catalog}\n\nFor "task" routes, also pick the single most relevant skill from the catalog above — or null when none clearly matches its description. For "direct" routes, skillId is always null.` + ? `\n${catalog}\n\nFor "task" routes, also pick the single BEST-MATCHING skill from the catalog above. Selecting a skill is the DEFAULT: match on the task's domain (cost, EC2, Jira, security, …), not on exact wording. Return null ONLY when nothing in the catalog is even loosely related to the request. For "direct" routes, skillId is always null.` : `\nskillId is always null.`; const sys = new SystemMessage( diff --git a/apps/web-ui/lib/queries/aiops-settings.ts b/apps/web-ui/lib/queries/aiops-settings.ts index b095bfce..41efa43d 100644 --- a/apps/web-ui/lib/queries/aiops-settings.ts +++ b/apps/web-ui/lib/queries/aiops-settings.ts @@ -17,11 +17,25 @@ export interface SubagentBudget { export interface SubagentBound { min: number; max: number; default: number } +export interface AiopsFeatureConfig { + chatTriageEnabled: boolean; + workingMemoryEnabled: boolean; + episodicMemoryEnabled: boolean; + proceduralMemoryEnabled: boolean; + memoryReconcileEnabled: boolean; + autoSkillCreationEnabled: boolean; + autoSkillMaturityThreshold: number; + skillSynthesisMinRules: number; + maxIterations: number; +} + export interface AiopsSubagentSettings { budget: SubagentBudget; bounds: Record, SubagentBound>; /** False when SUBAGENTS_ENABLED is off for the whole deployment. */ platformEnabled: boolean; + features: AiopsFeatureConfig; + featureBounds: Record<'autoSkillMaturityThreshold' | 'skillSynthesisMinRules' | 'maxIterations', SubagentBound>; } export function useAiopsSubagentSettings() { @@ -56,3 +70,22 @@ export function useSaveAiopsSubagentSettings() { onSuccess: () => qc.invalidateQueries({ queryKey: queryKeys.aiopsSettings.all }), }); } + +export function useSaveAiopsFeatureSettings() { + const qc = useQueryClient(); + return useMutation({ + mutationFn: async (features: AiopsFeatureConfig): Promise => { + const res = await fetch('/api/settings/aiops', { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ features }), + }); + const json = await res.json().catch(() => ({})); + if (!res.ok || !json.success) { + throw new Error(json.error || 'Failed to save AI Ops settings'); + } + return json.data.features as AiopsFeatureConfig; + }, + onSuccess: () => qc.invalidateQueries({ queryKey: queryKeys.aiopsSettings.all }), + }); +} From 75758b6646c9a99529a3391541827a24a08c8484 Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 2 Aug 2026 22:27:56 +0530 Subject: [PATCH 41/42] fix(chat): sub-agent stream corruption, console lockdown, and run-rail observability Root cause of the 'text-delta for missing text part' stream kills: sub-agent model calls run inside ToolNode, so their on_chat_model_* events drove the route's single-threaded text-part state machine and concurrent fan-outs corrupted part ids (also leaking narration into the transcript). Sub-agent model calls are now tagged and filtered like guard runs, with their usage still billed. Plus: stale running cards reconcile from agent_subagent_runs after stream death; cards get task subtitles, live action lines, elapsed timers and a budget denominator; the composer locks every input except Stop while streaming; the guard's deterministic allowlist covers the platform's own read-only tools so its LLM risk batch fires only for real mutations; the run's active skill (incl. triage auto-selection) is surfaced as a data-skill part in the rail; settings open from the console header menu instead of a navbar entry; AI SDK protocol errors get recoverable copy. Co-Authored-By: Claude Fable 5 --- .../api/chat/__tests__/stream-parts.test.ts | 54 ++++++++++- apps/web-ui/app/api/chat/route.ts | 39 +++++++- apps/web-ui/app/api/chat/stream-parts.ts | 33 +++++++ .../agent/chat/__tests__/run-rail.test.tsx | 60 +++++++++++- .../agent/chat/__tests__/run-state.test.ts | 30 ++++++ .../web-ui/components/agent/chat/run-rail.tsx | 95 +++++++++++++++---- .../web-ui/components/agent/chat/run-state.ts | 22 ++++- .../components/agent/chat/subagent-card.tsx | 59 +++++++++++- .../__tests__/transcript-header.test.tsx | 18 ++++ .../components/agent/workspace/composer.tsx | 33 ++++--- .../agent/workspace/events/tool-row.tsx | 3 +- .../agent/workspace/session-view.tsx | 21 ++++ .../agent/workspace/transcript-header.tsx | 3 +- .../agent-chat/use-chat-session-helpers.ts | 12 +++ apps/web-ui/lib/agent/dispatch-agent-tool.ts | 11 ++- apps/web-ui/lib/agent/subagent.test.ts | 44 +++++++++ apps/web-ui/lib/agent/subagent.ts | 29 +++++- apps/web-ui/lib/agent/tool-classifier.ts | 13 +++ apps/web-ui/lib/nav-config.ts | 2 + 19 files changed, 525 insertions(+), 56 deletions(-) diff --git a/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts b/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts index fddbeba3..3013cde9 100644 --- a/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts +++ b/apps/web-ui/app/api/chat/__tests__/stream-parts.test.ts @@ -1,7 +1,8 @@ import { describe, it, expect } from 'vitest'; import { buildMemoryPart, humanizeReflection, humanizePlanning, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from '@/app/api/chat/stream-parts'; import { buildUsagePart } from '../stream-parts'; -import { buildSubagentPart, buildHeartbeatChunks } from '../stream-parts'; +import { buildSubagentPart, buildHeartbeatChunks, isSubagentModelEvent } from '../stream-parts'; +import { SUBAGENT_MODEL_TAG } from '@/lib/agent/subagent'; const WM_JSON = JSON.stringify({ summary: 'User asked to connect Jira; site resource retrieved.', @@ -317,3 +318,54 @@ describe('buildHeartbeatChunks', () => { expect(JSON.stringify(chunks[0])).not.toContain('transcript'); }); }); + +// Sub-agent model calls run inside ToolNode, so their on_chat_model_* callback +// events flow through the SAME streamEvents pipeline as the orchestrator's own +// model runs. The route's text-part machinery is a single-threaded state machine; +// N concurrent sub-agent runs driving it interleave part ids, the client receives +// a text-delta for a part that never started, and the AI SDK kills the stream. +// isSubagentModelEvent is the filter that keeps them out. +describe('isSubagentModelEvent', () => { + it('matches chat-model events tagged by the sub-agent runtime', () => { + for (const name of ['on_chat_model_start', 'on_chat_model_stream', 'on_chat_model_end']) { + expect(isSubagentModelEvent({ event: name, tags: ['x', SUBAGENT_MODEL_TAG] })).toBe(true); + } + }); + + it('ignores untagged chat-model events (the orchestrator itself)', () => { + expect(isSubagentModelEvent({ event: 'on_chat_model_stream', tags: ['seq:step:2'] })).toBe(false); + expect(isSubagentModelEvent({ event: 'on_chat_model_stream' })).toBe(false); + }); + + it('ignores non-chat-model events even when tagged (tool events must still flow)', () => { + expect(isSubagentModelEvent({ event: 'on_tool_start', tags: [SUBAGENT_MODEL_TAG] })).toBe(false); + expect(isSubagentModelEvent({ event: 'on_chain_stream', tags: [SUBAGENT_MODEL_TAG] })).toBe(false); + }); + + it('tolerates malformed events', () => { + expect(isSubagentModelEvent({})).toBe(false); + expect(isSubagentModelEvent({ event: undefined, tags: undefined })).toBe(false); + }); +}); + +describe('buildSubagentPart live-detail fields', () => { + const base = { + id: 'sa-1', role: 'EC2 auditor', task: 'audit account 1', + toolCount: 3, tokensIn: 900, tokensOut: 120, + }; + + it('stamps a ts so the client can derive startedAt from the first part', () => { + const before = Date.now(); + const part = buildSubagentPart({ ...base, status: 'running' }); + const ts = (part.data as { ts: number }).ts; + expect(ts).toBeGreaterThanOrEqual(before); + expect(ts).toBeLessThanOrEqual(Date.now()); + }); + + it('passes lastTool through for the live action line, omitting it when absent', () => { + const withTool = buildSubagentPart({ ...base, status: 'running', lastTool: 'aws_read (+2 more)' }); + expect((withTool.data as { lastTool?: string }).lastTool).toBe('aws_read (+2 more)'); + const without = buildSubagentPart({ ...base, status: 'running' }); + expect('lastTool' in (without.data as object)).toBe(false); + }); +}); diff --git a/apps/web-ui/app/api/chat/route.ts b/apps/web-ui/app/api/chat/route.ts index f4da8582..f360d505 100644 --- a/apps/web-ui/app/api/chat/route.ts +++ b/apps/web-ui/app/api/chat/route.ts @@ -9,7 +9,7 @@ import { triageChatMessage } from '@/lib/agent/triage'; import { respondDirect } from './direct-chat'; import { resolveKnowledgeBaseIds } from '@/lib/agent/auto-kb-select'; import { acquireThreadLock, releaseThreadLock as releaseThreadLockDb } from '@/lib/agent/thread-lock'; -import { buildPlanPart, buildPhasePart, buildInterruptParts, buildMemoryPart, buildUsagePart, buildSubagentPart, buildHeartbeatChunks, humanizeReflection, humanizePlanning, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from './stream-parts'; +import { buildPlanPart, buildPhasePart, buildInterruptParts, buildMemoryPart, buildUsagePart, buildSubagentPart, buildHeartbeatChunks, buildActiveSkillPart, humanizeReflection, humanizePlanning, isSubagentModelEvent, isWorkingMemoryPayload, stripWorkingMemoryPrelude } from './stream-parts'; import { resolveResumedToolCallId, type ResumedPendingCall } from './resume-tool-id'; import type { PlanStep } from '@/lib/agent/agent-shared'; import { logRunSummary } from '@/lib/agent/run-timings'; @@ -576,7 +576,8 @@ export async function POST(req: Request) { resumedPendingCalls, syntheticDecisionResults, liveSubagents, - subagentSinkRef + subagentSinkRef, + effectiveSkill ? { slug: effectiveSkill, source: selectedSkill ? 'user' as const : 'auto' as const } : null ) }); } else { @@ -743,7 +744,8 @@ function processStream( resumedPendingCalls: ResumedPendingCall[] = [], syntheticDecisionResults: Array<{ toolCallId: string; toolName: string; args: Record; output: string }> = [], liveSubagents: Map = new Map(), - subagentSinkRef: { sink: ((chunk: UIMessageChunk) => void) | null } = { sink: null } + subagentSinkRef: { sink: ((chunk: UIMessageChunk) => void) | null } = { sink: null }, + activeSkill: { slug: string; source: 'user' | 'auto' } | null = null ): ReadableStream { return new ReadableStream({ async start(controller) { @@ -869,6 +871,12 @@ function processStream( try { if (!safeEnqueue({ type: 'start' })) return; + // Surface the run's ACTIVE skill (incl. triage auto-selection) so the + // rail can show it — a loaded skill must never be invisible. + if (activeSkill) { + safeEnqueue(buildActiveSkillPart(activeSkill.slug, activeSkill.source) as UIMessageChunk); + } + // When resuming from HITL approval, emit text content first // The tool-input events will be emitted in on_tool_start for each tool if (isResumedFromApproval && approvedToolId) { @@ -907,10 +915,33 @@ function processStream( event.event.startsWith("on_chat_model_") && event.metadata?.langgraph_node === 'guard'; - if (isGuardModelRun) { + // Sub-agent model calls run inside ToolNode and MUST NOT drive + // this single-threaded text-part state machine: N concurrent + // sub-agents interleaving through currentPartId/streamStarted + // emit deltas for parts that never started, which kills the + // client stream — and their narration would leak into the + // transcript. Their progress reaches the client only as + // data-subagent parts (emitSubagent). + const isSubagentModelRun = isSubagentModelEvent(event as { event?: string; tags?: string[] }); + + if (isGuardModelRun || isSubagentModelRun) { // Skip start/stream/end entirely. streamStarted stays false // (the previous run's on_chat_model_end already closed its // part), so nothing downstream references a dangling part id. + // + // Sub-agent tokens are still BILLED tokens: keep them in the + // run's usage counter even though the text/phase machinery + // never sees these runs. + if (isSubagentModelRun && event.event === 'on_chat_model_end') { + const subUsage = (event.data?.output as { usage_metadata?: { input_tokens?: number; output_tokens?: number } } | undefined)?.usage_metadata; + const subIn = Number(subUsage?.input_tokens) || 0; + const subOut = Number(subUsage?.output_tokens) || 0; + if (subIn || subOut) { + runUsage.input += subIn; + runUsage.output += subOut; + safeEnqueue(buildUsagePart(subIn, subOut) as UIMessageChunk); + } + } } else if (event.event === "on_chat_model_start") { const node = event.metadata?.langgraph_node; diff --git a/apps/web-ui/app/api/chat/stream-parts.ts b/apps/web-ui/app/api/chat/stream-parts.ts index 710dea3e..a3d47843 100644 --- a/apps/web-ui/app/api/chat/stream-parts.ts +++ b/apps/web-ui/app/api/chat/stream-parts.ts @@ -3,9 +3,29 @@ import { pendingToolCallsOf } from '@/lib/agent/guard'; import { extractJsonObject } from '@/lib/agent/llm-json'; import type { GuardVerdict, PlanStep, ReflectionState } from '@/lib/agent/agent-shared'; import type { SubagentEvent } from '@/lib/agent/dispatch-agent-tool'; +import { SUBAGENT_MODEL_TAG } from '@/lib/agent/subagent'; export interface DataPart { type: `data-${string}`; id?: string; data: unknown } +/** + * True for chat-model callback events that belong to a SUB-AGENT's model calls, + * which run inside ToolNode and therefore surface through the orchestrator's + * own streamEvents pipeline. The route's text-part state machine is single- + * threaded (one currentPartId/streamStarted/runMode); letting N concurrent + * sub-agent model runs drive it interleaves their events, emits text-deltas for + * part ids whose text-start belonged to a different run, and the AI SDK client + * kills the stream ("text-delta for missing text part"). It also leaks + * sub-agent narration into the transcript, which the design forbids. The route + * must skip these events entirely — sub-agent progress reaches the client only + * as data-subagent parts. + */ +export function isSubagentModelEvent(event: { event?: string; tags?: string[] }): boolean { + return typeof event.event === 'string' + && event.event.startsWith('on_chat_model_') + && Array.isArray(event.tags) + && event.tags.includes(SUBAGENT_MODEL_TAG); +} + export function buildPlanPart(threadId: string, steps: PlanStep[], updatedBy: string): DataPart { return { type: 'data-plan', id: `plan-${threadId}`, data: { steps, updatedBy } }; } @@ -14,6 +34,15 @@ export function buildPhasePart(phase: string, node: string): DataPart { return { type: 'data-phase', data: { phase, node, ts: Date.now() } }; } +/** + * Announces the ACTIVE skill for this run — including one auto-selected by + * triage, which the composer's own chip cannot know about. Without this part a + * correctly-loaded skill is invisible in the UI and reads as a silent failure. + */ +export function buildActiveSkillPart(slug: string, source: 'user' | 'auto'): DataPart { + return { type: 'data-skill', data: { slug, source } }; +} + export function buildUsagePart(input: number, output: number): DataPart { return { type: 'data-usage', data: { input, output } }; } @@ -50,6 +79,10 @@ export function buildSubagentPart(event: SubagentEvent): DataPart { toolCount: event.toolCount, tokensIn: event.tokensIn, tokensOut: event.tokensOut, + // Server clock at emit time. The client keeps the FIRST ts it sees per + // sub-agent as startedAt and ticks elapsed time from its own clock. + ts: Date.now(), + ...(event.lastTool ? { lastTool: event.lastTool } : {}), ...(event.summary ? { summary: event.summary } : {}), }, }; diff --git a/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx b/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx index fa5fe14b..92777878 100644 --- a/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx +++ b/apps/web-ui/components/agent/chat/__tests__/run-rail.test.tsx @@ -135,7 +135,7 @@ describe('RunRail', () => { } render() - expect(screen.getByText('Sub-agents (1 running)')).toBeTruthy() + expect(screen.getByText(/^Sub-agents \(1 running/)).toBeTruthy() expect(screen.getByText('EC2 scanner')).toBeTruthy() expect(screen.getByText('RDS scanner')).toBeTruthy() }) @@ -165,12 +165,13 @@ describe('RunRail', () => { await waitFor(() => expect(fetchMock).toHaveBeenCalledWith('/api/chat/subagents/thread-1')) expect(await screen.findByText('EC2 scanner')).toBeTruthy() - expect(screen.getByText('Sub-agents (0 running)')).toBeTruthy() + expect(screen.getByText(/^Sub-agents \(0 running/)).toBeTruthy() }) it('does not fetch persisted runs for a thread that never dispatched', () => { render() - expect(fetchMock).not.toHaveBeenCalled() + // The settings fetch (Context line) is allowed; the sub-agent store fetch is not. + expect(fetchMock).not.toHaveBeenCalledWith('/api/chat/subagents/thread-1') }) it('prefers live sub-agent state over a persisted fetch while streaming', () => { @@ -181,8 +182,8 @@ describe('RunRail', () => { } render() - expect(screen.getByText('Sub-agents (1 running)')).toBeTruthy() - expect(fetchMock).not.toHaveBeenCalled() + expect(screen.getByText(/^Sub-agents \(1 running/)).toBeTruthy() + expect(fetchMock).not.toHaveBeenCalledWith('/api/chat/subagents/thread-1') }) it('still shows the pending-approval status row', () => { @@ -197,3 +198,52 @@ describe('RunRail', () => { expect(screen.getByText('Awaiting approval')).toBeTruthy() }) }) + +// REGRESSION GUARD: a stream that dies mid-fan-out leaves live cards frozen at +// "running" — the terminal event was persisted server-side but never reached the +// client. Once streaming stops, the rail must reconcile those cards from +// agent_subagent_runs instead of spinning forever on a dead run. +describe('RunRail sub-agent reconciliation after stream death', () => { + const liveRunning = { + id: 'sa-9', role: 'EC2 auditor — 123456789012', task: 'audit', status: 'running' as const, + toolCount: 5, tokensIn: 100, tokensOut: 50, + } + + it('overrides a stale running card with the persisted terminal status', async () => { + fetchMock.mockResolvedValue({ + ok: true, + json: async () => ({ + success: true, + data: [{ + subagentId: 'sa-9', role: 'EC2 auditor — 123456789012', task: 'audit', status: 'failed', + toolCount: 17, tokensIn: 20000, tokensOut: 6800, summary: 'Model invocation was aborted.', transcript: null, + }], + }), + }) + render( + , + ) + await waitFor(() => { + expect(screen.getByText(/0 running/)).toBeTruthy() + }) + // The abort is presented as a cancellation, not a sub-agent mistake. + expect(screen.getByText(/Cancelled — the run was stopped/)).toBeTruthy() + }) + + it('leaves live running cards alone while the stream is still open', () => { + render( + , + ) + expect(screen.getByText(/1 running/)).toBeTruthy() + }) +}) diff --git a/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts b/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts index 8ce93896..4b89766d 100644 --- a/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts +++ b/apps/web-ui/components/agent/chat/__tests__/run-state.test.ts @@ -356,3 +356,33 @@ describe('deriveRunState — usedSubagents', () => { expect(state.usedSubagents).toBe(false); }); }); + +// startedAt must be STICKY (first part's ts survives later updates — otherwise the +// elapsed timer restarts on every progress tick) and lastTool must track the +// latest part so the card's action line stays current. +describe('deriveRunState sub-agent live details', () => { + const partFor = (data: Record) => ({ + role: 'assistant', + parts: [{ type: 'data-subagent', data }], + }); + + it('keeps the first ts as startedAt across updates and tracks lastTool', () => { + const base = { id: 'sa-1', role: 'r', task: 't', status: 'running', toolCount: 0, tokensIn: 0, tokensOut: 0 }; + const state = deriveRunState([ + partFor({ ...base, ts: 1000 }), + partFor({ ...base, ts: 2000, toolCount: 2, lastTool: 'aws_read' }), + partFor({ ...base, ts: 3000, toolCount: 4, lastTool: 'get_aws_credentials' }), + ] as any, new Set()); + expect(state.subagents).toHaveLength(1); + expect(state.subagents[0].startedAt).toBe(1000); + expect(state.subagents[0].lastTool).toBe('get_aws_credentials'); + expect(state.subagents[0].toolCount).toBe(4); + }); + + it('leaves startedAt undefined when no part carried a ts (old streams)', () => { + const state = deriveRunState([ + partFor({ id: 'sa-1', role: 'r', task: 't', status: 'running', toolCount: 0, tokensIn: 0, tokensOut: 0 }), + ] as any, new Set()); + expect(state.subagents[0].startedAt).toBeUndefined(); + }); +}); diff --git a/apps/web-ui/components/agent/chat/run-rail.tsx b/apps/web-ui/components/agent/chat/run-rail.tsx index 1879437a..df27e2ed 100644 --- a/apps/web-ui/components/agent/chat/run-rail.tsx +++ b/apps/web-ui/components/agent/chat/run-rail.tsx @@ -7,8 +7,13 @@ import { Plan, PlanContent, PlanStep } from "@/components/ai-elements/plan"; import { Spinner } from "@/components/ui/spinner"; import { SubagentCard } from "./subagent-card"; import { useSubagentRuns } from "@/lib/queries/subagents"; +import { useAiopsSubagentSettings } from "@/lib/queries/aiops-settings"; import type { RunState, SubagentState } from "./run-state"; +function formatTokensCompact(n: number): string { + return n >= 1000 ? `${Math.round(n / 1000)}k` : String(n); +} + const PHASE_LABELS: Record = { planning: "Planning", execution: "Executing", reflection: "Reflecting", revision: "Revising", final: "Finalizing", text: "Idle", @@ -60,23 +65,59 @@ export function RunRail({ const { plan, currentPhase, pendingApproval, pendingClarifications } = runState; // A reloaded thread has no data-subagent parts (they are not persisted), so the - // cards are rebuilt from agent_subagent_runs. Only fetched when the thread is - // known to have fanned out — a persisted dispatch_agent tool call — and only when - // there is no live state to prefer. - const needsPersistedSubagents = runState.usedSubagents && runState.subagents.length === 0; + // cards are rebuilt from agent_subagent_runs. Fetched in two cases: + // - reload: the thread fanned out (persisted dispatch_agent call) but there is + // no live card state at all; + // - stream death: the stream ended while a card still says "running". The + // terminal event was persisted by the server but never reached this client, + // so without reconciliation the spinner spins forever on a dead run. + const hasStaleRunning = !isStreaming && runState.subagents.some((s) => s.status === "running"); + const needsPersistedSubagents = runState.usedSubagents && (runState.subagents.length === 0 || hasStaleRunning); const { data: persistedRuns } = useSubagentRuns(threadId, needsPersistedSubagents); - const subagents: SubagentState[] = needsPersistedSubagents - ? (persistedRuns ?? []).map((run) => ({ - id: run.subagentId, - role: run.role, - task: run.task, - status: run.status === "done" || run.status === "failed" ? run.status : "running", - toolCount: run.toolCount, - tokensIn: run.tokensIn, - tokensOut: run.tokensOut, - ...(run.summary ? { summary: run.summary } : {}), - })) - : runState.subagents; + + let subagents: SubagentState[]; + if (runState.subagents.length === 0) { + subagents = needsPersistedSubagents + ? (persistedRuns ?? []).map((run) => ({ + id: run.subagentId, + role: run.role, + task: run.task, + status: run.status === "done" || run.status === "failed" ? run.status : "running", + toolCount: run.toolCount, + tokensIn: run.tokensIn, + tokensOut: run.tokensOut, + ...(run.summary ? { summary: run.summary } : {}), + })) + : runState.subagents; + } else if (hasStaleRunning && persistedRuns) { + // Merge: a persisted TERMINAL status wins over a live "running" card; a card + // the store still calls running stays as-is (the store row may simply lag). + const byId = new Map(persistedRuns.map((r) => [r.subagentId, r])); + subagents = runState.subagents.map((live) => { + if (live.status !== "running") return live; + const stored = byId.get(live.id); + if (!stored || (stored.status !== "done" && stored.status !== "failed")) return live; + return { + ...live, + status: stored.status, + toolCount: stored.toolCount, + tokensIn: stored.tokensIn, + tokensOut: stored.tokensOut, + ...(stored.summary ? { summary: stored.summary } : {}), + }; + }); + } else { + subagents = runState.subagents; + } + + // Budget context for the section header ("109k/400k tokens") and the + // enabled-but-idle signal in Context. Only fetched once the rail exists; + // failures degrade to plain counters. + const { data: aiopsSettings } = useAiopsSubagentSettings(); + const subagentBudget = aiopsSettings?.budget; + const subagentTokensUsed = subagents.reduce((sum, s) => sum + s.tokensIn + s.tokensOut, 0); + const subagentTokenBudget = subagentBudget?.enabled ? subagentBudget.maxSubagentTokensPerRun : null; + const subagentsEnabled = (aiopsSettings?.platformEnabled ?? false) && (subagentBudget?.enabled ?? false); const done = plan.filter((s) => s.status === "completed").length; const mutativePending = pendingApproval?.tools.some((t) => t.guard?.isMutative) ?? false; @@ -173,7 +214,7 @@ export function RunRail({ {subagents.length > 0 && ( s.status === "running").length} running)`} + title={`Sub-agents (${subagents.filter((s) => s.status === "running").length} running · ${formatTokensCompact(subagentTokensUsed)}${subagentTokenBudget ? `/${formatTokensCompact(subagentTokenBudget)}` : ""} tokens)`} >
{subagents.map((subagent) => ( @@ -227,7 +268,25 @@ export function RunRail({ {context.modelLabel || "Default model"} - {context.skillName &&
  • Skill: {context.skillName}
    • } + {context.skillName ? ( +
  • Skill: {context.skillName}
    • + ) : runState.activeSkill ? ( + // Triage auto-selected this skill — without this line a loaded + // skill is indistinguishable from no skill at all. +
  • + Skill: {runState.activeSkill.slug}{runState.activeSkill.source === 'auto' ? ' (auto)' : ''} +
    • + ) : null} + {/* Distinguishes "enabled but the model chose not to delegate" from + "feature off" — without this the two states are identical in the UI. */} + {subagentBudget && ( +
  • + + {subagentsEnabled + ? `Sub-agents: on (max ${subagentBudget.maxConcurrentSubagents} parallel)` + : "Sub-agents: off"} +
    • + )}
  • {context.kbLabel}{context.toolCount != null ? ` · ${context.toolCount} tools` : ""}
    • diff --git a/apps/web-ui/components/agent/chat/run-state.ts b/apps/web-ui/components/agent/chat/run-state.ts index ff8447b1..cf82d65b 100644 --- a/apps/web-ui/components/agent/chat/run-state.ts +++ b/apps/web-ui/components/agent/chat/run-state.ts @@ -21,6 +21,10 @@ export interface SubagentState { toolCount: number; tokensIn: number; tokensOut: number; + /** Tool (batch) currently executing — shown as the card's live action line. */ + lastTool?: string; + /** Server ts of the FIRST part seen for this sub-agent; drives the elapsed timer. */ + startedAt?: number; summary?: string; } @@ -40,6 +44,8 @@ export interface RunState { /** Sub-agents dispatched in this thread, in first-seen order. Later parts for * the same id replace the earlier entry in place. */ subagents: SubagentState[]; + /** Skill active for the latest run — user-pinned or triage auto-selected. */ + activeSkill?: { slug: string; source: 'user' | 'auto' } | null; /** True when this thread fanned out at all — set by a live data-subagent part * OR by a persisted dispatch_agent tool call. data-subagent parts are not * persisted, so after a reload this is the ONLY signal that the rail should @@ -65,6 +71,7 @@ export function deriveRunState( let tokenOut = 0; const subagents = new Map(); let usedSubagents = false; + let activeSkill: { slug: string; source: 'user' | 'auto' } | null = null; // Tool calls with an output-bearing tool part ANYWHERE in the thread. An // executed / rejected / answered tool can never be pending again, no matter // what a stale data-approval or data-clarification part claims (defense @@ -122,6 +129,12 @@ export function deriveRunState( } break; } + case 'data-skill': { + if (part.data?.slug) { + activeSkill = { slug: String(part.data.slug), source: part.data.source === 'user' ? 'user' : 'auto' }; + } + break; + } case 'data-usage': { tokenIn += Number(part.data?.input) || 0; tokenOut += Number(part.data?.output) || 0; @@ -133,7 +146,11 @@ export function deriveRunState( const id = String(part.data?.id ?? ''); if (!id) break; // Map.set on an existing key preserves insertion order, so the - // card stays put as it updates from running → done. + // card stays put as it updates from running → done. startedAt is + // sticky: the first part's ts survives every later update, so the + // elapsed timer never restarts mid-run. + const prior = subagents.get(id); + const ts = Number(part.data?.ts) || undefined; subagents.set(id, { id, role: String(part.data?.role ?? ''), @@ -143,6 +160,8 @@ export function deriveRunState( toolCount: Number(part.data?.toolCount) || 0, tokensIn: Number(part.data?.tokensIn) || 0, tokensOut: Number(part.data?.tokensOut) || 0, + ...(part.data?.lastTool ? { lastTool: String(part.data.lastTool) } : {}), + ...(prior?.startedAt ?? ts ? { startedAt: prior?.startedAt ?? ts } : {}), ...(part.data?.summary ? { summary: String(part.data.summary) } : {}), }); break; @@ -170,6 +189,7 @@ export function deriveRunState( tokenUsage: { input: tokenIn, output: tokenOut }, subagents: Array.from(subagents.values()), usedSubagents, + activeSkill, }; } diff --git a/apps/web-ui/components/agent/chat/subagent-card.tsx b/apps/web-ui/components/agent/chat/subagent-card.tsx index b9819459..7878b599 100644 --- a/apps/web-ui/components/agent/chat/subagent-card.tsx +++ b/apps/web-ui/components/agent/chat/subagent-card.tsx @@ -1,6 +1,6 @@ 'use client'; -import { useState } from 'react'; +import { useEffect, useState } from 'react'; import { Bot, CheckCircle2, ChevronDown, ChevronRight, Loader2, XCircle } from 'lucide-react'; import { cn } from '@/lib/utils'; @@ -11,12 +11,33 @@ function formatTokens(n: number): string { return n >= 1000 ? `${(n / 1000).toFixed(1)}k` : String(n); } +function formatElapsed(ms: number): string { + const s = Math.max(0, Math.floor(ms / 1000)); + return s >= 60 ? `${Math.floor(s / 60)}m ${s % 60}s` : `${s}s`; +} + +/** An abort surfaced as a failure — the run was stopped; the sub-agent did nothing wrong. */ +function isAbortSummary(summary: string | undefined): boolean { + return !!summary && /abort/i.test(summary); +} + const STATUS_ICON = { running: , done: , failed: , } as const; +/** Ticks once a second while `active`, so the elapsed label counts up live. */ +function useNowWhile(active: boolean): number { + const [now, setNow] = useState(() => Date.now()); + useEffect(() => { + if (!active) return; + const interval = setInterval(() => setNow(Date.now()), 1000); + return () => clearInterval(interval); + }, [active]); + return now; +} + export function SubagentCard({ subagent, threadId, @@ -27,7 +48,10 @@ export function SubagentCard({ threadId?: string; }) { const [expanded, setExpanded] = useState(false); - const canExpand = subagent.status !== 'running' && !!subagent.summary; + const isRunning = subagent.status === 'running'; + const canExpand = !isRunning && !!subagent.summary; + const now = useNowWhile(isRunning && !!subagent.startedAt); + const cancelled = subagent.status === 'failed' && isAbortSummary(subagent.summary); // Loaded only on expand — the transcript is never streamed (three interleaved // sub-agent narrations are unreadable) and never persisted in the message @@ -54,19 +78,46 @@ export function SubagentCard({ {STATUS_ICON[subagent.status]} - {subagent.role} + + {subagent.role} + {/* Task excerpt — disambiguates cards whose role strings collide. */} + + {subagent.task} + + {subagent.toolCount} tools · {formatTokens(subagent.tokensIn + subagent.tokensOut)} tokens + {isRunning && subagent.startedAt ? ` · ${formatElapsed(now - subagent.startedAt)}` : ''} + {/* Live action line — what makes the spinner trustworthy. */} + {isRunning && subagent.lastTool && ( +
    + › {subagent.lastTool} +
    + )} + + {cancelled && !expanded && ( +
    + Cancelled — the run was stopped before this sub-agent finished. +
    + )} + {expanded && subagent.summary && (

    Task

    {subagent.task}

    Findings

    -

    {subagent.summary}

    +

    + {cancelled + ? 'Cancelled — the run was stopped before this sub-agent finished.' + : subagent.summary} +

    {transcript.length > 0 && (
    diff --git a/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx b/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx index aa2e7e25..82dc1a9d 100644 --- a/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx +++ b/apps/web-ui/components/agent/workspace/__tests__/transcript-header.test.tsx @@ -242,6 +242,24 @@ describe('TranscriptHeader', () => { expect(onMenuAction).toHaveBeenCalledWith('export-md') }) + // Sub-agent (and future AI Ops) configuration opens from the console itself — + // there is deliberately no sidebar nav entry for it. + it('offers an "AI Ops settings" menu item that fires the settings action', async () => { + const onMenuAction = vi.fn() + render( + + ) + fireEvent.keyDown(screen.getByRole('button', { name: /more actions/i }), { key: 'Enter' }) + fireEvent.click(await screen.findByText('AI Ops settings')) + expect(onMenuAction).toHaveBeenCalledWith('settings') + }) + it('offers transcript-PDF and report-PDF export menu items', async () => { const onMenuAction = vi.fn() render( diff --git a/apps/web-ui/components/agent/workspace/composer.tsx b/apps/web-ui/components/agent/workspace/composer.tsx index 4e38f6d4..f85e23c1 100644 --- a/apps/web-ui/components/agent/workspace/composer.tsx +++ b/apps/web-ui/components/agent/workspace/composer.tsx @@ -86,6 +86,9 @@ export function Composer({ const isEmpty = !value.trim(); const canSubmit = !isEmpty && !disabled && !isStreaming; + // While a run is streaming, EVERY input is locked except the Stop button — + // an accidental Enter or picker change mid-run must not start/alter a chat. + const locked = disabled || isStreaming; const handleKeyDown = (e: React.KeyboardEvent) => { // Enter submits; Shift+Enter inserts a newline; an in-progress IME @@ -104,9 +107,9 @@ export function Composer({
    {/* Chips row — account/model/skill, removable */}
    - - - + + +
    {/* Textarea */} @@ -117,7 +120,7 @@ export function Composer({ onChange={(e) => onChange(e.target.value)} onKeyDown={handleKeyDown} placeholder="Ask the agent to plan, execute, reflect, and revise..." - disabled={disabled} + disabled={locked} data-testid="composer-input" rows={1} maxLength={MAX_CHARS} @@ -134,7 +137,7 @@ export function Composer({ type="button" variant="ghost" size="icon" - disabled={disabled} + disabled={locked} className="h-7 w-7 rounded-md text-muted-foreground hover:text-foreground" aria-label="Add knowledge, tools, or attachments" > @@ -142,13 +145,13 @@ export function Composer({ - - + +

    Attach images

    - +
    @@ -175,7 +178,7 @@ export function Composer({ type="button" variant="ghost" size="sm" - disabled={disabled} + disabled={locked} className="h-7 gap-1 px-2 text-xs text-muted-foreground hover:text-foreground" > @@ -185,7 +188,7 @@ export function Composer({
    - @@ -210,7 +213,7 @@ export function Composer({ id="composer-auto-approve" checked={autoApprove} onCheckedChange={onAutoApproveChange} - disabled={disabled} + disabled={locked} />
    @@ -221,7 +224,7 @@ export function Composer({ id="composer-show-tools" checked={showTools} onCheckedChange={onShowToolsChange} - disabled={disabled} + disabled={locked} />
    @@ -236,7 +239,7 @@ export function Composer({ id="composer-auto-skills" checked={autoLoadSkills} onCheckedChange={onAutoLoadSkillsChange} - disabled={disabled} + disabled={locked} />
    @@ -245,7 +248,7 @@ export function Composer({ @@ -256,7 +259,7 @@ export function Composer({ size="icon" data-testid="composer-enhance-button" onClick={onEnhance} - disabled={isEmpty || disabled || isStreaming || isEnhancing} + disabled={isEmpty || locked || isEnhancing} aria-label="Enhance prompt with AI" title="Enhance prompt with AI" className={cn( diff --git a/apps/web-ui/components/agent/workspace/events/tool-row.tsx b/apps/web-ui/components/agent/workspace/events/tool-row.tsx index 6e8f5a1b..95c81457 100644 --- a/apps/web-ui/components/agent/workspace/events/tool-row.tsx +++ b/apps/web-ui/components/agent/workspace/events/tool-row.tsx @@ -1,7 +1,7 @@ "use client" import * as React from "react" -import { ChevronDown, Terminal, FileText, FilePen, BookOpen, Wrench, Check, X } from "lucide-react" +import { ChevronDown, Terminal, FileText, FilePen, BookOpen, Bot, Wrench, Check, X } from "lucide-react" import type { TranscriptEvent } from "@/lib/agent-chat/events" import type { ToolGroup } from "@/lib/agent-chat/group-events" import { Collapsible, CollapsibleTrigger, CollapsibleContent } from "@/components/ui/collapsible" @@ -15,6 +15,7 @@ const TOOL_ICONS: Record> = read_file: FileText, write_file: FilePen, search_knowledge_base: BookOpen, + dispatch_agent: Bot, } function toolIcon(toolName: string): React.ComponentType<{ className?: string }> { diff --git a/apps/web-ui/components/agent/workspace/session-view.tsx b/apps/web-ui/components/agent/workspace/session-view.tsx index 0435cdfc..75c36e5a 100644 --- a/apps/web-ui/components/agent/workspace/session-view.tsx +++ b/apps/web-ui/components/agent/workspace/session-view.tsx @@ -20,6 +20,9 @@ import { useDistillSkill } from "@/lib/queries/skills"; import { useDistillScheduledTask } from "@/lib/queries/agent-ops-scheduled-tasks"; import { SCHEDULED_TASK_PREFILL_KEY } from "@/components/agent-ops/scheduled-task-dialog"; import { SkillFormDialog } from "@/components/skills/skill-form-dialog"; +import { AiopsSubagentSettings } from "@/components/settings/aiops-subagent-settings"; +import { AiopsFeatureSettings } from "@/components/settings/aiops-feature-settings"; +import { Dialog, DialogContent, DialogTitle } from "@/components/ui/dialog"; import type { FileAttachment } from "@/components/agent/file-upload"; import { Spinner } from "@/components/ui/spinner"; import { Transcript } from "./transcript"; @@ -202,6 +205,9 @@ export function SessionView({ threadId, ownerUserId, active, onStatusChange, onT const distillSkill = useDistillSkill(); const distillScheduledTask = useDistillScheduledTask(); const [skillDialogOpen, setSkillDialogOpen] = useState(false); + // AI Ops settings (sub-agent budget etc.) live IN the console — opened from + // the header overflow menu, not a sidebar nav entry. + const [settingsOpen, setSettingsOpen] = useState(false); const [skillDraft, setSkillDraft] = useState< { name: string; description: string; tier: string; content: string } | null >(null); @@ -270,6 +276,9 @@ export function SessionView({ threadId, ownerUserId, active, onStatusChange, onT case "skill": void handleSaveAsSkill(); break; + case "settings": + setSettingsOpen(true); + break; case "clear": handleClear(); break; @@ -380,6 +389,18 @@ export function SessionView({ threadId, ownerUserId, active, onStatusChange, onT initialDraft={skillDraft} sourceRunId={threadId} /> + + {/* AI Ops settings — the console owns its own configuration. */} + + + {/* The panels render their own headings; this title exists for a11y. */} + AI Ops settings +
    + + +
    +     +
    ); } diff --git a/apps/web-ui/components/agent/workspace/transcript-header.tsx b/apps/web-ui/components/agent/workspace/transcript-header.tsx index f8b2c3b0..04177379 100644 --- a/apps/web-ui/components/agent/workspace/transcript-header.tsx +++ b/apps/web-ui/components/agent/workspace/transcript-header.tsx @@ -12,7 +12,7 @@ import { import type { RunState } from "@/components/agent/chat/run-state"; import { formatTokens } from "@/lib/agent-chat/token-usage"; -export type TranscriptMenuAction = "export-report" | "export-md" | "export-pdf" | "copy" | "schedule" | "skill" | "clear"; +export type TranscriptMenuAction = "export-report" | "export-md" | "export-pdf" | "copy" | "schedule" | "skill" | "settings" | "clear"; export interface TranscriptHeaderProps { title: string; @@ -52,6 +52,7 @@ const MENU_ITEMS: Array<{ action: TranscriptMenuAction; label: string }> = [ { action: "copy", label: "Copy" }, { action: "schedule", label: "Convert to scheduled task" }, { action: "skill", label: "Save as skill" }, + { action: "settings", label: "AI Ops settings" }, { action: "clear", label: "Clear" }, ]; diff --git a/apps/web-ui/lib/agent-chat/use-chat-session-helpers.ts b/apps/web-ui/lib/agent-chat/use-chat-session-helpers.ts index 79bce2d0..4db627b1 100644 --- a/apps/web-ui/lib/agent-chat/use-chat-session-helpers.ts +++ b/apps/web-ui/lib/agent-chat/use-chat-session-helpers.ts @@ -4,10 +4,22 @@ // The AI SDK surfaces HTTP failures as Error(message) where message is usually // the raw response body — our API routes return `{"error":"..."}` — sometimes // with surrounding text. Extract the human-readable server error when present. +/** + * AI SDK stream-protocol violations ("text-delta for missing text part", bad + * chunk ordering, …). These are internal invariants, not something the user can + * act on from the raw wording — and the run's progress IS saved server-side, so + * point them at the recovery path instead of the protocol internals. + */ +const STREAM_PROTOCOL_ERROR = + /missing text part|text-(start|delta|end)" chunk|invalid chunk order/i; + export function extractServerErrorMessage(error: unknown): string | null { const message = error instanceof Error ? error.message : typeof error === "string" ? error : ""; if (!message.trim()) return null; + if (STREAM_PROTOCOL_ERROR.test(message)) { + return "The live stream was interrupted by a protocol error. The run's progress is saved — reload the thread to see its current state."; + } const jsonMatch = message.match(/\{[\s\S]*\}/); if (jsonMatch) { try { diff --git a/apps/web-ui/lib/agent/dispatch-agent-tool.ts b/apps/web-ui/lib/agent/dispatch-agent-tool.ts index a143aeec..b14682bf 100644 --- a/apps/web-ui/lib/agent/dispatch-agent-tool.ts +++ b/apps/web-ui/lib/agent/dispatch-agent-tool.ts @@ -25,6 +25,8 @@ export interface SubagentEvent { toolCount: number; tokensIn: number; tokensOut: number; + /** Name of the tool (batch) currently executing — the card's live action line. */ + lastTool?: string; summary?: string; transcript?: SubagentTranscriptEntry[]; } @@ -234,10 +236,15 @@ Do NOT use it for: - work that depends on another sub-agent's output (they cannot see each other) - a single quick lookup you could do in one tool call +SUB-AGENT CAPABILITIES — write the brief for THESE tools, no others: +- Sub-agents have NO shell. execute_command does not exist for them. Never write CLI command lines, pipes, or shell substitutions like $(date ...) into the task — a brief written as shell commands wastes the sub-agent's entire iteration budget reconciling instructions it cannot follow. +- Sub-agents reach AWS only through the structured aws_read tool: they name a service (e.g. ec2, cloudwatch), an operation (e.g. describe-instances, get-metric-statistics), and parameters. Write the brief as investigation GOALS plus which services/operations to use — e.g. "list EC2 instances via ec2 describe-instances, then fetch each instance's 14-day average CPUUtilization via cloudwatch get-metric-statistics". +- Sub-agents call get_aws_credentials themselves; give them the account id and region, not a profile name or bootstrap instructions. + CRITICAL: "task" must be completely self-contained. The sub-agent sees NONE of this conversation — no account ids, no prior findings, no user context unless you write them into the task. A vague brief returns a useless report.`, schema: z.object({ - role: z.string().describe('Short identity for this sub-agent, e.g. "EC2 idle-resource auditor for account 123456789012"'), - task: z.string().describe('Complete standalone brief: what to investigate, which account ids and regions, which tools to prefer, and any constraints. Assume zero shared context.'), + role: z.string().describe('Short identity for this sub-agent. MUST name the specific account id, region, or service it covers so parallel cards are distinguishable — e.g. "EC2 idle auditor — 123456789012 (ap-south-1)". Never reuse the same role string for two dispatches in one turn.'), + task: z.string().describe('Complete standalone brief: what to investigate, which account ids and regions, which aws_read service/operations to use, and any constraints. Goals and operations only — NO shell command lines. Assume zero shared context.'), expectedOutput: z.string().describe('Exactly what the report must contain, e.g. "instance ids with average CPU below 5% over 14 days, with the metric value for each"'), }), }, diff --git a/apps/web-ui/lib/agent/subagent.test.ts b/apps/web-ui/lib/agent/subagent.test.ts index c72fa973..c6c24335 100644 --- a/apps/web-ui/lib/agent/subagent.test.ts +++ b/apps/web-ui/lib/agent/subagent.test.ts @@ -3,6 +3,7 @@ import { isReadOnlyForSubagent, filterReadOnlyTools, runSubagent, + SUBAGENT_MODEL_TAG, SUBAGENT_REPORT_MAX_CHARS, type SubagentSpec, } from './subagent'; @@ -319,3 +320,46 @@ describe('hallucinated tool names', () => { expect(result.transcript.some(e => e.text.includes('REFUSED'))).toBe(true); }); }); + +// The sub-agent's model calls run INSIDE the orchestrator's LangGraph run, so +// their callback events reach the chat route's streamEvents pipeline. The tag is +// the only thing that lets the route filter them out of its single-threaded +// text-part state machine — without it, concurrent sub-agents corrupt part ids +// and the client kills the stream. +describe('sub-agent model call tagging', () => { + it('tags every model invocation with SUBAGENT_MODEL_TAG', async () => { + const model = scriptedModel([ + { content: 'look', tool_calls: [{ id: 'c1', name: 'describe_instances', args: {} }] }, + { content: 'done, findings: none' }, + ]); + await runSubagent(SPEC, { model, tools: [readTool], budget: BUDGET }); + expect(model.invoke.mock.calls.length).toBeGreaterThan(0); + for (const call of model.invoke.mock.calls) { + expect(call[1]).toMatchObject({ tags: [SUBAGENT_MODEL_TAG] }); + } + }); +}); + +describe('sub-agent progress lastTool', () => { + it('announces the tool batch before it runs and the last tool after', async () => { + const model = scriptedModel([ + { + content: 'looking', tool_calls: [ + { id: 'c1', name: 'describe_instances', args: {} }, + { id: 'c2', name: 'describe_instances', args: { next: true } }, + ], + }, + { content: 'done' }, + ]); + const events: Array<{ lastTool?: string }> = []; + await runSubagent(SPEC, { + model, tools: [readTool], budget: BUDGET, + onEvent: (p) => events.push(p), + }); + // Pre-batch event names the batch (first tool + count) so the card shows a + // live action line while the calls are still in flight. + expect(events.some(e => e.lastTool === 'describe_instances (+1 more)')).toBe(true); + // Post-batch event carries the last executed tool. + expect(events.some(e => e.lastTool === 'describe_instances')).toBe(true); + }); +}); diff --git a/apps/web-ui/lib/agent/subagent.ts b/apps/web-ui/lib/agent/subagent.ts index d40af5af..221b54ab 100644 --- a/apps/web-ui/lib/agent/subagent.ts +++ b/apps/web-ui/lib/agent/subagent.ts @@ -65,11 +65,25 @@ interface SubagentToolLike { invoke: (args: Record) => Promise; } +/** + * Tag attached to every model call a sub-agent makes. A sub-agent runs INSIDE + * the orchestrator's LangGraph run (dispatch_agent executes in ToolNode), so its + * on_chat_model_* callback events flow through the same streamEvents pipeline + * the chat route uses to build text parts. That pipeline is a single-threaded + * state machine (one currentPartId / streamStarted / runMode); N concurrent + * sub-agent model runs interleaving through it clobber each other's part ids — + * the client then sees text-deltas for parts that never started and kills the + * stream. The route filters chat-model events carrying this tag (see + * isSubagentModelEvent in app/api/chat/stream-parts.ts), which is also what + * keeps sub-agent narration out of the transcript by design. + */ +export const SUBAGENT_MODEL_TAG = 'nucleus-subagent'; + export interface SubagentDeps { - model: { bindTools?: (tools: unknown[]) => unknown; invoke: (messages: unknown[]) => Promise }; + model: { bindTools?: (tools: unknown[]) => unknown; invoke: (messages: unknown[], config?: Record) => Promise }; tools: SubagentToolLike[]; budget: SubagentBudgetConfig; - onEvent?: (progress: { toolCount: number; tokensIn: number; tokensOut: number }) => void; + onEvent?: (progress: { toolCount: number; tokensIn: number; tokensOut: number; lastTool?: string }) => void; } /** @@ -223,7 +237,7 @@ async function runSubagentLoop( for (let iteration = 0; iteration < budget.subagentMaxIterations; iteration++) { if (control.cancelled) return cancelledResult(); - const response = (await boundModel.invoke(messages)) as { + const response = (await boundModel.invoke(messages, { tags: [SUBAGENT_MODEL_TAG] })) as { content: unknown; tool_calls?: Array<{ id?: string; name: string; args?: Record }>; usage_metadata?: { input_tokens?: number; output_tokens?: number }; @@ -248,6 +262,13 @@ async function runSubagentLoop( messages.push(new AIMessage({ content: text, tool_calls: toolCalls as never })); + // Announce the batch BEFORE it runs: this is what lets the card show a + // live "current tool" line while the calls are still in flight. + deps.onEvent?.({ + toolCount, tokensIn, tokensOut, + lastTool: toolCalls.length === 1 ? toolCalls[0].name : `${toolCalls[0].name} (+${toolCalls.length - 1} more)`, + }); + // Independent calls in one turn run concurrently — the same parallelism // the orchestrator gets from ToolNode. const results = await Promise.all(toolCalls.map(async call => { @@ -312,7 +333,7 @@ async function runSubagentLoop( } control.progress.toolCount = toolCount; - deps.onEvent?.({ toolCount, tokensIn, tokensOut }); + deps.onEvent?.({ toolCount, tokensIn, tokensOut, lastTool: results[results.length - 1]?.call.name }); } return { diff --git a/apps/web-ui/lib/agent/tool-classifier.ts b/apps/web-ui/lib/agent/tool-classifier.ts index 36b7e8d1..5746ba2c 100644 --- a/apps/web-ui/lib/agent/tool-classifier.ts +++ b/apps/web-ui/lib/agent/tool-classifier.ts @@ -32,6 +32,19 @@ const MUTATIVE_PATTERNS = [ const READ_ONLY_ALLOWLIST = new Set([ 'get_aws_credentials', 'list_aws_accounts', + // Platform-internal read-only tools. Without these entries every turn that + // uses one is "unknown-named" and triggers the guard's LLM risk batch — + // pure latency on turns that cannot mutate anything. + 'search_knowledge_base', // vector search over tenant KB chunks + 'load_skill', // loads skill INSTRUCTIONS into context; enabled-only + 'get_file_from_s3', // reads the agent's own S3 scratch area + 'get_right_sizing_recommendations', // reads recommendation rows + 'dispatch_agent', // read-only BY CONSTRUCTION: sub-agents are jailed + // (no shell, aws_read allowlist, mutative refused); + // sub-agent recursion is blocked by the sub-agent + // DENYLIST, which runs before any allowlist. + 'web_search', // outbound read (Tavily) + 'glob', 'grep', 'ls', // filesystem inspection, no writes 'describe_instances', 'describe_services', 'describe_clusters', diff --git a/apps/web-ui/lib/nav-config.ts b/apps/web-ui/lib/nav-config.ts index 546108f1..af440204 100644 --- a/apps/web-ui/lib/nav-config.ts +++ b/apps/web-ui/lib/nav-config.ts @@ -36,6 +36,8 @@ export const navMenus: NavItem[] = [ { title: "Ask", href: "/app/knowledge-base/ask" }, { title: "Skills", href: "/app/skills" }, { title: "MCP Servers", href: "/app/agent-ops/mcp-settings" }, + // Sub-agent settings deliberately have no nav entry: they open from the + // AI Ops console's header menu ("AI Ops settings"), where they are used. // LLM providers power the agents — grouped with Agentic Ops, not Settings. { title: "Providers", href: "/app/settings/providers" }, ], From 19c9c18e50839a3eed636e41c27fe719f2dbf75d Mon Sep 17 00:00:00 2001 From: kartikmanimuthu Date: Sun, 2 Aug 2026 22:28:06 +0530 Subject: [PATCH 42/42] feat(agent): resource-utilization mandate + delegation/skill prompt discipline CORE_PRINCIPLES gains a model-independent mandate applied to every graph: load the matching skill BEFORE skill-covered work, treat recalled memory as data (verify conflicts, never ignore), search the knowledge base before answering 'not found', and pick the purpose-built tool (MCP, aws_read, skills) over improvising or credential-scavenging. The planner learns delegation exists (budget-gated); the executor's batch-vs-delegate triggers become disjoint and dispatch briefs must be aws_read intents, never shell lines; all delegation prompt text disappears when sub-agents are disabled. Skill-load logs now include the injected char count as observable proof of prompt injection; the tenant-tunable iteration cap replaces the hardcoded MAX_ITERATIONS in both agents. Co-Authored-By: Claude Fable 5 --- apps/web-ui/lib/agent/auto-skill-select.ts | 2 +- apps/web-ui/lib/agent/fast-agent.ts | 18 ++++++++---- apps/web-ui/lib/agent/planning-agent.ts | 34 +++++++++++++++------- apps/web-ui/lib/agent/prompt-templates.ts | 6 ++++ apps/web-ui/lib/agent/skill-tool.ts | 2 +- 5 files changed, 45 insertions(+), 17 deletions(-) diff --git a/apps/web-ui/lib/agent/auto-skill-select.ts b/apps/web-ui/lib/agent/auto-skill-select.ts index 4e77a418..03163d3f 100644 --- a/apps/web-ui/lib/agent/auto-skill-select.ts +++ b/apps/web-ui/lib/agent/auto-skill-select.ts @@ -24,7 +24,7 @@ export async function autoSelectSkill(params: { const sys = new SystemMessage( `You select the single most relevant skill for a user request, or none.\n\n${catalog}\n\n` + `Return ONLY a JSON object: {"skillId": "" | null, "reasoning": ""}\n` + - `Rules: pick a skill ONLY when the request clearly matches its description. When in doubt, return null.`, + `Rules: selecting a skill is the DEFAULT. Match on the request's domain (cost, EC2, Jira, security, …), not exact wording — a skill whose description covers the task's subject area is a match. Return null ONLY when nothing in the catalog is even loosely related.`, ); const resp = await reflector.invoke([sys, new HumanMessage(params.message.slice(0, 4000))]); const content = contentToText(resp.content); diff --git a/apps/web-ui/lib/agent/fast-agent.ts b/apps/web-ui/lib/agent/fast-agent.ts index 924d7eb7..dfdc7726 100644 --- a/apps/web-ui/lib/agent/fast-agent.ts +++ b/apps/web-ui/lib/agent/fast-agent.ts @@ -33,9 +33,15 @@ import { createMemoryRecallNode, createMemorySaveNode } from "./memory-nodes"; import { prepareContext, buildWorkingMemorySection } from "./memory/working-memory"; import { createGuardNode } from "./guard"; import { routeAfterGuard } from "./gate-routing"; +import { getAiopsFeatures, resolveAiopsFeatures } from './aiops-features'; // --- FAST GRAPH (Reflection Agent Mode) --- export async function createFastGraph(config: GraphConfig) { + // Tenant-tunable run cap (AI Ops console settings) — same knob as the + // planning agent; MAX_ITERATIONS is only the default. + const maxIterations = config.tenantId + ? (await resolveAiopsFeatures(config.tenantId).catch(() => getAiopsFeatures(config.tenantId))).maxIterations + : MAX_ITERATIONS; const { model: modelConfig, autoApprove, accounts, accountId, accountName, selectedSkill, mcpServerIds, tenantId } = config; const modelId = modelConfig.modelId; const checkpointer = await getCheckpointer(); @@ -45,7 +51,9 @@ export async function createFastGraph(config: GraphConfig) { // prompt and reused by the reflector below — no repeated queries. const skillContent = selectedSkill && tenantId ? (await getSkillContent(tenantId, selectedSkill)) || '' : ''; if (selectedSkill) { - console.log(skillContent ? `[FastAgent] Loaded skill: ${selectedSkill}` : `[FastAgent] No content for skill: ${selectedSkill}`); + // The char count is the observable proof of injection: this exact string is + // concatenated into the system prompt below. (LLM_AUDIT=true dumps it verbatim.) + console.log(skillContent ? `[FastAgent] Loaded skill: ${selectedSkill} (${skillContent.length.toLocaleString()} chars injected into system prompt)` : `[FastAgent] No content for skill: ${selectedSkill}`); } // Skill catalog for progressive disclosure — gated by the console "Auto // skills" toggle (default on). The zero-skill sentinel string must not leak @@ -97,7 +105,7 @@ export async function createFastGraph(config: GraphConfig) { const { messages, iterationCount, memoryContext } = state; console.log(`\n================================================================================`); - console.log(`🚀 [FAST AGENT] Generator Iteration ${iterationCount + 1}/${MAX_ITERATIONS}`); + console.log(`🚀 [FAST AGENT] Generator Iteration ${iterationCount + 1}/${maxIterations}`); console.log(` Model: ${modelId}`); console.log(`================================================================================\n`); @@ -298,14 +306,14 @@ Please narrow the question or ask me to continue from here.`; const hasPendingToolCalls = !!(lastMessage.tool_calls && lastMessage.tool_calls.length > 0); // Hard cap FIRST — prevents unbounded loops when model keeps generating tool_calls - if (iterationCount >= MAX_ITERATIONS) { + if (iterationCount >= maxIterations) { // If the model still wants tools, they will not run — synthesize a final answer // so the user never gets an empty response and we never persist an orphaned tool_use. if (hasPendingToolCalls) { - console.log(`⚠️ Max iterations (${MAX_ITERATIONS}) reached with pending tool calls. Synthesizing final answer.`); + console.log(`⚠️ Max iterations (${maxIterations}) reached with pending tool calls. Synthesizing final answer.`); return "finalize"; } - console.log(`⚠️ Max iterations (${MAX_ITERATIONS}) reached. Stopping.`); + console.log(`⚠️ Max iterations (${maxIterations}) reached. Stopping.`); return "memory_save"; } diff --git a/apps/web-ui/lib/agent/planning-agent.ts b/apps/web-ui/lib/agent/planning-agent.ts index 358b06d7..d26016b7 100644 --- a/apps/web-ui/lib/agent/planning-agent.ts +++ b/apps/web-ui/lib/agent/planning-agent.ts @@ -37,6 +37,7 @@ import { prepareContext, buildWorkingMemorySection, estimateTokens } from "./mem import { createGuardNode } from "./guard"; import { routeAfterGuard } from "./gate-routing"; import { recordNodeTiming } from "./run-timings"; +import { getAiopsFeatures, resolveAiopsFeatures } from "./aiops-features"; import { resolveSubagentBudget } from "./subagent-budget"; import { createRunBudgetLedger, createDispatchAgentTool } from "./dispatch-agent-tool"; import { filterReadOnlyTools } from "./subagent"; @@ -199,7 +200,9 @@ export async function createReflectionGraph(config: GraphConfig) { // Pre-fetch skill content once (tenant-scoped DB lookup). No repeated queries. const skillContent = selectedSkill && tenantId ? (await getSkillContent(tenantId, selectedSkill)) || '' : ''; if (selectedSkill) { - console.log(skillContent ? `[PlanningAgent] Loaded skill: ${selectedSkill}` : `[PlanningAgent] No content for skill: ${selectedSkill}`); + // The char count is the observable proof of injection: this exact string is + // concatenated into the system prompt below. (LLM_AUDIT=true dumps it verbatim.) + console.log(skillContent ? `[PlanningAgent] Loaded skill: ${selectedSkill} (${skillContent.length.toLocaleString()} chars injected into system prompt)` : `[PlanningAgent] No content for skill: ${selectedSkill}`); } // Skill catalog for progressive disclosure — gated by the console "Auto @@ -236,6 +239,12 @@ export async function createReflectionGraph(config: GraphConfig) { ? await resolveSubagentBudget(tenantId) : { enabled: false, maxConcurrentSubagents: 1, maxSubagentsPerRun: 1, maxSubagentTokensPerRun: 0, subagentMaxIterations: 2, subagentTimeoutMs: 30_000 }; + // Tenant-tunable run cap (AI Ops console settings). Resolved once per graph + // build; MAX_ITERATIONS from agent-shared remains only as the default. + const maxIterations = tenantId + ? (await resolveAiopsFeatures(tenantId).catch(() => getAiopsFeatures(tenantId))).maxIterations + : MAX_ITERATIONS; + const dispatchAgentTool = subagentBudget.enabled ? createDispatchAgentTool({ model, @@ -319,7 +328,8 @@ Work through three phases when building the plan: - The plan is INTERNAL COORDINATION. The user sees it in a progress rail — it is never part of the answer, so optimize it for execution, not for presentation. - Match plan depth to the request. A simple lookup or single-command request needs only 1–2 steps (credentials + the query); a conversational or no-tool request gets exactly one step: ["Answer the user's request directly"]. Only genuinely multi-phase investigations warrant 5+ steps. -- Keep the plan SHORT: never more than 7 steps. Merge related read-only queries into a single step (e.g., one step for "inventory EC2 + RDS + EBS across regions", not one step per service per region). The executor issues the individual queries in that step as parallel tool calls, so a merged step costs no more time than a narrow one. +- Keep the plan SHORT: never more than 7 steps. Merge related read-only queries into a single step (e.g., one step for "inventory EC2 + RDS + EBS across regions", not one step per service per region). The executor issues the individual queries in that step as parallel tool calls, so a merged step costs no more time than a narrow one.${subagentBudget.enabled ? ` +- The executor can DELEGATE via dispatch_agent: parallel read-only sub-agents, each investigating one account/region/service in its own context and returning a compressed report. When the task fans out into per-account (or per-region/per-service) investigations that each need multiple steps (e.g. list resources, then fetch metrics per resource), write ONE plan step shaped for delegation — "Dispatch one sub-agent per account to investigate X and report Y" — instead of merging that fan-out into a direct-query step. Delegation keeps the main context small; use it whenever the fan-out exceeds ~3 independent multi-step investigations.` : ''} - Every step except the last must be a concrete tool action. Do NOT create steps for aggregating, analyzing, summarizing, cross-referencing, or "evaluating" data — that thinking happens inside execution, not as plan line items. - Do NOT add a knowledge-base search step unless the user asked for it or the task clearly depends on tenant-specific documented context. - The LAST step is always: compose the final answer to the user's request from the gathered data. @@ -401,7 +411,7 @@ Only return the JSON array, nothing else.`); const { messages, plan, iterationCount, memoryContext } = state; console.log(`\n================================================================================`); - console.log(`⚡ [EXECUTOR] Iteration ${iterationCount + 1}/${MAX_ITERATIONS}`); + console.log(`⚡ [EXECUTOR] Iteration ${iterationCount + 1}/${maxIterations}`); console.log(` Current Step: ${plan.find(s => s.status === 'pending')?.step || 'Executing...'}`); console.log(` Model: ${modelId}`); console.log(`================================================================================\n`); @@ -433,7 +443,7 @@ ${accountContext} ## Execution Discipline -- Execute the current step. When that step covers several INDEPENDENT read-only lookups (different accounts, regions, or services), issue them as MULTIPLE tool calls in this single turn — they run in parallel. One call per turn for independent reads wastes minutes. +- Execute the current step. When that step is a handful of INDEPENDENT single-call lookups (each unit = ONE tool call, e.g. get_aws_credentials for several accounts), issue them as MULTIPLE tool calls in this single turn — they run in parallel. One call per turn for independent reads wastes minutes.${dispatchAgentTool ? ' When each unit instead needs its own multi-step investigation, delegate it — see Delegation below; do not flatten a fan-out of investigations into direct calls.' : ''} - Calls that DEPEND on each other must stay sequential: if call B needs call A's output (an account id, a resource id, a profile name), make call A now and call B on the next turn. - Never batch mutations. Issue at most ONE state-changing call per turn so each is reviewed and approved individually. - Do not skip ahead to later plan steps. @@ -441,12 +451,16 @@ ${accountContext} - If the current step is a simple question or greeting that requires no tools, answer directly and concisely. - If the current step has nothing to execute (empty inventory, prerequisite returned no data), move on silently — NEVER run echo or other no-op commands just to mark a step done, and never write an explanation of why there was nothing to do. -## Delegation +${dispatchAgentTool ? `## Delegation -- When the current step splits into INDEPENDENT investigations — one per account, region, or service — emit several dispatch_agent calls in THIS turn. They run in parallel and each returns a compressed report. +- DELEGATE (dispatch_agent) when the current step splits into INDEPENDENT investigations — one per account, region, or service — where each unit needs SEVERAL tool calls of its own (list resources, then fetch metrics/details per resource). Emit several dispatch_agent calls in THIS turn; they run in parallel and each returns a compressed report. BATCH direct tool calls instead only when each unit is a single call. - Write each sub-agent brief so it stands completely alone: it sees none of this conversation, so spell out account ids, regions, time windows, and what the report must contain. +- Sub-agents have NO shell. Never put CLI command lines, --profile flags, or shell syntax like $(date ...) in a brief. Describe goals plus aws_read service/operations (e.g. "ec2 describe-instances, then cloudwatch get-metric-statistics for CPUUtilization"); the sub-agent obtains its own credentials from the account id you give it. +- Give every sub-agent a DISTINCT role naming its account/region/service — identical role strings make the progress cards indistinguishable. - Do NOT delegate work that changes state — sub-agents are read-only. Do NOT delegate a single quick lookup. Do NOT chain sub-agents; they cannot see each other's results. -- If dispatch_agent replies that the budget is exhausted, simply do that work yourself with your own tools. +- If dispatch_agent replies that the budget is exhausted, simply do that work yourself with your own tools.` : `## Delegation + +- Sub-agents are DISABLED for this organization. There is no dispatch_agent tool — never reference or attempt it. Do all work yourself with your own tools, batching independent read-only calls into a single turn.`} ## Narration Discipline (critical) @@ -582,7 +596,7 @@ The user sees plan progress in a UI rail — your prose must NEVER duplicate it: console.log(`\n================================================================================`); console.log(`🤔 [REFLECTOR] Analyzing execution results`); - console.log(` Iteration: ${iterationCount}/${MAX_ITERATIONS}`); + console.log(` Iteration: ${iterationCount}/${maxIterations}`); console.log(` Model: ${modelId}`); console.log(`================================================================================`); @@ -608,7 +622,7 @@ Original Task: ${taskDescription} Plan Status: ${plan.map((s, i) => `${i + 1}. [${s.status}] ${s.step}`).join('\n')} -Iteration: ${iterationCount}/${MAX_ITERATIONS} +Iteration: ${iterationCount}/${maxIterations} ## Input Notes @@ -744,7 +758,7 @@ ${suggestions !== "None" ? `💡 **Suggestions:** ${suggestions}` : ""} **Task Complete:** ${isComplete ? "✅ Yes" : "❌ No, continuing..."}`; - if (iterationCount >= MAX_ITERATIONS && !isComplete) { + if (iterationCount >= maxIterations && !isComplete) { console.log(`⚠️ Max iterations (${MAX_ITERATIONS}) reached. Forcing completion.`); isComplete = true; } diff --git a/apps/web-ui/lib/agent/prompt-templates.ts b/apps/web-ui/lib/agent/prompt-templates.ts index 29512ea5..ccb2fc79 100644 --- a/apps/web-ui/lib/agent/prompt-templates.ts +++ b/apps/web-ui/lib/agent/prompt-templates.ts @@ -43,6 +43,12 @@ export const CORE_PRINCIPLES = ` 3. **Be specific** — include resource IDs, account names, regions, and numeric values in every response. 4. **Fail forward** — if a command fails, capture the full error, diagnose root cause, and attempt a corrective action. Never silently skip. 5. **Lead with action** — respond with the finding or first action, not a restatement of the question. + +## Resource Utilization Mandate (applies to every step, non-negotiable) +6. **Load the matching skill FIRST.** If a skill catalog is listed in this prompt and any skill's description covers the task — or a phase of it — call load_skill with that id BEFORE doing the work, then follow the loaded instructions. Doing skill-covered work unaided while the matching skill sits unloaded is an error, not a style choice. +7. **Ground every step in recalled memory.** When a memory section (known facts, operating rules, past experience) is present, it is DATA, not decoration: reuse its identifiers, baselines, and prior findings; apply its rules; and when live data contradicts a recalled fact, verify with a live query and state which one is current. Never produce a conclusion that silently ignores a recalled fact that bears on it. +8. **Knowledge base before "not found".** Before answering that information is unavailable, unknown, or undocumented — or guessing at tenant-specific context (naming conventions, runbooks, board/project mappings, architecture) — call search_knowledge_base if it is available. "I couldn't find it" is only a valid answer AFTER the knowledge base has been searched. +9. **Pick the purpose-built tool at every decision.** Survey ALL tools available in this run — MCP integrations, structured AWS tools, skills, knowledge base, memory — and use the one built for the job instead of improvising. If an MCP integration covers the target system (Jira, Slack, GitHub, …), use it; never scavenge the filesystem or environment for credentials, and never declare a system unreachable while its tool sits unused. `; // --------------------------------------------------------------------------- diff --git a/apps/web-ui/lib/agent/skill-tool.ts b/apps/web-ui/lib/agent/skill-tool.ts index 7264133d..4e6d92c9 100644 --- a/apps/web-ui/lib/agent/skill-tool.ts +++ b/apps/web-ui/lib/agent/skill-tool.ts @@ -28,7 +28,7 @@ export function createLoadSkillTool(tenantId: string) { { name: 'load_skill', description: - 'Load the full instructions of a specialized skill by its id. The available skills are listed in your system prompt under "Available Skills". Call this BEFORE starting work that a skill\'s description covers — the loaded instructions define privileges, safety rules, and workflow for that scope. You may load multiple skills over the course of one task as different phases require them. Do not call it for skills already loaded in this conversation.', + 'Load the full instructions of a specialized skill by its id. The available skills are listed in your system prompt under "Available Skills". Calling this is MANDATORY, not optional: whenever a listed skill\'s description covers the work you are about to do (match on domain — cost, EC2, Jira, security, … — not exact wording), load it BEFORE doing that work. The loaded instructions define privileges, safety rules, and workflow for that scope; doing skill-covered work without loading the skill is an error. You may load multiple skills over one task as different phases require them. Do not call it for skills already loaded in this conversation.', schema: z.object({ skill_id: z.string().describe('The skill id (slug) exactly as listed in the Available Skills catalog'), }),