Skip to content

CLAUDE.md

Latest commit

 

History

History
261 lines (218 loc) · 15.1 KB

CLAUDE.md

File metadata and controls

261 lines (218 loc) · 15.1 KB
Raw

CLAUDE.md — AI Agent Instructions for CrisisMode

This file provides context and instructions for AI agents (Claude Code, Copilot, Cursor, etc.) working in this repository.

Project Overview

CrisisMode is an AI crisis recovery framework with a hub-and-spoke architecture. Spokes execute recovery plans close to target systems. The hub provides coordination, analytics, and management. The framework is designed for crisis conditions — when infrastructure is degraded and the cost of wrong actions is highest.

Architecture

Layers

  • Layer 1 (Execution) + Layer 2 (Safety) — run in the spoke
  • Layer 3 (Coordination) + Layer 4 (Enrichment) — run in the hub

Key abstractions

  • RecoveryAgent (src/agent/interface.ts) — the contract every agent implements: assessHealth(), diagnose(), plan(), replan()
  • ExecutionBackend (src/framework/backend.ts) — shared contract for execution backends (executeCommand(), evaluateCheck(), optional listCapabilityProviders())
  • PgBackend / RedisBackend / EtcdBackend / KafkaBackend / K8sBackend / CephBackend / FlinkBackend / DnsBackend / TlsBackend / DiskBackend — agent-specific backend interfaces that extend ExecutionBackend with system-specific diagnosis methods
  • ExecutionEngine (src/framework/engine.ts) — executes plans step-by-step with safety checks
  • GraphEngine (src/framework/graph-engine.ts) — LangGraph-based graph execution engine for complex recovery workflows
  • SymptomRouter (src/framework/symptom-router.ts) — routes symptoms to the appropriate recovery agent
  • ProviderRegistry (src/framework/provider-registry.ts) — resolves which capability providers can handle each step
  • CapabilityRegistry (src/framework/capability-registry.ts) — global registry of standard recovery capabilities (e.g., db.query.read, db.replica.disconnect)
  • OperatorSummary (src/framework/operator-summary.ts) — builds human-readable health and readiness summaries for operators
  • IncidentReport (src/framework/incident-report.ts) — generates structured incident reports from recovery executions
  • NetworkProfile (src/framework/network-profile.ts) — network diagnostics and profiling
  • AI Diagnosis (src/framework/ai-diagnosis-universal.ts) — universal AI-powered diagnosis for any agent via Claude API
  • ForensicRecorder — immutable audit trail for every execution

Agent SDK (packages/agent-sdk/)

  • All public types are defined in @crisismode/agent-sdk and re-exported from src/types/index.ts
  • Zero runtime dependencies — types only
  • Source-of-truth for: RecoveryAgent, ExecutionBackend, AgentManifest, RecoveryPlan, step types, health types
  • The main package depends on @crisismode/agent-sdk via pnpm workspace

Playbook System (src/framework/playbook/)

  • parser.ts — Parses Markdown + YAML frontmatter into ParsedPlaybook objects
  • runtime.ts — Converts ParsedPlaybook to RecoveryPlan (same type the execution engine uses)
  • discovery.ts — Scans ~/.crisismode/playbooks/, ./playbooks/, CRISISMODE_PLAYBOOK_PATH for .md files
  • Playbooks use the same safety infrastructure as code-based agents (no shortcuts)
  • Format spec: specs/foundational/playbook-format.md

Hook System (src/framework/hooks/)

  • types.ts — 9 lifecycle hook points (plan:validate, step:before, step:after, etc.)
  • registry.ts — Priority-ordered, multi-subscriber hook registry with timeout protection
  • builtin.ts — Built-in hooks for logging and summary (priority 0-99, non-removable)
  • Engine integration: optional HookRegistry parameter in LegacyExecutionEngine constructor

Agent Plugin Registry (src/framework/registry/)

  • types.ts — AgentPluginManifest (crisismode-agent.json schema)
  • local.ts — Discovery from ~/.crisismode/agents/, ./agents/, CRISISMODE_AGENT_PATH, node_modules/@crisismode/
  • Manifest spec: specs/foundational/registry-manifest.md

Execution modes

  • dry-run — reads from real systems, logs mutations without executing
  • execute — runs all operations including SQL mutations

Code Conventions

  • TypeScript with strict mode, ESM modules ("type": "module")
  • Module resolution: NodeNext — all imports use .js extensions
  • No default exports — use named exports
  • Async by default — backend interfaces return Promise<T>, engine methods are async
  • Type imports — use import type { ... } for type-only imports

CLI

The crisismode CLI (src/cli/index.ts) provides a unified interface with the following commands:

Command Description
scan Zero-config health scan with scored summary (default when no command given)
diagnose Health check + AI-powered diagnosis (read-only)
recover Full recovery flow with execution planning
status Quick health probe
ask Natural language AI diagnosis
demo Simulator demo mode
init Generate crisismode.yaml configuration
webhook Start webhook receiver for AlertManager
watch Continuous shadow observation
readiness Scale-readiness report (read-only): will this stack break under load?
playbook validate Validate a playbook file
playbook list List discovered playbooks
playbook dry-run Preview compiled recovery plan
agent list List all registered agents
agent info Show agent details
bundle ingest Read-only AI diagnosis of an SRE evidence bundle (v1)
bundle respond Emit AdapterResponse v1 (ranked hypotheses, policy-gated actions)
bundle execute Translate a bundle into a RecoveryPlan (dry-run)
registry list / search / install Discover and install check plugins
mcp Start MCP server on stdio — 8 read-only diagnosis tools (src/mcp/server.ts); the MCP surface never mutates infrastructure
completions Generate bash/zsh/fish shell completions

Output Modes

Three output modes are supported:

  • human (default for TTY): colored, interactive, emoji severity indicators
  • pipe (auto-detected when stdout is not a TTY): plain text, no ANSI, tab-separated
  • machine (--json): structured JSON/JSONL with metadata

Escalation Levels

Five progressive escalation levels surface in scan output and recovery proposals:

  1. Observe — read-only health checks, no system interaction
  2. Diagnose — read-only queries against live systems
  3. Suggest — generate recovery plans without executing
  4. Repair (safe) — execute routine/elevated risk actions
  5. Repair (destructive) — execute high/critical risk actions

Supporting modules: detect.ts (system detection), autodiscovery.ts (zero-config agent detection), output.ts (structured output formatting), errors.ts (error formatting), status-presentation.ts (single source for status → presentation mappings). The five-level escalation model lives in src/framework/escalation.ts.

Agent Pattern

Every agent follows this structure:

src/agent/<system>/
  backend.ts        # Interface (PgBackend, RedisBackend, etc.)
  simulator.ts      # In-memory implementation for demos/tests
  live-client.ts    # Real infrastructure client
  manifest.ts       # AgentManifest — capabilities, risk profile, triggers
  agent.ts          # RecoveryAgent implementation
  registration.ts   # Lazy factory for the agent registry

When building a new agent:

  1. Define the backend interface with async methods
  2. Build the simulator first — it enables demo mode and testing
  3. The live client queries real infrastructure (database, cache, API)
  4. The manifest declares what the agent targets, its max risk level, and trigger conditions
  5. The agent uses diagnosis findings to dynamically build plans — never hardcode IPs or hostnames
  6. Create registration.ts with a lazy factory and register in src/config/builtin-agents.ts

Recovery Plan Steps

7 step types are available (defined in src/types/step-types.ts):

  • diagnosis_action — read-only data gathering
  • human_notification — send alerts to stakeholders
  • checkpoint — capture state before mutations
  • system_action — execute commands with preconditions, success criteria, blast radius
  • human_approval — gate execution pending human decision
  • replanning_checkpoint — agent can revise the remaining plan mid-flight
  • conditional — branch execution based on system state

Safety Rules

  • Every system_action at elevated risk or higher MUST have statePreservation.before captures
  • Every plan with elevated+ steps MUST include a human_notification step
  • Plans MUST have a rollbackStrategy
  • Step IDs must be unique within a plan
  • No nested conditionals
  • Blast radius must declare affected components

These are enforced by the validator (src/framework/validator.ts).

Testing

Test environment

  • ./test/podman/scripts/start.sh — starts PG primary/replica, Prometheus, AlertManager, mock hub
  • ./test/smoke/run-all.sh — validates the test environment (16 checks)
  • ./test/failures/*.sh — inject specific failures into PostgreSQL

Running against real infrastructure

  • pnpm run live — dry-run against podman test PG
  • pnpm run live -- --execute — execute mode (will mutate real PG)
  • pnpm run webhook — start webhook receiver for AlertManager

Unit tests

  • pnpm test — runs vitest unit tests (src/__tests__/*.test.ts)
  • pnpm run test:watch — runs vitest in watch mode
  • Configuration in vitest.config.ts

Type checking

  • pnpm run typecheck — runs tsc --noEmit

Linting

  • pnpm run lint — ESLint (flat config in eslint.config.js); pnpm run lint:fix to autofix
  • Lint-time TypeScript is pinned to 6.0.2 via .pnpmfile.cjs (typescript-eslint does not yet support the TS 7 native compiler); tsc stays on TS 7

CI

  • GitHub Actions (.github/workflows/ci.yml) — runs typecheck, unit tests, and gitleaks on push to main and PRs

Key Files

File Purpose
src/agent/interface.ts RecoveryAgent contract — start here for understanding the agent model
src/framework/engine.ts ExecutionEngine — how plans are executed step by step
src/framework/graph-engine.ts LangGraph-based graph execution engine
src/framework/symptom-router.ts Routes symptoms to appropriate recovery agents
src/framework/ai-diagnosis-universal.ts Universal AI-powered diagnosis for any agent
src/framework/incident-report.ts Structured incident report generation
src/framework/network-profile.ts Network diagnostics and profiling
src/types/step-types.ts All 7 recovery step types
src/types/recovery-plan.ts RecoveryPlan structure
src/cli/index.ts Unified CLI entry point
src/cli/commands/ CLI subcommands (scan, diagnose, recover, status, ask, demo, init, webhook, watch, readiness, agent, playbook, bundle, registry, completions)
src/mcp/server.ts MCP server — 8 read-only diagnosis tools exposed via crisismode mcp
src/readiness/ Scale-readiness rule registry + capacity ceilings/weak-link (readiness command + MCP tool)
src/framework/escalation.ts Five-level progressive escalation model
src/agent/pg-replication/ Reference agent implementation (PostgreSQL)
src/agent/redis/ Redis memory pressure recovery agent
src/agent/etcd/ etcd consensus recovery agent
src/agent/kafka/ Kafka broker recovery agent
src/agent/kubernetes/ Kubernetes cluster recovery agent
src/agent/ceph/ Ceph storage recovery agent
src/agent/flink/ Flink stream processing recovery agent
src/agent/ai-provider/ AI service failover and fallback agent
src/agent/config-drift/ Configuration drift detection and remediation agent
src/agent/db-migration/ Database migration safety and rollback agent
src/agent/deploy-rollback/ Deployment rollback orchestration agent
src/agent/queue-backlog/ Queue backlog and lag recovery agent
src/agent/dns/ DNS resolution failure recovery agent
src/agent/tls/ TLS certificate health and expiry agent
src/agent/disk/ Local disk exhaustion detection agent
src/agent/backup/ Backup verification and DR readiness agent
src/agent/aws-s3/ AWS S3 backup configuration agent
src/agent/aws-dynamodb/ AWS DynamoDB PITR verification agent
src/agent/aws-rds/ AWS RDS backup and snapshot agent
src/config/builtin-agents.ts Built-in agent registration
src/config/agent-registry.ts Global agent registry
src/integrations/ External integrations (GitHub, Sentry)
specs/foundational/recovery-agent-contract.md The authoritative specification
src/framework/backend.ts ExecutionBackend contract — shared interface for all backends
src/framework/provider-registry.ts Resolves capability providers for plan steps
src/framework/capability-registry.ts Global registry of standard recovery capabilities
src/framework/operator-summary.ts Builds operator-facing health and readiness summaries
src/types/health.ts Health assessment and operator summary types
src/types/plugin.ts Plugin ecosystem types (capability providers, domain packs, etc.)
specs/deployment/operations.md Hub-and-spoke deployment architecture
specs/architecture/plugin-platform.md Plugin platform architecture guide
specs/architecture/operator-health-and-ai-services.md Operator summary, AI services, and site config spec
packages/agent-sdk/ @crisismode/agent-sdk — public types package
src/framework/playbook/ Markdown playbook parser, runtime, and discovery
src/framework/hooks/ Pluggable lifecycle hook system
src/framework/registry/ Agent plugin discovery and manifest handling
specs/foundational/playbook-format.md Playbook format specification
specs/foundational/registry-manifest.md Agent manifest specification
playbooks/examples/ Reference playbook implementations
CONTRIBUTING.md Contribution guide (agents and playbooks)
docs/architecture.md System architecture overview
docs/readiness.md Scale-readiness usage and extension guide (rules, ceilings, honesty contract)
docs/guides/creating-a-recovery-agent.md Agent development tutorial
docs/playbook-authoring.md Playbook authoring guide

Commit Conventions

Use Conventional Commits:

type(scope): description

Types: feat, fix, docs, style, refactor, test, chore, ci, perf, build

What NOT to Do

  • Don't hardcode IPs, hostnames, or infrastructure identifiers in agents — discover them at diagnosis time
  • Don't skip pre-commit hooks (--no-verify) unless explicitly asked
  • Don't add dependencies without considering the spoke's resource footprint (256Mi memory target)
  • Don't bypass safety validations — they exist because wrong actions during a crisis are catastrophic
  • Don't store secrets in code — credentials come from K8s Secrets or environment variables at runtime
  • Don't create agents with maxRiskLevel: 'critical' without explicit discussion — critical operations require the highest level of human oversight