ai-incident-responder

A containerised AI-assisted incident response pipeline: Alertmanager webhooks, enrichment with Prometheus, runbook generation via Ollama or Perplexity, storage in PostgreSQL, and a React dashboard. The pipeline can be run locally for experimentation or extended to production with external LLM APIs. Built for DevOps and SRE experimentation with LLM-driven runbook generation on a single host.

What this is

Manual incident response is slow. Engineers correlate alerts with metrics, search runbooks, and decide escalation. This project implements a full pipeline from alert fire to stored incident and dashboard. Alertmanager POSTs to the backend. The backend enriches each firing alert with a Prometheus query (for example, up or up{instance="..."}), calls an LLM (Ollama on the host or Perplexity API when PPLX_API_KEY is set) with a fixed SRE-style system prompt to produce a runbook, stores the incident and linked alert row in PostgreSQL, optionally posts a summary to Slack or logs to /app/logs/slack.log, and returns. The React frontend lists incidents and shows incident detail including the LLM-generated runbook (markdown). The approach is minimal and explicit: one Compose stack, optional Ollama on the host, no Kubernetes or managed services.

A few limitations. There is no authentication on the backend or frontend. The stack should not be exposed to the internet. When Ollama is used, it must run on the host. The backend reaches it at host.docker.internal:11434. When PPLX_API_KEY is set in .env, keep .env out of version control (it is in .gitignore). The key is never logged or exposed to the frontend. The setup is single-node with no HA, no queue, and no idempotency for duplicate webhooks. Enrichment is minimal (one Prometheus instant query per alert). Slack notification is best-effort (fire-and-forget). Base images may contain upstream OS-level CVEs. See SECURITY.md.

Complexity and what this project covers

This repo is a single, self-contained incident-response pipeline. The complexity is moderate. It provides a working webhook to enrich to LLM to store to UI flow that can be run locally and later pointed at Perplexity or another LLM API.

Tools and technologies

FastAPI for the HTTP API: webhook receiver, incident CRUD, health (Postgres + Ollama).
PostgreSQL 16 for storing incidents and alert rows (SQLAlchemy 2 async with asyncpg).
Ollama (on the host) for runbook generation when PPLX_API_KEY is not set (llama3). The app calls Ollama over HTTP from inside Docker.
Perplexity API (optional): when PPLX_API_KEY is set in .env, runbooks are generated via Perplexity (model PPLX_MODEL, default sonar) for faster, parallel-friendly responses.
LangChain (langchain-ollama) for Ollama. httpx for Perplexity. No chain orchestration. The flow is explicit in routers and services.
Prometheus and Alertmanager in the same Compose stack. Alertmanager routes alerts to the backend /webhook.
React (Vite, Tailwind, react-markdown) for the dashboard. CORS enabled for localhost:3000.
Docker Compose to run backend, frontend, Postgres, Prometheus, and Alertmanager. Ollama stays on the host when used.
pytest with pytest-asyncio and httpx (ASGITransport) for integration tests. Optional script to validate runbook accuracy.

Topics actually covered

Alertmanager webhook ingestion and parallel processing of firing alerts.
Prometheus enrichment (instant query for up or up{instance="..."}).
LLM runbook generation (Ollama or Perplexity) with a fixed SRE prompt (root cause, immediate actions, runbook, escalation).
Incident and alert storage in Postgres. Resolve via PATCH.
Optional Slack notification or file logging.
Health checks for Postgres and Ollama.
React dashboard for active incidents, history, and runbook detail.

What this project does not cover

No authentication or rate limiting.
No idempotency or deduplication of webhook payloads.
No multi-query or custom metric enrichment. Single up-style query only.
No production deployment (Kubernetes, TLS, secrets manager). The README describes production considerations. The rest is left to the operator.

Prerequisites

Docker Engine and Docker Compose (v2) are required. When using Ollama, install it on the host and pull the model before starting the stack. Ports 3000, 8000, 9090, and 9093 must be free (Postgres 5432 is internal).

# If using Ollama (optional when PPLX_API_KEY is set)
ollama pull llama3

Quick start

From the repo root (for example after a git clone):

make up
curl -s http://localhost:8000/health
make trigger-alert

The stack needs a few seconds after make up for Postgres and the backend to be ready. The dashboard is at http://localhost:3000. Optional: make resolve resolves the first open incident. make resolve ID=3 resolves incident 3. The pytest suite runs inside the backend container via make test.

End-to-end workflow (how everything works)

This section walks through one full path from an incoming alert to a runbook in the dashboard.

1. Start the stack

Running make up builds the backend and frontend images and starts Postgres (with healthcheck), backend, frontend, Prometheus, and Alertmanager. The backend waits for Postgres to be healthy, then starts. It exposes /health, /webhook, /incidents, and /metrics. When Ollama is used it is not in Docker. The backend reaches it at host.docker.internal:11434. When PPLX_API_KEY is set, runbooks use Perplexity instead and Ollama is optional.

2. Webhook receipt

Alertmanager (or make trigger-alert, which sends 10 varied alerts) POSTs an Alertmanager-format payload to POST /webhook. The payload has status and alerts[]. Each alert has status, labels, annotations, startsAt, endsAt. Only alerts with status == "firing" are processed. Firing alerts are processed in parallel. Each alert is enriched, sent to the LLM, then stored and optionally sent to Slack.

3. Enrichment and LLM runbook

For each firing alert, the enricher calls Prometheus GET /api/v1/query with query up or up{instance="<instance>"} (from labels). The result is formatted into a short text block. The backend then calls get_llm_runbook(alertname, summary, description, enriched_context). If PPLX_API_KEY is set, runbooks are generated via the Perplexity API (model from PPLX_MODEL, max_tokens 1024). Otherwise Ollama (llama3) is used. The system prompt asks for technically precise root cause, immediate actions, runbook steps, and escalation. The response is stored as the incident’s runbook or an error string on failure.

4. Storage and Slack

An Incident row is created (alertname, severity, summary, description, instance, enriched_context, llm_runbook, status=open). An Alert row is created linking to the incident. If SLACK_WEBHOOK_URL is set, a formatted summary is POSTed to Slack. In all cases the same text is appended to /app/logs/slack.log.

5. Dashboard

The React app at http://localhost:3000 calls GET /incidents and GET /incidents/:id. It displays active incidents and incident detail with markdown-rendered runbook. Resolve is done via PATCH /incidents/:id/resolve with {"status":"resolved"}.

Summary

Alerts hit the webhook. The backend enriches with Prometheus, generates a runbook via Ollama or Perplexity, stores the incident in Postgres, optionally notifies Slack, and returns. The frontend lists and displays incidents. The rest of this README has concrete curl examples, the full command list, and troubleshooting.

Commands

Command	What it does
`make up`	Build and start backend, frontend, Postgres, Prometheus, Alertmanager in the background
`make down`	Stop containers and remove volumes
`make status`	List running containers
`make logs`	Follow Compose logs
`make test`	Run pytest in the backend container (`pytest /app/tests -v`)
`make trigger-alert`	Run `scripts/trigger_alerts.py`: send 10 varied alerts to /webhook with progress output
`make resolve`	PATCH first open incident as resolved. Use `make resolve ID=3` to resolve incident 3

Tests

The tests are integration tests. They hit the real app over ASGI with real Postgres inside the backend container. When testing Perplexity, PPLX_API_KEY must be set in the environment (for example, from .env when running via Compose). The suite is run with make test.

When Ollama is used, it must be reachable from the host (the app uses host.docker.internal). Event-loop or “different loop” errors are avoided by running make test as defined in the Makefile (inside the backend container), not from the host.

Sample input and output

With the stack up, the API is at http://localhost:8000.

Health

curl -s http://localhost:8000/health

Example response:

{
  "status": "ok",
  "checks": {
    "postgres": "ok",
    "ollama": "ok"
  }
}

Trigger alert (same shape as make trigger-alert)

curl -s -X POST http://localhost:8000/webhook \
  -H "Content-Type: application/json" \
  -d '{"receiver":"backend-webhook","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"HighErrorRate","severity":"critical","instance":"backend:8000"},"annotations":{"summary":"High error rate detected","description":"Error rate above threshold"},"startsAt":"2025-03-07T12:00:00Z","endsAt":"0001-01-01T00:00:00Z"}]}'

Example response:

{"status":"ok"}

List open incidents

curl -s "http://localhost:8000/incidents?status=open"

Example response (abbreviated):

[
  {
    "id": 1,
    "alertname": "HighErrorRate",
    "severity": "critical",
    "summary": "High error rate detected",
    "status": "open",
    "created_at": "2025-03-07T12:00:05.123456",
    "resolved_at": null
  }
]

Get incident and resolve

curl -s http://localhost:8000/incidents/1
curl -s -X PATCH http://localhost:8000/incidents/1/resolve -H "Content-Type: application/json" -d '{"status":"resolved"}'

Screenshots

Screenshots live in docs/screenshots/. Add the following filenames and they will render below.

Stack status

Dashboard Active incidents

Dashboard Runbook

Dashboard History

Pipeline demo

Health check

Speed and performance

Rough numbers for the default local setup:

Scenario	Typical behaviour
Webhook (enrich, LLM, store)	Enrichment about 50 to 200 ms. LLM (Ollama) 2 to 15 s per alert. With Perplexity and parallel processing, 10 alerts complete in roughly the time of the slowest request.
GET /incidents, GET /incidents/:id	Under 100 ms
PATCH resolve	Under 50 ms
Health	Postgres under 10 ms. Ollama check up to 3 s timeout, often under 500 ms if local.

Repeated webhook payloads are processed in parallel (asyncio.gather). With Perplexity, runbooks are generated concurrently to reduce total time.

Architecture

The system runs as a Docker Compose stack. The backend depends on Postgres and optionally Ollama on the host or the Perplexity API. Alertmanager sends alerts to the backend. The frontend calls the backend for incidents and resolve.

flowchart LR
  subgraph external["Host"]
    Ollama["Ollama :11434 (llama3)"]
  end

  subgraph stack["Stack"]
    AM["Alertmanager :9093"]
    Backend["Backend :8000 webhook enricher LLM DB"]
    PG[(Postgres)]
    Prom["Prometheus :9090"]
    FE["Frontend :3000"]
  end

  AM -->|POST /webhook| Backend
  Backend -->|query| Prom
  Backend -->|runbook| Ollama
  Backend -->|read/write| PG
  Backend -.->|optional| Slack["Slack / file log"]
  FE -->|GET /incidents, PATCH resolve| Backend
  Backend -.->|exposes /metrics| Prom

Webhook path. Alertmanager POSTs to /webhook. The backend parses the payload, filters firing alerts, and for each enriches via Prometheus, calls the LLM (Ollama or Perplexity), stores the incident and alert in Postgres, and optionally posts to Slack. Frontend path. The React app fetches GET /incidents and GET /incidents/:id and sends PATCH /incidents/:id/resolve to mark incidents resolved.

API summary

Method	Endpoint	Use
GET	/health	Postgres and Ollama connectivity. Returns JSON with status and checks.
POST	/webhook	Alertmanager webhook. Body: WebhookPayload. Returns {"status":"ok"}.
GET	/incidents	List incidents. Optional query `status` (open or resolved).
GET	/incidents/{id}	Single incident by id. 404 if not found.
PATCH	/incidents/{id}/resolve	Resolve incident. Body: {"status":"resolved"}. Returns updated incident.
GET	/metrics	Prometheus text format

Troubleshooting

Health shows Ollama failed

Ollama must be running on the host and the backend must reach it at host.docker.internal:11434. Start Ollama before the stack and ensure the model is pulled (ollama pull llama3). On Linux without Docker Desktop, host.docker.internal may not resolve. Add extra_hosts in docker-compose to point to the host. When using Perplexity only, Ollama is optional and the health check may still report Ollama as failed unless the health logic is changed.

LLM runbook is “LLM runbook generation failed: …”

If using Ollama: run ollama pull llama3, check Ollama logs, and ensure OLLAMA_MODEL=llama3. If using Perplexity: ensure PPLX_API_KEY is set in the environment (e.g. in .env and passed via Compose) and that the key is valid and has quota.

Postgres check failed in /health

Postgres may not be ready yet or DATABASE_URL may not match docker-compose (user, pass, host postgres, db incidents). Wait for the Postgres healthcheck to pass and confirm the URL.

Port 3000 or 8000 in use

Another stack (e.g. obs-stack-local) or a local process is using the port. Stop the other stack or change ports in docker-compose.yml.

trigger-alert returns 422

The request body must match the WebhookPayload shape: receiver, status, alerts array with status, labels, annotations, startsAt, endsAt. See Sample input and output above.

No incidents in UI

No webhook was received or only resolved alerts were in the payload. Call make trigger-alert or fire an alert from Alertmanager. The backend only creates incidents for alerts with status "firing".

Tests fail (event loop, async)

Run make test inside the backend container as the Makefile does. The tests use pytest-asyncio and a session-scoped event loop. The app is lazy-imported in fixtures to avoid “different loop” errors.

Production considerations

The same codebase can be used in production with stronger auth, TLS, and external LLM or notification services. Add authentication and authorization for /webhook and /incidents. Use TLS and restrict access to Prometheus and Alertmanager. For the LLM, consider a dedicated service with rate limits and retries. Running Ollama on the host is for simplicity only. Deduplicate webhooks by alert fingerprint or (alertname, instance, startsAt) to avoid duplicate incidents. Extend the enricher with more Prometheus queries and custom logic per alertname. Add retries and dead-letter handling for Slack. Run multiple backend replicas behind a load balancer with a shared Postgres and consider a queue for webhook processing. Set memory and CPU limits for the backend and Postgres in docker-compose. Do not commit .env. Use a secrets manager or deployment env for API keys.

Project structure

.
├── LICENSE
├── Makefile
├── README.md
├── SECURITY.md
├── alertmanager
│   └── alertmanager.yml
├── backend
│   ├── Dockerfile
│   ├── core.py
│   ├── main.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── database.py
│   │   └── schemas.py
│   ├── pytest.ini
│   ├── requirements.txt
│   ├── routers
│   │   ├── __init__.py
│   │   ├── health.py
│   │   ├── incidents.py
│   │   └── webhook.py
│   └── services
│       ├── __init__.py
│       ├── enricher.py
│       ├── llm_service.py
│       └── slack_service.py
├── docker-compose.yml
├── docs
│   ├── incident-runbook-review.md
│   └── screenshots
│       ├── dashboard-active.png
│       ├── dashboard-history.png
│       ├── dashboard-runbook.png
│       ├── health-check.png
│       ├── pipeline-demo.png
│       └── stack-status.png
├── frontend
│   ├── Dockerfile
│   ├── index.html
│   ├── package.json
│   ├── postcss.config.js
│   ├── src
│   │   ├── App.jsx
│   │   ├── components
│   │   │   ├── ActiveIncidents.jsx
│   │   │   ├── HistoryTable.jsx
│   │   │   ├── IncidentDetail.jsx
│   │   │   └── StatusBadge.jsx
│   │   ├── index.css
│   │   └── main.jsx
│   ├── tailwind.config.js
│   └── vite.config.js
├── prometheus
│   ├── alert_rules.yml
│   └── prometheus.yml
├── scripts
│   ├── check_incidents.py
│   └── trigger_alerts.py
└── tests
    ├── conftest.py
    ├── test_enricher.py
    ├── test_incidents.py
    ├── test_pplx.py
    └── test_webhook.py

Environment

Copy .env.example to .env to override defaults or set PPLX_API_KEY. Compose passes through PPLX_API_KEY and PPLX_MODEL from the host environment (for example, from .env in the project root). The backend also reads DATABASE_URL, OLLAMA_BASE_URL, OLLAMA_MODEL, PROMETHEUS_URL, and SLACK_WEBHOOK_URL. Do not commit .env. It is in .gitignore.

For others who want to experiment

The repo can be cloned and run locally. Docker and Docker Compose are required. When not using Perplexity, Ollama with llama3 is required. Optional: set PPLX_API_KEY in .env for faster runbooks via Perplexity. After cloning, copy .env.example to .env when using Perplexity or Slack, run make up, wait a few seconds, then run curl -s http://localhost:8000/health, make trigger-alert, and open http://localhost:3000. Incidents are resolved with make resolve or make resolve ID=3. The test suite runs with make test. Optional: python3 scripts/check_incidents.py (with backend up) to validate runbook accuracy. Tear down with make down.

Disclaimer

This project is for learning and personal experimentation only. It is not intended as production-ready software, professional advice, or a guarantee of any result. Use at your own risk. The author provides no warranty and is not liable for any use or misuse of this code. Base images may contain known CVEs. See SECURITY.md.

License

MIT. See LICENSE.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

ai-incident-responder

What this is

Complexity and what this project covers

Prerequisites

Quick start

End-to-end workflow (how everything works)

Commands

Tests

Sample input and output

Screenshots

Speed and performance

Architecture

API summary

Troubleshooting

Production considerations

Project structure

Environment

For others who want to experiment

Disclaimer

License

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
alertmanager		alertmanager
backend		backend
docs		docs
frontend		frontend
prometheus		prometheus
scripts		scripts
tests		tests
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation

ai-incident-responder

What this is

Complexity and what this project covers

Prerequisites

Quick start

End-to-end workflow (how everything works)

Commands

Tests

Sample input and output

Screenshots

Speed and performance

Architecture

API summary

Troubleshooting

Production considerations

Project structure

Environment

For others who want to experiment

Disclaimer

License

About

Topics

Resources

License

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages