Production monitoring stack for Django + PostgreSQL + Celery
Prometheus · Grafana · Alertmanager · OpenTelemetry Collector · OpenSearch on LKE · exporters
Dashboards are provisioned automatically:
| Dashboard | Source ID |
|---|---|
| Django | 17658 |
| PostgreSQL | 9628 |
| Celery | 17508 |
cp .env.example .env
# Edit .env: Grafana password, Postgres DSN, Redis/Celery broker, Django scrape target
docker-compose -f docker-compose.monitoring.yml up -d| Service | Default URL (localhost-bound) |
|---|---|
| Grafana | http://127.0.0.1:3000 |
| Prometheus | http://127.0.0.1:9090 |
| Alertmanager | http://127.0.0.1:9093 |
| OTLP gRPC | 127.0.0.1:4317 |
| OTLP HTTP | 127.0.0.1:4318 |
| Flower | not published — attach to the network or publish behind a reverse proxy |
Exporter ports (9187, 9808) and collector internals (8888, 13133) stay on the private monitoring network only.
- Install and configure
django-prometheus:
pip install django-prometheusINSTALLED_APPS += ["django_prometheus"]
MIDDLEWARE = [
"django_prometheus.middleware.PrometheusBeforeMiddleware",
# ... your middleware ...
"django_prometheus.middleware.PrometheusAfterMiddleware",
]
DATABASES = {
"default": {
"ENGINE": "django_prometheus.db.backends.postgresql",
# ...
}
}# urls.py
urlpatterns = [
path("", include("django_prometheus.urls")),
# ...
]- Point the stack at your metrics endpoint in
.env:
# Django on the host, monitoring in Docker:
DJANGO_METRICS_TARGET=host.docker.internal:8000
# Django as another Compose service on a shared network:
DJANGO_METRICS_TARGET=web:8000- Restrict
/metricsin production (firewall, allowlist Prometheus IPs, or auth). Do not expose it publicly.
The collector accepts OTLP traces, metrics, and logs. Traces and logs are exported to OpenSearch hot nodes on LKE. Metrics stay on Prometheus (django-prometheus / exporters) so existing Grafana dashboards and alerts keep working.
- Install the SDK and instrumentations in your Django app:
pip install opentelemetry-sdk opentelemetry-exporter-otlp \
opentelemetry-instrumentation-django \
opentelemetry-instrumentation-psycopg \
opentelemetry-instrumentation-celery \
opentelemetry-instrumentation-redis- Point the app at the collector:
# Django on the host, monitoring in Docker:
export OTEL_SERVICE_NAME=django
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
# Django as another Compose service on a shared network:
# export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# export OTEL_EXPORTER_OTLP_PROTOCOL=grpc- Set the OpenSearch hot endpoint in
.env(see OpenSearch on LKE). Confirm spans arrive:
docker-compose -f docker-compose.monitoring.yml logs -f otel-collectorOTLP has no authentication. Leave the ports on 127.0.0.1 or put the collector behind a mesh/proxy that enforces TLS and auth.
Use LKE with an HA control plane (3 replicas). That is separate from these worker pools:
| Pool | Nodes | OpenSearch roles | What lands here |
|---|---|---|---|
| master | 3 | cluster_manager only |
Cluster state — no ingest, no data |
| hot | 3 | data + ingest |
New traces/logs (node.attr.temp=hot) |
| medium | 3 | data |
Indices older than 7 days |
The collector writes only to the hot service (djangoobserve-hot). Masters are never ingest targets.
Suggested Linode pool sizes (leave headroom for kubelet):
- master: 8 GB RAM (
g6-standard-4) - hot: 16 GB RAM (
g6-standard-6), SSD - medium: 16 GB RAM, larger disks
- Create the three LKE node pools and label every node:
kubectl label node <master-nodes> djangoobserve.io/pool=master
kubectl label node <hot-nodes> djangoobserve.io/pool=hot
kubectl label node <medium-nodes> djangoobserve.io/pool=mediumOptional taint so only OpenSearch / collector pods schedule there:
djangoobserve.io/pool=<master|hot|medium>:NoSchedule
- Create admin credentials, install the OpenSearch operator, then apply this repo's manifests:
kubectl create namespace observability
kubectl create secret generic djangoobserve-admin-credentials \
-n observability \
--from-literal=username=admin \
--from-literal=password='choose-a-strong-password'
helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/
helm install opensearch-operator opensearch-operator/opensearch-operator \
-n opensearch-operator --create-namespace
kubectl apply -k k8s/lkeWait until the cluster is green and djangoobserve-admin-credentials is present before expecting collector pods to export.
- After the cluster is green, apply the hot→medium ISM policy and index template (new writes stay on hot, then move to medium at 7 days):
# Use admin credentials from secret djangoobserve-admin-credentials
curl -k -u admin:"$OPENSEARCH_PASSWORD" \
-H "Content-Type: application/json" \
-X PUT "https://<hot-or-dashboards-host>/_plugins/_ism/policies/otel-hot-medium" \
--data @k8s/lke/ism-hot-medium.json
curl -k -u admin:"$OPENSEARCH_PASSWORD" \
-H "Content-Type: application/json" \
-X PUT "https://<hot-or-dashboards-host>/_index_template/ss4o-otel" \
--data @k8s/lke/index-template.json- Point Compose (or any off-cluster collector) at the hot pool:
OPENSEARCH_ENDPOINT=https://<hot-loadbalancer>:9200
OPENSEARCH_USERNAME=admin
OPENSEARCH_PASSWORD=...In-cluster, k8s/lke/otel-collector.yaml already targets https://djangoobserve-hot.observability.svc.cluster.local:9200 and publishes OTLP on a LoadBalancer (4317/4318).
Create a read-only monitoring role if you can:
CREATE USER monitor_user WITH PASSWORD 'secret';
GRANT pg_monitor TO monitor_user;Then set:
POSTGRES_EXPORTER_DATA_SOURCE_NAME=postgresql://monitor_user:secret@db:5432/postgres?sslmode=requireIf Postgres is not on the Compose network, use the host IP / host.docker.internal and ensure the network allows the connection.
CELERY_BROKER_URL=redis://:password@redis:6379/1
FLOWER_BASIC_AUTH=admin:strong-passwordCelery workers should emit events for richer exporter metrics:
celery -A your_project worker -ETo use Flower's UI in production, put it behind TLS + SSO (or publish only on localhost):
# optional overlay snippet
services:
flower:
ports:
- "127.0.0.1:5555:5555"Default rules live in prometheus/rules/alerts.yml (target down, Django 5xx/latency, Postgres connections, Celery failures).
Alertmanager starts with a no-op receiver (alerts visible in the UI). For Slack:
cp alertmanager/alertmanager.slack.yml.example alertmanager/alertmanager.yml
# set slack_api_url / channel, then:
docker-compose -f docker-compose.monitoring.yml up -d alertmanager- Secrets live in
.env(gitignored). Never commit real passwords. - UIs and OTLP receivers bind to
127.0.0.1by default — put Grafana behind Nginx/Traefik/Caddy with TLS and real auth (OAuth/LDAP) for public access. Do not publish4317/4318on a public interface. - OpenSearch ingest must go to hot nodes only. Do not send OTLP or bulk index traffic to master nodes.
- The collector skips OpenSearch TLS verify (
insecure_skip_verify) so operator-generated certs work; pin the cluster CA in production. - Prometheus retention defaults to
15d/10GB(override in.env). - Dashboards and the Prometheus datasource are provisioned from
grafana/; rebuilds stay reproducible. - Prefer
sslmode=require(or stronger) for Postgres when not on a private network.
If Prometheus exits with permission errors on /prometheus, fix the named volume once:
docker run --rm -v "$(basename "$PWD")_prometheus_data:/p" busybox chown -R 65534:65534 /pThen start the stack again.
docker-compose -f docker-compose.monitoring.yml ps
docker-compose -f docker-compose.monitoring.yml logs -f prometheus
docker-compose -f docker-compose.monitoring.yml logs -f otel-collector
docker-compose -f docker-compose.monitoring.yml down