AI-Observability-Platform-Architecture.md

This is my architectural plan to build out an AI network observability/security agent

AI-Driven Infrastructure Observability Platform

Engineering Architecture Document

---

1. Full System Architecture

The platform is a four-layer stack: Collection → Transport → Intelligence → Presentation.

Collection Layer — Lightweight agents and sensors deployed on every monitored VM. Node Exporter exposes host metrics over HTTP. Filebeat tails syslog, application logs, and auth logs. Zeek (or Suricata) performs passive network analysis on a mirrored interface, producing structured connection logs (conn.log, dns.log, http.log, etc.).

Transport Layer — All telemetry converges on a centralized ingestion bus. Prometheus scrapes metric endpoints on a 15-second interval. Filebeat ships logs to Elasticsearch via Logstash (which handles parsing, enrichment, and routing). Zeek logs are ingested either via Filebeat's Zeek module or by writing directly to a shared NFS mount consumed by the ML pipeline.

Intelligence Layer — A dedicated ML node runs scheduled Python jobs (orchestrated by Airflow or cron). Three model pipelines operate independently: an Isolation Forest anomaly detector on network telemetry, a Prophet time-series forecaster on system metrics, and a TF-IDF + Logistic Regression classifier on log text. Model outputs (anomaly scores, forecasts, log labels) are written back to Elasticsearch as enriched indices and exposed as Prometheus custom metrics via a Python pushgateway exporter.

Presentation Layer — Grafana serves as the single pane of glass. It queries Prometheus for real-time metrics and forecasts, Elasticsearch for log classification results and anomaly events, and renders alerting rules that trigger on ML-derived thresholds.

┌─────────────────────────────────────────────────────────────────────┐
│                      PRESENTATION LAYER                             │
│                  Grafana (dashboards + alerts)                       │
└──────────┬────────────────────┬─────────────────────────────────────┘
           │                    │
     Prometheus            Elasticsearch
           │                    │
┌──────────┴────────────────────┴─────────────────────────────────────┐
│                     INTELLIGENCE LAYER                               │
│  ┌──────────────┐  ┌──────────────────┐  ┌────────────────────┐     │
│  │ Anomaly Det. │  │ Capacity Forcast │  │ Log Classifier     │     │
│  │ IsolationFor │  │ Prophet          │  │ TF-IDF + LogReg    │     │
│  └──────────────┘  └──────────────────┘  └────────────────────┘     │
└──────────┬────────────────────┬──────────────────┬──────────────────┘
           │                    │                  │
┌──────────┴────────────────────┴──────────────────┴──────────────────┐
│                      TRANSPORT LAYER                                 │
│  Prometheus scrape  │  Logstash pipeline  │  Zeek log shipping      │
└──────────┬────────────────────┬──────────────────┬──────────────────┘
           │                    │                  │
┌──────────┴────────────────────┴──────────────────┴──────────────────┐
│                      COLLECTION LAYER                                │
│  Node Exporter  │  Filebeat  │  Zeek/Suricata  │  Custom exporters  │
│           (deployed per VM)                                          │
└─────────────────────────────────────────────────────────────────────┘

---

2. Virtual Machine Topology

Six VMs are recommended. All run Ubuntu 22.04 LTS.

VM Role	Hostname	vCPU	RAM	Disk	Purpose
Observability Server	obs-server	4	8 GB	100 GB SSD	Prometheus, Grafana, Alertmanager
Log Server	log-server	4	12 GB	200 GB SSD	Elasticsearch (single-node), Logstash, Kibana
Network Sensor	net-sensor	2	4 GB	50 GB	Zeek/Suricata, 2 NICs (mgmt + mirror/span)
ML Node	ml-node	4	8 GB	80 GB SSD	Python ML pipelines, Airflow scheduler, model storage
Telemetry Node A	telem-a	2	4 GB	40 GB	Simulated workload, Node Exporter, Filebeat
Telemetry Node B	telem-b	2	4 GB	40 GB	Simulated workload, Node Exporter, Filebeat

Total resource requirement: 18 vCPU, 40 GB RAM, 510 GB disk. Fits on a workstation with 32+ GB RAM if telem nodes are kept lean.

Networking: All VMs on a shared VMware NAT or host-only network (172.16.0.0/24). The network sensor's second NIC is connected to a port mirror/promiscuous VLAN segment carrying traffic from both telemetry nodes. VMware Workstation supports promiscuous mode on virtual switches to enable this.

---

3. Telemetry Pipeline

Metrics Pipeline

1. Node Exporter (port 9100) on every monitored VM exposes ~800 metrics: CPU per-core utilization, memory pressure, disk I/O latency, network bytes/packets, filesystem usage.
2. Prometheus scrapes all exporters every 15s. Recording rules pre-compute 5m rate averages for CPU, memory, and disk. Retention set to 30 days.
3. Custom Python exporter on ml-node pushes forecast metrics and anomaly scores to Prometheus Pushgateway, making ML outputs queryable via PromQL.

Log Pipeline

1. Filebeat on each telemetry node tails /var/log/syslog, /var/log/auth.log, /var/log/kern.log, and any application logs. Each event is tagged with host, log_source, and timestamp.
2. Logstash (on log-server) receives Filebeat output, applies grok filters to extract structured fields (severity, service, PID, message body), and writes to Elasticsearch index logs-YYYY.MM.DD.
3. ML enrichment: The log classifier reads raw logs from Elasticsearch via the Python client, classifies them, and writes results back to a logs-classified index with an added ml_category field.

Network Telemetry Pipeline

1. Zeek on net-sensor runs in cluster mode (single worker for homelab scale) on the mirrored NIC. It produces structured TSV logs: conn.log (connection 5-tuples, durations, byte counts), dns.log, http.log, ssl.log, weird.log.
2. Filebeat Zeek module ships these logs to Elasticsearch for archival and dashboard queries.
3. ML consumption: The anomaly detector reads conn.log features (duration, orig_bytes, resp_bytes, protocol, service, conn_state) directly from disk or Elasticsearch, computes feature vectors, and scores them.

---

4. Machine Learning Architecture

4.1 Network Anomaly Detection — Isolation Forest

Why Isolation Forest: Anomaly detection on network traffic is an unsupervised problem — you don't have labeled "attack" data in a homelab. Isolation Forest is specifically designed for this: it isolates outliers by randomly partitioning feature space, and anomalies require fewer partitions to isolate. It handles high-dimensional numeric data well, trains fast, and requires no distributional assumptions.

Features (derived from Zeek conn.log):

• duration (connection length)
• orig_bytes, resp_bytes (data volume)
• orig_pkts, resp_pkts (packet counts)
• orig_ip_bytes / duration (throughput rate)
• service (one-hot encoded: dns, http, ssl, other)
• conn_state (encoded: S0, S1, SF, REJ, RSTO, etc.)
• hour_of_day, day_of_week (temporal features)

Training: Fit on 7 days of "normal" baseline traffic. Contamination parameter set to 0.01–0.05 (tuned via domain knowledge). Retrain weekly.

Output: Anomaly score per connection (-1 = anomaly, 1 = normal). Connections scoring below the threshold are flagged and written to an anomalies Elasticsearch index with the original connection metadata.

4.2 Capacity Forecasting — Facebook Prophet

Why Prophet: It handles daily/weekly seasonality natively (which infrastructure metrics exhibit — think cron jobs at midnight, lower weekend utilization). It's robust to missing data points, requires minimal hyperparameter tuning, and produces interpretable confidence intervals — exactly what an SRE needs for capacity planning.

Metrics forecasted:

• CPU utilization (%) — 5-minute averaged
• Memory usage (%) — absolute used vs. total
• Disk usage (%) — per mount point
• Network throughput (bytes/sec) — interface level

Training: 30 days of Prometheus data exported via the HTTP API (/api/v1/query_range). Each metric gets its own Prophet model. Models are retrained daily.

Output: 7-day and 30-day forecasts with 80% and 95% confidence intervals. A "days to exhaustion" metric is computed by extrapolating when the upper confidence bound crosses 85% (warning) or 95% (critical). These values are pushed to Prometheus as custom gauge metrics.

4.3 Log Classification — TF-IDF + Logistic Regression

Why TF-IDF + Logistic Regression: For structured log classification, this pipeline is well-proven and production-grade. TF-IDF converts log messages into sparse feature vectors that capture discriminative terms ("segfault," "OOM," "connection refused" vs. "session opened," "cron started"). Logistic Regression is fast to train, interpretable (you can inspect feature weights), and performs well with high-dimensional sparse inputs. More complex models (BERT, etc.) are overkill for this domain and impractical in a homelab.

Classes:

Label	Description	Example patterns
critical	Service-impacting failures	OOM killer, segfault, disk full, kernel panic
warning	Degradation indicators	high latency, retries, connection timeout
informational	Normal operational events	service started, user login, cron executed
noise	Non-actionable entries	DHCP renewal, NTP sync, routine heartbeats

Training data: Manually label 2,000–5,000 log lines from your homelab (this is realistic — a weekend of labeling). Use stratified sampling to ensure class balance. Augment with public datasets like the Loghub collection (https://github.com/logpai/loghub).

Pipeline: raw_text → regex cleanup (strip timestamps, PIDs) → TF-IDF vectorizer (max_features=10000, ngram_range=(1,2)) → Logistic Regression (C=1.0, class_weight='balanced').

Output: Each log line gets a predicted label and confidence score. Low-confidence predictions (< 0.6) are flagged for human review, creating a feedback loop for model improvement.

---

5. Data Flow

TELEMETRY NODES                NETWORK SENSOR
  │ Node Exporter                  │ Zeek conn.log
  │ Filebeat (logs)                │ Filebeat (Zeek logs)
  │                                │
  ▼                                ▼
┌─────────────────────────────────────────────────┐
│              TRANSPORT / STORAGE                 │
│                                                  │
│  Prometheus ◄── metric scrape                    │
│  (TSDB, 30d)                                     │
│                                                  │
│  Logstash ◄── log events ──► Elasticsearch       │
│  (parse/enrich)              (raw logs index)    │
│                              (zeek logs index)   │
└───────┬──────────────────────────┬───────────────┘
        │                          │
        ▼                          ▼
┌─────────────────────────────────────────────────┐
│                ML NODE                           │
│                                                  │
│  1. Query Prometheus HTTP API ──► Prophet         │
│     (30d CPU/RAM/disk/net)       (forecast)      │
│        │                                         │
│        └──► Push forecast metrics to Pushgateway │
│                                                  │
│  2. Query Elasticsearch ──► TF-IDF + LogReg      │
│     (raw log text)           (classify)          │
│        │                                         │
│        └──► Write classified logs back to ES     │
│                                                  │
│  3. Read Zeek conn.log ──► Isolation Forest      │
│     (from ES or NFS)        (score anomalies)    │
│        │                                         │
│        └──► Write anomaly events to ES           │
│            Push anomaly scores to Pushgateway    │
└───────┬──────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────┐
│              GRAFANA                             │
│                                                  │
│  Data sources:                                   │
│    Prometheus → metrics + forecasts + anomaly    │
│    Elasticsearch → logs + classifications +      │
│                    anomaly events + Zeek data     │
│                                                  │
│  Dashboards → Alerts → PagerDuty/Slack webhook   │
└─────────────────────────────────────────────────┘

---

6. Implementation Roadmap

Phase 1: Foundation (Week 1–2)

1. Provision all 6 VMs in VMware Workstation. Assign static IPs on 172.16.0.0/24.
2. Install and configure Node Exporter on telem-a and telem-b.
3. Deploy Prometheus on obs-server. Validate scrape targets are up.
4. Install Grafana on obs-server. Connect Prometheus data source. Import the Node Exporter Full dashboard (ID: 1860).
5. Validate end-to-end: generate CPU load with stress-ng, confirm it appears in Grafana within 30 seconds.

Phase 2: Log Pipeline (Week 3)

6. Deploy Elasticsearch (single-node) and Logstash on log-server.
7. Install Filebeat on telem-a/b. Configure syslog and auth.log inputs.
8. Build Logstash grok pipeline for syslog parsing. Validate structured output in Elasticsearch.
9. Connect Elasticsearch as a Grafana data source. Build a basic log explorer dashboard.

Phase 3: Network Monitoring (Week 4)

10. Configure net-sensor's second NIC in promiscuous mode.
11. Install Zeek. Tune local.zeek for your network (set Site::local_nets).
12. Validate Zeek is producing conn.log, dns.log, http.log.
13. Ship Zeek logs to Elasticsearch via Filebeat Zeek module.
14. Generate synthetic traffic with hping3, nmap, or curl loops to create a baseline dataset.

Phase 4: ML — Log Classification (Week 5)

15. Export 5,000 log lines from Elasticsearch. Manually label them (use a simple CSV + spreadsheet workflow).
16. Build the TF-IDF + Logistic Regression pipeline in a Jupyter notebook on ml-node.
17. Train/evaluate with 80/20 split. Target F1 > 0.85 on each class.
18. Productionize: write a Python script that queries ES for new logs, classifies them, and writes results back. Schedule with cron (every 5 minutes).

Phase 5: ML — Capacity Forecasting (Week 6)

19. Query Prometheus API for 30 days of CPU, RAM, disk, network metrics.
20. Fit Prophet models per metric per host.
21. Generate 7-day and 30-day forecasts. Compute days-to-exhaustion.
22. Push forecasts to Prometheus Pushgateway. Build Grafana forecast overlay dashboards.
23. Set Grafana alerts: warn at < 14 days to exhaustion, critical at < 7 days.

Phase 6: ML — Network Anomaly Detection (Week 7)

24. Extract and featurize 7 days of Zeek conn.log data.
25. Train Isolation Forest. Tune contamination parameter.
26. Score new connections in batch (every 10 minutes via cron).
27. Write anomaly events to Elasticsearch. Build anomaly dashboard in Grafana.

Phase 7: Integration and Hardening (Week 8)

28. Unify all Grafana dashboards into a single "Platform Overview" home dashboard.
29. Configure Alertmanager with routing rules and a Slack/Discord webhook.
30. Write a stress_test.sh script that simulates failures (disk fill, CPU spike, port scan, log flood) to validate end-to-end detection.
31. Document everything. Write the README. Record a demo walkthrough.

---

7. Recommended Software Stack

Layer	Tool	Version	Purpose
Metrics collection	Prometheus Node Exporter	1.7+	Host-level metrics
Metrics storage	Prometheus	2.50+	TSDB, scraping, PromQL
Metrics gateway	Prometheus Pushgateway	1.7+	ML-derived metric ingestion
Log shipper	Filebeat	8.x	Lightweight log forwarding
Log processing	Logstash	8.x	Parsing, enrichment, routing
Log/event store	Elasticsearch	8.x (single-node)	Full-text search, analytics
Network monitor	Zeek	6.x	Passive traffic analysis, structured logs
Alt network monitor	Suricata	7.x	IDS/IPS with EVE JSON output
Packet analysis	Wireshark / tshark	4.x	Ad hoc deep inspection
Visualization	Grafana	10.x	Dashboards, alerting
Alerting	Alertmanager	0.27+	Alert routing, dedup, silencing
ML framework	scikit-learn	1.4+	Isolation Forest, Logistic Regression, TF-IDF
Forecasting	Prophet (via prophet)	1.1+	Time-series capacity forecasting
Data handling	pandas, numpy	latest	Feature engineering, data wrangling
ES client	elasticsearch-py	8.x	Query/write Elasticsearch from Python
Prometheus client	prometheus-client	0.20+	Push custom metrics from Python
Scheduling	cron or Apache Airflow	—	ML pipeline orchestration
Notebooks	JupyterLab	4.x	Model development, EDA
Traffic gen	hping3, nmap, stress-ng	—	Synthetic workload/traffic for testing

---

8. GitHub Repository Structure

ai-infra-observability/
├── README.md                          # Project overview, architecture, setup guide
├── LICENSE
├── docs/
│   ├── architecture.md                # This document
│   ├── vm-setup.md                    # VM provisioning instructions
│   ├── runbook.md                     # Operational procedures
│   └── screenshots/                   # Dashboard screenshots for portfolio
│
├── infrastructure/
│   ├── ansible/                       # (Optional) Automated VM provisioning
│   │   ├── inventory.yml
│   │   ├── playbook-prometheus.yml
│   │   ├── playbook-elk.yml
│   │   ├── playbook-zeek.yml
│   │   └── roles/
│   ├── prometheus/
│   │   ├── prometheus.yml             # Scrape config
│   │   ├── recording_rules.yml        # Pre-computed PromQL expressions
│   │   └── alerting_rules.yml         # Alert definitions
│   ├── logstash/
│   │   ├── pipelines.yml
│   │   └── conf.d/
│   │       ├── 01-filebeat-input.conf
│   │       ├── 02-syslog-filter.conf
│   │       └── 03-elasticsearch-output.conf
│   ├── filebeat/
│   │   └── filebeat.yml               # Per-node Filebeat config
│   ├── zeek/
│   │   ├── local.zeek                 # Zeek site policy
│   │   └── node.cfg                   # Zeek cluster config
│   └── grafana/
│       └── provisioning/
│           ├── dashboards/            # JSON dashboard definitions
│           └── datasources/           # Prometheus + ES datasource configs
│
├── ml/
│   ├── requirements.txt               # Python dependencies
│   ├── common/
│   │   ├── config.py                  # Shared configuration (ES hosts, Prometheus URL)
│   │   ├── es_client.py               # Elasticsearch helper functions
│   │   └── prom_client.py             # Prometheus query helper
│   ├── anomaly_detection/
│   │   ├── feature_engineering.py     # Zeek conn.log → feature vectors
│   │   ├── train.py                   # Isolation Forest training
│   │   ├── detect.py                  # Batch scoring script (cron target)
│   │   └── models/                    # Serialized model files (.joblib)
│   ├── capacity_forecast/
│   │   ├── extract_metrics.py         # Prometheus API data extraction
│   │   ├── train.py                   # Prophet model training
│   │   ├── forecast.py                # Generate forecasts + push to Pushgateway
│   │   └── models/
│   ├── log_classification/
│   │   ├── label_studio_export.py     # (Optional) export from labeling tool
│   │   ├── preprocess.py              # Text cleaning, TF-IDF fitting
│   │   ├── train.py                   # Logistic Regression training
│   │   ├── classify.py                # Batch classification (cron target)
│   │   └── models/
│   └── notebooks/
│       ├── 01_eda_system_metrics.ipynb
│       ├── 02_eda_zeek_connections.ipynb
│       ├── 03_log_labeling_analysis.ipynb
│       ├── 04_anomaly_model_tuning.ipynb
│       ├── 05_forecast_validation.ipynb
│       └── 06_classifier_evaluation.ipynb
│
├── scripts/
│   ├── stress_test.sh                 # End-to-end validation: simulated failures
│   ├── generate_traffic.sh            # Synthetic network traffic generation
│   ├── export_training_data.py        # Bulk export from ES for labeling
│   └── cron_setup.sh                  # Install cron jobs for ML pipelines
│
└── tests/
    ├── test_feature_engineering.py
    ├── test_log_preprocessing.py
    └── test_es_integration.py

---

9. Observability Dashboards

Dashboard 1: Platform Health Overview (Home)

Single-pane summary. Four stat panels at top: total anomalies (24h), critical logs (24h), nearest capacity exhaustion date, overall system health score (composite). Below: a row of sparklines per host showing CPU, RAM, disk. A log volume time series broken down by ML classification. A network anomaly timeline. This is the "glance and know" dashboard.

Dashboard 2: Capacity Forecasting

One row per metric (CPU, RAM, disk, network). Each row contains: a time series panel showing 30 days of historical data overlaid with the Prophet forecast line and shaded confidence intervals (80% and 95%). A gauge showing current utilization. A stat panel showing computed "days to exhaustion." Alert annotations appear on the time series when thresholds are crossed.

Dashboard 3: Network Anomaly Detection

Top row: anomaly count over time (bar chart, 1-hour buckets), top 10 anomalous source IPs (table), protocol distribution of anomalous connections (pie chart). Bottom row: a scatter plot of connection duration vs. bytes transferred, colored by anomaly score. A filterable table of raw anomaly events with columns for timestamp, source IP, dest IP, port, service, duration, bytes, and anomaly score. Clicking a row links to the Zeek log entry in Elasticsearch.

Dashboard 4: Log Intelligence

Top row: log volume by classification over time (stacked area chart — critical in red, warning in amber, informational in blue, noise in gray). A stat panel showing classifier confidence distribution. Middle row: a table of critical and warning logs with timestamp, host, service, message, ML confidence. Bottom row: low-confidence predictions flagged for human review. A "classification accuracy" panel updated from periodic manual spot-checks.

Dashboard 5: Per-Host Deep Dive

Variable selector for hostname. Shows all Node Exporter metrics for that host: CPU per-core, memory breakdown (used/buffered/cached/free), disk I/O operations and latency, network interface traffic, filesystem usage per mount. Includes log volume from that host and any anomalies associated with its IP.

---

10. Evaluation Metrics

Network Anomaly Detection (Isolation Forest)

Since this is unsupervised with no ground-truth labels, evaluation uses a pragmatic approach:

• Injection testing: Deliberately generate known-bad traffic (port scans via nmap, DNS exfiltration simulation, high-volume data transfers) and measure detection rate. Target: >90% of injected anomalies flagged.
• False positive rate: Manually review 100 random flagged anomalies per week. Track the ratio of true anomalies to false alarms. Target: FPR < 20% (acceptable for alerting with human-in-the-loop).
• Silhouette score: Measure cluster separation in the feature space to validate that the model is finding meaningful structure, not random noise.
• Contamination sensitivity analysis: Sweep contamination from 0.005 to 0.10 and plot precision/recall on the injected anomaly set to find the optimal operating point.

Capacity Forecasting (Prophet)

• MAPE (Mean Absolute Percentage Error): Primary metric. Compute on a held-out 7-day test window. Target: MAPE < 10% for CPU and memory, < 15% for disk I/O and network (inherently noisier signals).
• Coverage probability: Verify that the 95% confidence interval actually contains the true value ≥ 90% of the time. If coverage is significantly below 95%, the model is overconfident.
• Residual analysis: Plot residuals over time. Check for autocorrelation (Durbin-Watson test) — correlated residuals indicate the model is missing a pattern.
• Backtesting: Use Prophet's built-in cross-validation (cross_validation() with initial='21 days', period='7 days', horizon='7 days') to compute rolling MAPE and coverage.

Log Classification (TF-IDF + Logistic Regression)

• Per-class precision, recall, F1-score: Reported via classification_report. Critical class is the most important — target recall > 0.95 (never miss a critical log). Acceptable to trade precision on noise class.
• Macro-averaged F1: Overall model quality across all four classes. Target: > 0.85.
• Confusion matrix: Visualize where misclassifications occur. The most costly error is critical → noise (missed incident); least costly is noise → informational.
• Confidence calibration: Plot predicted probability vs. actual accuracy (reliability diagram). Use Platt scaling if the model is poorly calibrated.
• Drift monitoring: Track weekly F1 on new data. If F1 drops > 5% from baseline, trigger retraining. Log new vocabulary terms (unseen tokens) as an early indicator of distribution shift.
• Human-in-the-loop validation: Randomly sample 50 classified logs per week for manual review. Compute agreement rate between model and human labels. This is the ground-truth feedback loop.

Cross-Cutting Evaluation Practices

• All models are versioned with timestamps and stored as .joblib files in the models/ directory.
• Every training run logs hyperparameters, dataset size, and evaluation metrics to a training_log.csv for reproducibility.
• A monthly "model review" compares current model performance against the previous month to catch silent degradation.
• The stress test script (scripts/stress_test.sh) runs an end-to-end integration test: inject known anomalies, fill disk to 90%, flood logs — then verify that all three ML systems detect and surface the events within their next processing cycle.