Skip to content

Commit d6d8934

Browse files
aayostemaayostem
committed
adding tutorial
1 parent af92c61 commit d6d8934

9 files changed

Lines changed: 18513 additions & 612 deletions

File tree

‎docker-compose.yml‎

Lines changed: 0 additions & 417 deletions
Large diffs are not rendered by default.

‎infrastructure/Dockerfile.mlflow‎

Lines changed: 0 additions & 145 deletions
Original file line numberDiff line numberDiff line change
@@ -34,148 +34,3 @@ RUN chmod +x /app/start.sh
3434

3535
CMD ["/app/start.sh"]
3636

37-
38-
# FROM python:3.9-slim
39-
40-
# WORKDIR /app
41-
42-
# # Install system dependencies
43-
# RUN apt-get update && apt-get install -y \
44-
# curl \
45-
# && rm -rf /var/lib/apt/lists/*
46-
47-
# # Install MLflow with PostgreSQL support
48-
# RUN pip install mlflow psycopg2-binary
49-
50-
# # Create artifacts directory
51-
# RUN mkdir -p /artifacts
52-
53-
# EXPOSE 5000
54-
55-
# # Use PostgreSQL explicitly in the CMD
56-
# CMD ["mlflow", "server", \
57-
# "--backend-store-uri", "postgresql://mlflow:mlflow@postgres/mlflow", \
58-
# "--default-artifact-root", "/artifacts", \
59-
# "--host", "0.0.0.0", \
60-
# "--port", "5000"]
61-
62-
# # FROM python:3.9-slim
63-
64-
# # WORKDIR /app
65-
66-
# # # Install system dependencies
67-
# # RUN apt-get update && apt-get install -y \
68-
# # curl \
69-
# # && rm -rf /var/lib/apt/lists/*
70-
71-
# # # Install MLflow with PostgreSQL support
72-
# # RUN pip install mlflow psycopg2-binary
73-
74-
# # # Create artifacts directory
75-
# # RUN mkdir -p /artifacts
76-
77-
# # EXPOSE 5000
78-
79-
# # CMD ["mlflow", "server", \
80-
# # "--backend-store-uri", "postgresql://mlflow:mlflow@postgres/mlflow", \
81-
# # "--default-artifact-root", "/artifacts", \
82-
# # "--host", "0.0.0.0", \
83-
# # "--port", "5000"]
84-
85-
# # FROM python:3.9-slim
86-
87-
# # WORKDIR /app
88-
89-
# # # Install system dependencies
90-
# # RUN apt-get update && apt-get install -y \
91-
# # curl \
92-
# # sqlite3 \
93-
# # && rm -rf /var/lib/apt/lists/*
94-
95-
# # # Install MLflow
96-
# # RUN pip install mlflow psycopg2-binary # Add PostgreSQL support
97-
98-
# # # Create non-root user (optional, but good practice)
99-
# # RUN groupadd -r mlflow -g 1000 && \
100-
# # useradd -r -u 1000 -g mlflow mlflow
101-
102-
# # # Create directories and set permissions
103-
# # RUN mkdir -p /mlflow-data /artifacts && \
104-
# # chown -R mlflow:mlflow /mlflow-data /artifacts /app
105-
106-
# # USER mlflow
107-
108-
# # EXPOSE 5000
109-
110-
# # # Use environment variables for flexibility
111-
# # CMD ["mlflow", "server", \
112-
# # "--backend-store-uri", "${MLFLOW_BACKEND_STORE_URI:-sqlite:////mlflow-data/mlflow.db}", \
113-
# # "--default-artifact-root", "${MLFLOW_ARTIFACT_ROOT:-/artifacts}", \
114-
# # "--host", "0.0.0.0", \
115-
# # "--port", "5000"]
116-
117-
118-
# # # # # infrastructure/Dockerfile.mlflow
119-
# # # # FROM python:3.9-slim
120-
121-
# # # # WORKDIR /app
122-
123-
# # # # # Install system dependencies
124-
# # # # RUN apt-get update && apt-get install -y \
125-
# # # # curl \
126-
# # # # && rm -rf /var/lib/apt/lists/*
127-
128-
# # # # # Copy requirements
129-
# # # # COPY requirements-docker.txt .
130-
131-
# # # # # Install Python dependencies
132-
# # # # RUN pip install --no-cache-dir -r requirements-docker.txt
133-
134-
# # # # # Create directories for MLflow
135-
# # # # RUN mkdir -p /mlruns /artifacts
136-
137-
# # # # # Create non-root user
138-
# # # # RUN useradd --create-home --shell /bin/bash mlflow
139-
# # # # USER mlflow
140-
141-
# # # # # Health check
142-
# # # # HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
143-
# # # # CMD curl -f http://localhost:5000/health || exit 1
144-
145-
# # # # # Expose MLflow port
146-
# # # # EXPOSE 5000
147-
148-
# # # # # Start MLflow server
149-
# # # # CMD ["mlflow", "server", \
150-
# # # # "--host", "0.0.0.0", \
151-
# # # # "--port", "5000", \
152-
# # # # "--backend-store-uri", "sqlite:///mlruns/mlflow.db", \
153-
# # # # "--default-artifact-root", "file:///artifacts"]
154-
155-
156-
157-
# # # FROM python:3.9-slim
158-
159-
# # # WORKDIR /app
160-
161-
# # # # Install system dependencies
162-
# # # RUN apt-get update && apt-get install -y \
163-
# # # curl \
164-
# # # && rm -rf /var/lib/apt/lists/*
165-
166-
# # # # Install MLflow
167-
# # # RUN pip install mlflow
168-
169-
# # # # Create non-root user
170-
# # # RUN groupadd -r mlflow && useradd -r -g mlflow mlflow
171-
# # # RUN chown -R mlflow:mlflow /app
172-
173-
# # # USER mlflow
174-
175-
# # # EXPOSE 5000
176-
177-
# # # CMD ["mlflow", "server", \
178-
# # # "--backend-store-uri", "sqlite:////mlflow-data/mlflow.db", \
179-
# # # "--default-artifact-root", "/artifacts", \
180-
# # # "--host", "0.0.0.0", \
181-
# # # "--port", "5000"]

‎src/data/__pycache__/pipeline.cpython-313.pyc‎

5.12 KB
Binary file not shown.

‎src/mlops_drift_detection.egg-info/PKG-INFO‎

Lines changed: 108 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -34,49 +34,111 @@ Requires-Dist: seaborn>=0.12.0; extra == "notebooks"
3434

3535
# Enterprise-Grade MLOps Pipeline: Model Drift Detection System
3636

37-
38-
39-
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
40-
│ Data Sources │───▶│ Prefect Pipeline │───▶│ MLflow Tracking │
41-
│ │ │ │ │ │
42-
│ • S3/PostgreSQL │ │ • Data Validation │ │ • Experiments │
43-
│ • CSV/Parquet │ │ • Feature Engineering│ │ • Model Registry │
44-
└─────────────────┘ │ • Model Training │ │ • Artifact Store │
45-
│ • Evaluation │ └─────────────────┘
46-
┌─────────────────┐ └──────────────────┘ │
47-
│ Monitoring │ │ │
48-
│ │ │ ▼
49-
│ • Evidently AI │◀─────────────┘ ┌─────────────────┐
50-
│ • Grafana │ │ FastAPI Service │
51-
│ • Prometheus │ │ │
52-
└─────────────────┘ │ • REST API │
53-
│ │ • Swagger Docs │
54-
▼ │ • Health Checks │
55-
┌─────────────────┐ └─────────────────┘
56-
│ Alerting │ │
57-
│ │ ▼
58-
│ • Slack/Email │ ┌─────────────────┐
59-
│ • PagerDuty │ │ Frontend UI │
60-
└─────────────────┘ │ │
61-
│ • Model Dashboard │
62-
│ • Drift Visualize │
63-
└─────────────────┘
64-
65-
66-
67-
# Project structure
68-
mlops-drift-detection/
69-
├── docker-compose.yml
70-
├── infrastructure/
71-
│ ├── Dockerfile
72-
│ └── nginx.conf
73-
├── src/
74-
│ ├── data/
75-
│ ├── features/
76-
│ ├── models/
77-
│ ├── monitoring/
78-
│ └── api/
79-
├── tests/
80-
├── notebooks/
81-
├── requirements.txt
82-
└── README.md
37+
mlops-platform/
38+
├── 📁 infrastructure/
39+
│ ├── 📁 terraform/
40+
│ │ ├── 📄 main.tf # Primary infrastructure configuration
41+
│ │ ├── 📄 variables.tf # Terraform variables
42+
│ │ ├── 📄 outputs.tf # Terraform outputs
43+
│ │ └── 📄 providers.tf # Cloud provider configurations
44+
│ ├── 📁 kubernetes/
45+
│ │ ├── 📄 cluster-config.yaml # K8s cluster configuration
46+
│ │ ├── 📄 networking.yaml # Network policies & service mesh
47+
│ │ └── 📄 storage.yaml # Persistent volume claims
48+
│ └── 📁 helm-charts/
49+
│ ├── 📁 mlops-monitoring/ # Custom monitoring stack
50+
│ ├── 📁 mlops-pipelines/ # ML pipeline components
51+
│ └── 📁 mlops-remediation/ # Auto-remediation services
52+
├── 📁 src/
53+
│ ├── 📁 api/
54+
│ │ ├── 📄 app.py # FastAPI main application
55+
│ │ ├── 📄 routes/
56+
│ │ │ ├── 📄 monitoring.py # Monitoring endpoints
57+
│ │ │ ├── 📄 remediation.py # Remediation endpoints
58+
│ │ │ └── 📄 analysis.py # RCA endpoints
59+
│ │ └── 📄 middleware/
60+
│ │ ├── 📄 auth.py # Authentication middleware
61+
│ │ └── 📄 logging.py # Request logging
62+
│ ├── 📁 core/
63+
│ │ ├── 📄 config.py # Application configuration
64+
│ │ ├── 📄 database.py # Database connections
65+
│ │ └── 📄 security.py # Security utilities
66+
│ ├── 📁 services/
67+
│ │ ├── 📁 monitoring/
68+
│ │ │ ├── 📄 anomaly_detector.py # Multivariate anomaly detection
69+
│ │ │ ├── 📄 drift_detector.py # Data & concept drift detection
70+
│ │ │ └── 📄 performance_tracker.py # Model performance tracking
71+
│ │ ├── 📁 remediation/
72+
│ │ │ ├── 📄 engine.py # Remediation execution engine
73+
│ │ │ ├── 📄 actions.py # Available remediation actions
74+
│ │ │ └── 📄 safety_controller.py # Safety checks & rollback
75+
│ │ ├── 📁 analysis/
76+
│ │ │ ├── 📄 rca_engine.py # Root cause analysis engine
77+
│ │ │ ├── 📄 correlation_analyzer.py # Signal correlation
78+
│ │ │ └── 📄 pattern_detector.py # Temporal pattern detection
79+
│ │ └── 📁 alerting/
80+
│ │ ├── 📄 manager.py # Alert management & routing
81+
│ │ ├── 📄 notifier.py # Multi-channel notifications
82+
│ │ └── 📄 prioritizer.py # Intelligent priority assignment
83+
│ ├── 📁 models/
84+
│ │ ├── 📄 schemas.py # Pydantic models for data validation
85+
│ │ └── 📄 database_models.py # SQLAlchemy database models
86+
│ ├── 📁 utils/
87+
│ │ ├── 📄 logging_utils.py # Logging configuration
88+
│ │ ├── 📄 metrics_utils.py # Metrics collection utilities
89+
│ │ └── 📄 data_processor.py # Data processing helpers
90+
│ └── 📁 ml/
91+
│ ├── 📁 pipelines/
92+
│ │ ├── 📄 training_pipeline.py # End-to-end training pipeline
93+
│ │ ├── 📄 inference_pipeline.py # Real-time inference pipeline
94+
│ │ └── 📄 preprocessing.py # Data preprocessing
95+
│ ├── 📁 feature_store/
96+
│ │ ├── 📄 manager.py # Feature store management
97+
│ │ └── 📄 online_store.py # Online feature serving
98+
│ └── 📁 model_registry/
99+
│ ├── 📄 manager.py # Model version management
100+
│ └── 📄 deployment.py # Model deployment orchestration
101+
├── 📁 tests/
102+
│ ├── 📁 unit/
103+
│ │ ├── 📄 test_monitoring.py # Monitoring service tests
104+
│ │ ├── 📄 test_remediation.py # Remediation engine tests
105+
│ │ └── 📄 test_analysis.py # RCA engine tests
106+
│ ├── 📁 integration/
107+
│ │ ├── 📄 test_api_endpoints.py # API integration tests
108+
│ │ └── 📄 test_workflows.py # End-to-end workflow tests
109+
│ ├── 📁 performance/
110+
│ │ └── 📄 load_testing.py # Performance & load testing
111+
│ └── 📁 fixtures/
112+
│ └── 📄 test_data.py # Test data generators
113+
├── 📁 deployments/
114+
│ ├── 📄 Dockerfile # Application Dockerfile
115+
│ ├── 📄 docker-compose.yml # Local development setup
116+
│ ├── 📄 kubernetes.yaml # K8s deployment manifests
117+
│ └── 📄 helm-values.yaml # Helm configuration values
118+
├── 📁 docs/
119+
│ ├── 📄 architecture.md # System architecture
120+
│ ├── 📄 api_reference.md # API documentation
121+
│ ├── 📄 deployment_guide.md # Deployment instructions
122+
│ └── 📄 troubleshooting.md # Troubleshooting guide
123+
├── 📁 scripts/
124+
│ ├── 📄 setup_environment.sh # Environment setup script
125+
│ ├── 📄 deploy_cluster.sh # Cluster deployment script
126+
│ ├── 📄 run_migrations.sh # Database migration script
127+
│ └── 📄 backup_restore.sh # Backup & restore utilities
128+
├── 📁 monitoring/
129+
│ ├── 📄 prometheus.yml # Prometheus configuration
130+
│ ├── 📄 alert_rules.yml # Alerting rules
131+
│ ├── 📄 grafana_dashboards/ # Grafana dashboard JSON files
132+
│ └── 📄 custom_metrics/ # Custom metric definitions
133+
├── 📁 database/
134+
│ ├── 📄 migrations/ # Database migration scripts
135+
│ ├── 📄 schemas/ # Database schema definitions
136+
│ └── 📄 seeds/ # Initial data seeds
137+
├── 📄 requirements.txt # Python dependencies
138+
├── 📄 requirements-dev.txt # Development dependencies
139+
├── 📄 Makefile # Build & deployment commands
140+
├── 📄 .env.example # Environment variables template
141+
├── 📄 .gitignore # Git ignore rules
142+
├── 📄 README.md # Project overview & setup
143+
├── 📄 LICENSE # Project license
144+
└── 📄 pyproject.toml # Python project configuration

‎src/mlops_drift_detection.egg-info/SOURCES.txt‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,22 @@
11
README.md
22
pyproject.toml
3+
src/api/config.py
4+
src/api/main.py
35
src/config/settings.py
46
src/data/pipeline.py
7+
src/explainability/shap_explainer.py
8+
src/fairness/bias_detector.py
9+
src/feature_store/service.py
510
src/mlops_drift_detection.egg-info/PKG-INFO
611
src/mlops_drift_detection.egg-info/SOURCES.txt
712
src/mlops_drift_detection.egg-info/dependency_links.txt
813
src/mlops_drift_detection.egg-info/entry_points.txt
914
src/mlops_drift_detection.egg-info/requires.txt
1015
src/mlops_drift_detection.egg-info/top_level.txt
11-
src/model/training.py
12-
src/pipeline/training_pipeline.py
16+
src/models/training.py
17+
src/monitoring/advanced_drift_detector.py
18+
src/monitoring/dashboard.py
19+
src/monitoring/drift.py
20+
src/monitoring/drift/drift_service.py
21+
src/optimization/resource_optimizer.py
22+
src/pipelines/training_pipeline.py

‎src/mlops_drift_detection.egg-info/top_level.txt‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1+
api
12
config
23
data
3-
model
4-
pipeline
4+
explainability
5+
fairness
6+
feature_store
7+
models
8+
monitoring
9+
optimization
10+
pipelines

‎src/pipelines/training_pipeline.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33
from data.pipeline import data_pipeline_flow
44
from models.training import train_and_evaluate_models, register_model_in_registry
55
from monitoring.drift import setup_drift_monitoring
6+
7+
# from prefect.blocks.system import JSON
8+
from prefect.blocks.system import Secret
69
import mlflow
710

811

0 commit comments

Comments
 (0)