Skip to content

DOCKER_README.md

Latest commit

 

History

History
336 lines (248 loc) · 8.52 KB

DOCKER_README.md

File metadata and controls

336 lines (248 loc) · 8.52 KB
Raw

🚀 Advanced MLOps Platform - Docker Deployment

Developed by JALENDAR REDDY
© 2025 JALENDAR REDDY - Advanced MLOps Solutions

🐳 Complete Docker Deployment Guide

This MLOps platform is designed to run entirely in Docker containers, providing a complete isolated environment with all dependencies and services.

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                  Docker Container                       │
│  ┌─────────────────┐    ┌─────────────────────────────┐  │
│  │   Streamlit     │    │        MLflow UI            │  │
│  │   Port: 8501    │    │       Port: 5001            │  │
│  │                 │    │                             │  │
│  │ • Authentication│    │ • Experiment Tracking      │  │
│  │ • 20+ Datasets  │    │ • Model Registry           │  │
│  │ • 15+ Algorithms│    │ • Metrics Visualization    │  │
│  │ • Real-time Logs│    │ • Model Comparison         │  │
│  │ • Visualizations│    │                             │  │
│  └─────────────────┘    └─────────────────────────────┘  │
│                                                         │
│  ┌─────────────────────────────────────────────────────┐  │
│  │              ML Training Engine                     │  │
│  │                                                     │  │
│  │ • Unique ID Tracking    • Comprehensive Logging    │  │
│  │ • Error Handling        • Metrics Calculation      │  │
│  │ • Model Persistence     • Result Management        │  │
│  └─────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

📋 Prerequisites

  • Docker: Version 20.10+
  • Docker Compose: Version 2.0+
  • System Requirements:
    • RAM: 4GB minimum, 8GB recommended
    • Disk: 2GB free space
    • CPU: 2 cores minimum

🚀 Quick Start (One-Command Deployment)

# Clone or navigate to the project directory
cd docker_mlops

# Run the automated deployment script
./deploy.sh

The deployment script will:

  1. ✅ Check Docker installation
  2. 🧹 Clean up existing containers
  3. 📁 Create necessary directories
  4. 🏗️ Build the Docker image
  5. 🚀 Start all services
  6. 🔍 Health check services
  7. 🌐 Provide access URLs

🛠️ Manual Deployment

If you prefer manual control:

# 1. Build the Docker image
docker-compose build

# 2. Start the services
docker-compose up -d

# 3. View logs
docker-compose logs -f

🌐 Access URLs

Once deployed, access the platform at:

🔐 Authentication

Default login credentials:

  • Username: user
  • Password: password

📁 Data Persistence

All data is persisted outside the container:

docker_mlops/
├── uploads/        # Uploaded datasets
├── models/         # Trained models
├── mlruns/         # MLflow experiment data
└── logs/           # Application logs

🎯 Platform Features

📊 Datasets (20+ Available)

  • Toy Datasets: Iris, Diabetes, Wine, Breast Cancer, Digits, Linnerud
  • Real-World: California Housing, Olivetti Faces, Forest Cover Type
  • Generated: Regression samples, Classification samples, Blobs, Circles, Moons

🤖 Machine Learning Algorithms (15+)

Regression:

  • Linear Regression, Ridge, Lasso
  • Random Forest, Gradient Boosting
  • Support Vector Regression
  • K-Neighbors, Decision Tree

Classification:

  • Logistic Regression, Random Forest
  • Gradient Boosting, SVM
  • Naive Bayes, K-Neighbors, Decision Tree

🔧 Advanced Features

  • ✅ Unique ID Tracking: Every training run gets a unique identifier
  • ✅ Real-time Logging: Comprehensive training logs with timestamps
  • ✅ Error Handling: Robust error handling and reporting
  • ✅ Interactive Visualizations: Plotly-based charts and graphs
  • ✅ Model Export: Download trained models and deployment packages
  • ✅ MLflow Integration: Full experiment tracking and model registry

🛡️ Security Features

  • User Authentication: Login system with session management
  • Container Isolation: All services run in isolated containers
  • Non-root User: Application runs as non-privileged user
  • Health Checks: Automated service health monitoring

📊 Monitoring & Management

View Real-time Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f ml-trainer

Container Management

# Restart services
docker-compose restart

# Stop services
docker-compose down

# Check status
docker-compose ps

Resource Usage

# Check container stats
docker stats mlops_platform_by_jalendar_reddy

🔧 Configuration

Environment Variables

The platform supports configuration via environment variables:

environment:
  - MLFLOW_TRACKING_URI=http://localhost:5001
  - STREAMLIT_SERVER_PORT=8501
  - STREAMLIT_SERVER_ADDRESS=0.0.0.0

Port Configuration

To change ports, modify docker-compose.yml:

ports:
  - "8502:8501"  # Change Streamlit to port 8502
  - "5002:5001"  # Change MLflow to port 5002

🗑️ Cleanup

Stop and Remove Everything

# Stop containers and remove volumes
docker-compose down -v

# Remove images as well
docker-compose down -v --rmi all

# Remove persistent data (CAUTION: This deletes all models and experiments)
rm -rf uploads/ models/ mlruns/ logs/

🚨 Troubleshooting

Common Issues

1. Port Already in Use

# Check what's using the port
lsof -i :8501
lsof -i :5001

# Kill the process or change ports in docker-compose.yml

2. Container Won't Start

# Check logs
docker-compose logs ml-trainer

# Rebuild image
docker-compose build --no-cache

3. Permission Issues

# Fix permissions
sudo chown -R $USER:$USER uploads/ models/ mlruns/ logs/

4. Out of Memory

# Check Docker memory limit
docker system info | grep -i memory

# Increase Docker memory allocation in Docker Desktop settings

Health Checks

The platform includes automated health checks:

# Check health status
docker inspect mlops_platform_by_jalendar_reddy | grep -i health

🔄 Updates and Maintenance

Update the Platform

# Pull latest changes
git pull

# Rebuild and restart
docker-compose down
docker-compose build --no-cache
docker-compose up -d

Backup Data

# Create backup
tar -czf mlops_backup_$(date +%Y%m%d).tar.gz uploads/ models/ mlruns/

# Restore backup
tar -xzf mlops_backup_YYYYMMDD.tar.gz

📈 Performance Optimization

Resource Limits

Add resource limits to docker-compose.yml:

deploy:
  resources:
    limits:
      cpus: '2.0'
      memory: 4G
    reservations:
      cpus: '1.0'
      memory: 2G

Volume Optimization

For better performance with large datasets:

volumes:
  - ./uploads:/app/uploads:cached
  - ./models:/app/models:cached
  - ./mlruns:/app/mlruns:cached

🌟 Advanced Usage

Custom Datasets

  1. Place CSV files in the uploads/ directory
  2. Use the file upload feature in the Streamlit interface
  3. Or mount additional volumes:
volumes:
  - /path/to/your/data:/app/data:ro

Model Deployment

Export trained models and create deployment packages:

  1. Train models through the web interface
  2. Use the "Model Export" tab
  3. Download the generated deployment package
  4. Deploy using the included Docker files

📞 Support

For issues, questions, or contributions:

  • Developer: JALENDAR REDDY
  • Platform: Advanced MLOps Solutions
  • Year: 2025

📄 License

© 2025 JALENDAR REDDY - All Rights Reserved

🚀 Happy Machine Learning! 🚀