Skip to content

render_production_setup.md

Latest commit

 

History

History
268 lines (217 loc) · 7.84 KB

render_production_setup.md

File metadata and controls

268 lines (217 loc) · 7.84 KB
Raw

🌐 Render Production Deployment with Managed Kafka

🎯 Production Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Render Web    │───▶│  Managed Kafka   │───▶│   Your Trading  │
│   Service       │    │   (Upstash/      │    │   Application   │
│                 │    │   CloudKarafka)  │    │                 │
│ • Auto-scaling  │    │                  │    │ • Same code as  │
│ • Load balancer │    │ • High availability│    │   local         │
│ • SSL/HTTPS     │    │ • Auto-scaling   │    │ • Auto-restart  │
│ • Git deploy    │    │ • Monitoring     │    │ • Environment   │
└─────────────────┘    └──────────────────┘    └─────────────────┘

📋 Step-by-Step Production Setup

Step 1: Choose Managed Kafka Service

Option A: Upstash (Recommended - FREE tier)

# 1. Go to https://upstash.com
# 2. Sign up with GitHub/Google
# 3. Create Kafka cluster (select region close to your users)
# 4. Copy connection details:

Bootstrap Server: tops-stingray-12345-us1-kafka.upstash.io:9092
Username: dG9wcy1zdGluZ3JheS0xMjM0NSQ...
Password: your-secure-password
SASL Mechanism: SCRAM-SHA-256
Security Protocol: SASL_SSL

Option B: CloudKarafka (Alternative)

# 1. Go to https://cloudkarafka.com
# 2. Create account
# 3. Create instance (FREE tier: 25MB storage)
# 4. Get connection URL from console

Step 2: Configure Render Environment Variables

In your Render service dashboard → Environment tab:

# Kafka Configuration (Required)
KAFKA_BOOTSTRAP_SERVERS=tops-stingray-12345-us1-kafka.upstash.io:9092
KAFKA_SASL_USERNAME=dG9wcy1zdGluZ3JheS0xMjM0NSQ...
KAFKA_SASL_PASSWORD=your-secure-password
KAFKA_SASL_MECHANISM=SCRAM-SHA-256
KAFKA_SECURITY_PROTOCOL=SASL_SSL

# Application Settings
KAFKA_CLIENT_ID=trading_app_prod
ENVIRONMENT=production

# Database & Other Services
DATABASE_URL=[Auto-generated by Render]
REDIS_URL=[If using Redis addon]

# Trading API Keys
UPSTOX_API_KEY=your-api-key
UPSTOX_API_SECRET=your-api-secret
# ... other broker APIs

# Security
JWT_SECRET_KEY=your-production-secret
DEBUG=false
LOG_LEVEL=INFO

Step 3: Deploy Application

Method 1: GitHub Integration (Recommended)

# 1. Push your code to GitHub
git add .
git commit -m "Production deployment ready"
git push origin main

# 2. In Render dashboard:
# - Connect to your GitHub repository
# - Select branch: main
# - Build command: pip install -r requirements.txt
# - Start command: python app.py

# 3. Render automatically deploys on every push

Method 2: Manual Deploy

# 1. Install Render CLI
npm install -g @render/cli

# 2. Deploy from local machine
render deploy

Step 4: Verify Deployment

Check Application Logs

# In Render dashboard → Logs tab, you should see:
✅ Kafka config loaded: tops-stingray-12345-us1-kafka.upstash.io:9092
✅ Created topic: trading.market_data.raw
✅ Created topic: trading.market_data.processed
✅ Kafka producer initialized
✅ Started consumer: market_data_processor
🚀 Trading application started successfully

Test Kafka Connection

# Add this health check endpoint to your app.py:

from services.simple_kafka_system import get_kafka_system

@app.get("/health/kafka")
async def kafka_health():
    kafka_system = get_kafka_system()
    health = await kafka_system.health_check()
    return health

# Visit: https://your-app.onrender.com/health/kafka
# Should return:
{
  "kafka_connected": true,
  "producer_ready": true,
  "active_consumers": 3,
  "system_running": true,
  "topics_available": 8
}

📊 Production Monitoring

Kafka Service Monitoring

  • Upstash Console: View message throughput, consumer lag, error rates
  • CloudKarafka Dashboard: Monitor cluster health, topic statistics

Render Application Monitoring

  • Logs: Real-time application logs
  • Metrics: CPU, memory, response time
  • Alerts: Set up notifications for errors/downtime

Application Health Checks

# Add comprehensive health check
@app.get("/health")
async def full_health_check():
    kafka_system = get_kafka_system()
    
    health = {
        "status": "healthy",
        "timestamp": int(time.time() * 1000),
        "kafka": await kafka_system.health_check(),
        "database": check_database_connection(),
        "redis": check_redis_connection() if redis_enabled else "disabled",
        "websocket": check_websocket_connections()
    }
    
    # Overall status
    if not health["kafka"]["kafka_connected"]:
        health["status"] = "unhealthy"
    
    return health

🔧 Production Optimization

Render Service Configuration

# render.yaml (optional)
services:
  - type: web
    name: trading-app
    env: python
    plan: starter  # or professional for higher resources
    region: oregon  # choose region close to your users
    buildCommand: "pip install -r requirements.txt"
    startCommand: "python app.py"
    healthCheckPath: "/health"
    
    envVars:
      - key: KAFKA_BOOTSTRAP_SERVERS
        value: tops-stingray-12345-us1-kafka.upstash.io:9092
      - key: KAFKA_SASL_USERNAME
        value: your-username
      # ... other env vars

Kafka Topic Optimization for Production

# In simple_kafka_config.py, adjust for production load:

class TradingKafkaTopics:
    def __init__(self):
        # Increase partitions for production load
        self.topics = [
            SimpleTopic("trading.market_data.raw", partitions=3, replication_factor=3),
            SimpleTopic("trading.market_data.processed", partitions=6, replication_factor=3),
            SimpleTopic("trading.signals.breakout", partitions=4, replication_factor=2),
            SimpleTopic("trading.ui.price_updates", partitions=8, replication_factor=2),
            # ... other topics
        ]

💰 Production Costs

Upstash Kafka Pricing

  • Free Tier: 10,000 messages/day
  • Pay-as-you-go: $0.2 per 100K messages
  • Pro Plan: $20/month for 10M messages

Render Pricing

  • Starter Plan: $7/month (512MB RAM, 0.5 CPU)
  • Professional Plan: $25/month (2GB RAM, 1 CPU)

Total Estimated Monthly Cost

  • Development/Small App: $7 (Render) + $0 (Upstash free) = $7/month
  • Production App: $25 (Render) + $20 (Upstash) = $45/month

⚠️ Production Troubleshooting

Common Issues & Solutions

Issue: "Kafka connection timeout"

# Check environment variables in Render dashboard
# Verify managed Kafka service is running
# Check service region proximity

Issue: "SASL authentication failed"

# Verify username/password from Kafka provider
# Check SASL mechanism (usually SCRAM-SHA-256)
# Ensure Security Protocol is SASL_SSL

Issue: "Application crashes on startup"

# Check Render logs for specific error
# Verify all environment variables are set
# Test Kafka connection using health endpoint

Issue: "High latency/slow performance"

# Choose Kafka region close to Render region
# Increase Render plan (more CPU/RAM)
# Optimize Kafka consumer batch sizes

🚀 Deployment Checklist

  • Created managed Kafka service (Upstash/CloudKarafka)
  • Configured Render environment variables
  • Set up GitHub integration for auto-deploy
  • Added health check endpoints
  • Configured monitoring/alerts
  • Tested application functionality
  • Verified Kafka message flow
  • Set up backup/disaster recovery plan

Your production deployment is now ready! 🎉