RL Portfolio Optimization System

A complete, runnable reinforcement learning system for multi-asset portfolio optimization using PPO and SAC algorithms.

Overview

This system trains RL agents to dynamically allocate capital across multiple assets to maximize risk-adjusted returns. It includes:

• 2 RL Algorithms: PPO (Proximal Policy Optimization) and SAC (Soft Actor-Critic)
• Multi-Asset Universe: Equities, bonds, commodities, and crypto
• Full Training Loop: End-to-end training, evaluation, and visualization
• Performance Metrics: Returns, Sharpe ratio, drawdown, turnover

Quick Start

1. Train Both Agents

python3 train.py

This will:

• Generate synthetic market data (or load real data if available)
• Train PPO agent for 100 episodes
• Train SAC agent for 100 episodes
• Evaluate both on test data
• Compare against equal-weight benchmark
• Save models and generate performance plots

Expected Runtime: 3-5 minutes

Outputs:

• ppo_model.npz - Trained PPO model
• sac_model.npz - Trained SAC model
• ppo_performance.png - PPO visualization
• sac_performance.png - SAC visualization

2. Evaluate Trained Models

python3 evaluate.py

This loads saved models and:

• Runs evaluation on test data
• Exports portfolio weights to CSV
• Exports trade history to CSV

Outputs:

• ppo_weights.csv - Portfolio allocations over time
• sac_weights.csv - Portfolio allocations over time
• ppo_trades.csv - Trade-by-trade details
• sac_trades.csv - Trade-by-trade details

3. Generate Custom Data

python3 data_generator.py

Generates synthetic_prices.csv with correlated asset prices.

System Components

1. Data Generator (data_generator.py)

• Generates synthetic multi-asset price data
• Configurable correlations, returns, and volatility
• Default: 6 assets with realistic characteristics

2. Portfolio Environment (portfolio_env.py)

• Gym-style RL environment
• Observation: Recent returns, volatility, correlations, current weights
• Action: Target portfolio weights (continuous)
• Reward: Portfolio return - risk penalty - transaction costs
• Handles rebalancing and cost simulation

3. PPO Agent (ppo_agent.py)

• On-policy algorithm
• Gaussian policy with learnable std
• Value function for advantage estimation
• Clipped objective for stable updates

4. SAC Agent (sac_agent.py)

• Off-policy algorithm with replay buffer
• Twin Q-networks for value estimation
• Maximum entropy framework
• Soft target updates

5. Training Script (train.py)

• Complete training pipeline
• Train/test data split
• Performance comparison
• Visualization and logging

6. Evaluation Script (evaluate.py)

• Load trained models
• Evaluate on new data
• Export results to CSV

Asset Universe

Default assets with realistic parameters:

Asset	Type	Annual Return	Annual Vol	Description
EQUITY_1	Stock	8%	18%	High growth
EQUITY_2	Stock	6%	15%	Moderate
EQUITY_3	Stock	5%	12%	Low volatility
BOND	Fixed Income	3%	5%	Bonds
COMMODITY	Commodity	2%	22%	Commodities
CRYPTO	Crypto	15%	60%	High risk

Performance Metrics

The system tracks:

• Total Return: Cumulative portfolio return
• Sharpe Ratio: Risk-adjusted return (annualized)
• Max Drawdown: Largest peak-to-trough decline
• Average Turnover: Portfolio rebalancing frequency
• Final Portfolio Value: Ending capital

Training Configuration

PPO Hyperparameters

• Learning rate: 3e-4
• Discount factor (gamma): 0.99
• Clipping epsilon: 0.2
• Update frequency: Every 512 steps
• Training epochs: 10 per update
• Episodes: 100

SAC Hyperparameters

• Learning rate: 3e-4
• Discount factor (gamma): 0.99
• Soft update tau: 0.005
• Entropy temperature (alpha): 0.2
• Replay buffer: 10,000 transitions
• Update frequency: Every step
• Episodes: 100

Environment Parameters

• Initial capital: $100,000
• Transaction cost: 0.1% per trade
• Window size: 20 days
• Train/test split: 70/30

Customization

Using Real Data

Replace synthetic data with real prices:

# In train.py, replace:
price_data = generate_synthetic_data()

# With:
price_data = pd.read_csv('your_data.csv', index_col=0, parse_dates=True)

Data format: CSV with dates as index, asset prices as columns.

Adjusting Asset Universe

Modify data_generator.py to add/remove assets:

assets = ['SPY', 'TLT', 'GLD', 'BTC']  # Your assets
params = {
    'SPY': {'mu': 0.08, 'sigma': 0.15},
    'TLT': {'mu': 0.03, 'sigma': 0.08},
    # ... add your parameters
}

Tuning Reward Function

Edit _calculate_reward() in portfolio_env.py:

def _calculate_reward(self, portfolio_return, turnover, cost):
    # Customize reward components
    reward = portfolio_return  # Base return
    reward -= 0.5 * turnover   # Turnover penalty
    reward -= 2.0 * volatility # Risk penalty
    return reward * scaling

Hyperparameter Tuning

In train.py, adjust agent parameters:

agent = PPOAgent(
    lr=1e-4,        # Lower learning rate
    gamma=0.95,     # Different discount factor
    epsilon=0.1,    # Tighter clipping
)

Expected Results

After training, typical results:

PPO Agent:

• Total Return: 0-5%
• Sharpe Ratio: 0.1-0.3
• Max Drawdown: -15% to -20%

SAC Agent:

• Total Return: 10-15%
• Sharpe Ratio: 0.3-0.5
• Max Drawdown: -10% to -15%

Benchmark (Equal Weight):

• Total Return: 10-15%
• Sharpe Ratio: 0.4-0.5
• Max Drawdown: -15%

Note: Performance varies with random seed and market conditions

Limitations

This is a minimal viable system focused on runnability:

• Simplified RL implementations (not production-grade)
• Basic feature engineering
• No advanced risk constraints
• No transaction cost optimization
• Limited state representation

Next Steps for Improvement

1. Better State Representation

   • Add momentum features
   • Include regime indicators
   • Technical indicators
   • Macro variables

2. Advanced Risk Management

   • VaR constraints
   • Position limits
   • Leverage controls
   • Risk budgeting

3. Enhanced Training

   • Curriculum learning
   • Multi-task learning
   • Ensemble methods
   • Hyperparameter tuning

4. Real-World Features

   • Realistic slippage models
   • Market impact
   • Order execution
   • Live trading integration

Troubleshooting

Training is slow:

• Reduce n_episodes in train.py
• Increase update_freq for PPO
• Use smaller window_size

Poor performance:

• Adjust reward function penalties
• Tune learning rates
• Increase training episodes
• Check data quality

Memory issues:

• Reduce replay buffer size (SAC)
• Decrease window_size
• Use fewer assets

File Structure

.
├── data_generator.py      # Synthetic data generation
├── portfolio_env.py       # RL environment
├── ppo_agent.py          # PPO implementation
├── sac_agent.py          # SAC implementation
├── train.py              # Training script
├── evaluate.py           # Evaluation script
├── README.md             # This file
├── ppo_model.npz         # Saved PPO model
├── sac_model.npz         # Saved SAC model
├── ppo_performance.png   # PPO visualization
├── sac_performance.png   # SAC visualization
├── ppo_weights.csv       # PPO portfolio weights
├── sac_weights.csv       # SAC portfolio weights
├── ppo_trades.csv        # PPO trade history
└── sac_trades.csv        # SAC trade history

License

This is a learning/research implementation. Use at your own risk.

Acknowledgments

Built with:

• NumPy for numerical computing
• Pandas for data handling
• Matplotlib for visualization

RL algorithms based on:

• PPO: Schulman et al., 2017
• SAC: Haarnoja et al., 2018