Modernize Value at Risk solution to 2025 standards#5
Open
calreynolds wants to merge 11 commits into
Open
Conversation
Collaborator
calreynolds
commented
Jul 15, 2025
calreynolds
commented
- Add DAB (Databricks Asset Bundle) structure with databricks.yml
- Integrate Unity Catalog for data governance
- Update to MLflow 2.8+ for experiment tracking
- Add modern CI/CD pipeline with GitHub Actions
- Remove hardcoded DBR 10.4ML runtime dependency
- Update dependencies to latest compatible versions
- Add deployment and cleanup scripts
- Enhance README with modern setup instructions
- Add DAB (Databricks Asset Bundle) structure with databricks.yml - Integrate Unity Catalog for data governance - Update to MLflow 2.8+ for experiment tracking - Add modern CI/CD pipeline with GitHub Actions - Remove hardcoded DBR 10.4ML runtime dependency - Update dependencies to latest compatible versions - Add deployment and cleanup scripts - Enhance README with modern setup instructions
- Remove leftover code and improve configuration robustness - Make warehouse configuration flexible (no hardcoded warehouse names) - Add comprehensive error handling and logging to deployment scripts - Implement industry-standard CLI patterns with help, verbose, dry-run modes - Add production environment safeguards and confirmation prompts - Update configuration files to use modern Unity Catalog patterns - Provide detailed environment configuration guidance in env.example - Improve GitHub Actions workflow to handle different environments gracefully
Key improvements to 01_var_market_etl.py: - Replace deprecated pandas_udf syntax with modern patterns - Add comprehensive error handling and logging - Implement Unity Catalog storage with audit columns - Add data quality validation and monitoring - Use modern visualization with enhanced plotting Key improvements to 02_var_model.py: - Integrate MLflow 2.8+ with Unity Catalog model registry - Add modern artifact management with proper cleanup - Implement comprehensive feature correlation analysis - Use type hints and modern Python patterns - Add structured logging throughout Key improvements to 03_var_monte_carlo.py: - Modernize Monte Carlo simulation with distributed computing - Add configuration validation and error handling - Integrate tempo library for time series operations - Use Unity Catalog for volatility storage - Implement comprehensive logging and monitoring All scripts now follow modern industry standards: - Unity Catalog integration for data governance - Comprehensive error handling and logging - Modern Python type hints and documentation - MLflow 2.8+ integration for experiment tracking - Proper resource management and cleanup
- Fully modernized 04_var_aggregation.py with enterprise-grade risk aggregation - Added multiple confidence levels support and comprehensive error handling - Implemented modern MLflow integration for risk metrics tracking - Complete rewrite of 05_var_compliance.py for Basel Committee compliance - Added Basel III traffic light system (Green/Yellow/Red zones) - Implemented comprehensive backtesting with 250-day windows - Added modern time series joins using tempo library - Created compliance visualization dashboard with modern matplotlib - Added automated compliance reporting and recommendations - Enhanced all notebooks with Unity Catalog patterns and modern Python practices
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Explicitly set mlflow.set_registry_uri("databricks-uc")
- Set both tracking and registry URIs as recommended
- Tested bundle validation, deployment, and workflow execution
- Fixed the CONFIG_NOT_AVAILABLE error for spark.mlflow.modelRegistryUri
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed MLflow configuration in both 00_var_context.py and configure_notebook.py - Removed invalid spark.mlflow.modelRegistryUri configuration - Used environment variables for MLflow registry configuration - Fixed missing environment parameter in all DAB bundle tasks - Simplified MLflow experiment naming to avoid directory issues - Tested and verified first task runs successfully 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.