A comprehensive Python terminal dashboard for email migration, backup, restore, and synchronization with a modern, interactive interface built with Textual.
- Interactive Terminal Dashboard - Real-time operation monitoring with Textual framework supporting migrate, backup, restore, and sync modes
- Email Migration - Migrate emails between IMAP servers with folder structure preservation
- Domain Batch Processing - Process multiple accounts grouped by domain in parallel
- Backup & Restore - Export to Maildir++ format and restore with full metadata preservation
- Synchronization - One-way and incremental sync strategies between accounts
- Checkpoint & Resume - Automatic state persistence with resume capability after interruptions
- Multi-Source Credential Provider - Load credentials from environment variables, encrypted files, or interactive prompts
- AES-256 Encryption - Secure credential storage using Fernet encryption
- Windows Credential Manager - Integration with Windows Credential Manager via keyring
- OAuth2 Authentication - Support for Gmail, Outlook, and Yahoo OAuth2 authentication
- Discord Notifications - Webhook notifications for account completion, batch completion, and errors
- Rate Limit Detection - Automatic detection and handling of IMAP rate limits
- Retry Logic - Configurable exponential backoff retry for transient errors
- Error Classification - Intelligent error classification (transient, permanent, rate_limit)
- YAML Configuration - Comprehensive YAML-based configuration with validation
- Folder Mapping Presets - Built-in presets for Gmail, Outlook, and Yahoo folder mappings
- Command-Line Overrides - Override configuration settings via CLI arguments
- Cross-Platform Support - Full Windows and Linux compatibility
- Python 3.9 or higher
- pip
pip install -r requirements.txtpip install -e .-
Create a configuration file (see
example_config.yaml):cp example_config.yaml ~/.shadows-mail-tools/config.yaml # Edit with your account details
-
Launch the dashboard:
python -m shadows_mail_tools.cli dashboard --config ~/.shadows-mail-tools/config.yamlOr use the default config location:
# Place config at ~/.shadows-mail-tools/config.yaml python -m shadows_mail_tools.cli dashboard -
Select operation mode from the dropdown (Migrate, Backup, Restore, or Sync)
-
Configure and start your operation
-
Monitor progress with interactive controls:
p- Pause/Resumes- Stop gracefullyr- Retry failed accountsd- Toggle domain/account viewq- Quit
The dashboard requires a configuration file to function. Without it, you'll see a message to load configuration.
Create your config file at ~/.shadows-mail-tools/config.yaml or specify a custom path with --config.
See example_config.yaml for a complete example with all available options.
For the dashboard to work, you need at least:
settings:
concurrency: 3
accounts:
- name: "account1"
source_host: "imap.source.com"
source_port: 993
source_username: "user@source.com"
source_password: "password"
source_use_ssl: true
dest_host: "imap.dest.com"
dest_port: 993
dest_username: "user@dest.com"
dest_password: "password"
dest_use_ssl: true
domain: "source.com"settings:
concurrency: 3 # Number of domain batches to process in parallel
checkpoint_interval: 30 # Checkpoint save interval in secondsaccounts:
- name: "account1"
source_host: "imap.source.com"
source_port: 993
source_username: "user@source.com"
source_password: "password" # Or use credential references
source_use_ssl: true
dest_host: "imap.dest.com"
dest_port: 993
dest_username: "user@dest.com"
dest_password: "password"
dest_use_ssl: truenotifications:
discord_webhook_url: "https://discord.com/api/webhooks/your-webhook-url"folder_mappings:
"INBOX.Sent": "Sent"
"INBOX.Trash": "Trash"Or use presets for Gmail, Outlook, or Yahoo.
IMPORTANT: The dashboard requires a configuration file to enable all operation modes.
Launch the dashboard with a configuration file:
python -m shadows_mail_tools.cli dashboard --config config.yamlOr use the default config location:
# Place config at ~/.shadows-mail-tools/config.yaml
python -m shadows_mail_tools.cli dashboardWithout a config file, the dashboard will show limited functionality with a message to load configuration.
With a config file, you'll see:
- Operation Mode selector (Migrate, Backup, Restore, Sync)
- Mode-specific configuration forms
- Domain batch view
- Real-time progress tracking
- Full interactive controls
Once the dashboard is running with a config file, use the Operation Mode dropdown to select your desired operation:
- Select "Migrate" from the Operation Mode dropdown
- The dashboard will use accounts from your config file
- Click "Start Migration" to begin
- Monitor progress in Domain Batch View or Account View (toggle with 'd' key)
What it does:
- Migrates emails between source and destination accounts
- Uses accounts from configuration file
- Processes accounts in domain batches
- Shows domain batch progress and per-account status
- Select "Backup" from the Operation Mode dropdown
- Enter the backup directory path (e.g.,
/home/user/email-backupsorC:\Users\User\email-backups) - Click "Start Backup" to export emails to Maildir++ format
- Monitor files being written in the activity log
What it does:
- Exports emails to Maildir++ format
- Backs up all configured accounts
- Preserves folder structure and IMAP flags
- Creates organized directory structure per account
- Select "Restore" from the Operation Mode dropdown
- Enter the backup directory path containing Maildir++ files
- The dashboard will use destination accounts from your config
- Click "Start Restore" to import emails back to IMAP
- Monitor folder recreation and message uploads
What it does:
- Imports emails from Maildir++ format
- Restores to configured destination accounts
- Recreates folder structure and restores flags
- Preserves original message metadata
- Select "Sync" from the Operation Mode dropdown
- Choose sync strategy (One-way or Incremental)
- Click "Start Sync" to synchronize accounts
- Monitor only new messages being processed (for incremental)
What it does:
- Synchronizes emails between accounts
- Supports One-Way (source → destination) or Incremental (only new messages) strategies
- Syncs all configured accounts
- Tracks sync state to prevent duplicates
Dashboard Controls:
p- Pause/Resume operations- Stop operation gracefullyr- Retry failed accounts (migrate mode)d- Toggle between domain batch view and account viewq- Quit application
Dashboard Features:
- Real-time progress tracking with mode-appropriate metrics
- Domain batch view showing progress per domain
- Account-level status view
- Live operation log with timestamps
- Statistics panel (total, completed, failed, remaining, throughput)
- Discord notifications for operation completion and errors
Migrate emails between accounts:
python -m shadows_mail_tools.cli migrate --config config.yamlOptions:
--config, -c- Path to configuration file--concurrency- Override concurrency setting--checkpoint-interval- Override checkpoint interval--resume- Resume from last checkpoint--migration-id- Migration ID for checkpoint resume
Example:
python -m shadows_mail_tools.cli migrate \
--config config.yaml \
--concurrency 5 \
--checkpoint-interval 60Backup emails to Maildir++ format:
python -m shadows_mail_tools.cli backup \
--config config.yaml \
--output-dir /path/to/backup \
--account account1 # Optional: backup specific accountRestore emails from Maildir++ format:
python -m shadows_mail_tools.cli restore \
--config config.yaml \
--input-dir /path/to/backup \
--account account1 # Optional: restore specific accountSynchronize emails between accounts:
python -m shadows_mail_tools.cli sync \
--config config.yaml \
--strategy incremental # or "one-way" \
--account account1 # Optional: sync specific accountProblem: Dashboard launched without configuration file.
Solution:
# Create a config file (see example_config.yaml)
python -m shadows_mail_tools.cli dashboard --config /path/to/config.yamlProblem: Configuration file not loaded or invalid.
Solution:
- Verify config file exists and is valid YAML
- Check config file has
accountssection with at least one account - Run with --config flag explicitly:
python -m shadows_mail_tools.cli dashboard --config config.yaml
Problem: Required fields missing for selected operation mode.
Solution:
- Backup mode: Requires backup directory path
- Restore mode: Requires restore directory path
- Sync mode: Requires sync strategy selection
- Migrate mode: Requires valid config file with accounts
Problem: Dashboard launched without --config flag and no default config found.
Solution:
- Create config file at
~/.shadows-mail-tools/config.yaml, or - Use
--configflag to specify config file path:python -m shadows_mail_tools.cli dashboard --config /path/to/config.yaml
Problem: YAML syntax errors or missing required fields.
Solution:
- Validate YAML syntax using an online YAML validator
- Check that
accountssection exists with at least one account - Verify all required fields are present (see Minimum Required Configuration above)
- Compare with
example_config.yamlfor correct structure
Set credentials using environment variables:
export SHADOWS_CRED_account1_USER="user@example.com"
export SHADOWS_CRED_account1_PASS="password"Store credentials in an encrypted file at ~/.shadows-mail-tools/credentials.enc:
# Credentials are encrypted with AES-256 using a master password
# The master password is prompted once per sessionOn Windows, credentials can be stored in Windows Credential Manager automatically.
Use credential references in your config:
accounts:
- name: "account1"
source_password: "${account1}" # References credential provider
dest_password: "${account1_dest}"Accounts are automatically grouped by source domain and processed in parallel batches. This helps:
- Respect rate limits per domain
- Process related accounts together
- Track progress per domain
Migration state is automatically saved every 30 seconds (configurable). If migration is interrupted:
- Resume from last checkpoint
- No duplicate messages
- Continue from exact point of interruption
Built-in presets for common email providers:
- Gmail: Maps
[Gmail]/Sent Mail→Sent - Outlook: Maps
Sent Items→Sent - Yahoo: Maps provider-specific folders to standard names
Custom folder mappings can be defined per account or globally.
Support for OAuth2 authentication:
- Gmail (Google OAuth2)
- Outlook (Microsoft OAuth2)
- Yahoo (Yahoo OAuth2)
OAuth2 tokens are securely stored and automatically refreshed.
Automatic detection and handling of IMAP rate limits:
- Detects rate limit errors
- Pauses affected domain batches
- Sends Discord notifications
- Resumes after cooldown period
Discord webhook notifications for:
- Account completion
- Domain batch completion
- Critical errors
- Rate limit events
Notifications include retry logic with exponential backoff.
shadows-mail-tools/
├── shadows_mail_tools/
│ ├── cli.py # Command-line interface
│ ├── dashboard.py # Interactive terminal dashboard
│ ├── config_loader.py # YAML configuration loader
│ ├── config_schema.py # Configuration data models
│ ├── credential_provider.py # Multi-source credential provider
│ ├── crypto_utils.py # Encryption utilities
│ ├── domain_batch_processor.py # Domain batch processing
│ ├── email_client.py # IMAP email client
│ ├── folder_mapping.py # Folder mapping presets
│ ├── maildir_operations.py # Maildir backup/restore
│ ├── migration.py # Email migration logic
│ ├── notification_service.py # Discord notifications
│ ├── oauth2_handler.py # OAuth2 authentication
│ ├── platform_utils.py # Cross-platform utilities
│ ├── state_manager.py # Checkpoint state management
│ └── sync_operations.py # Sync operations
├── tests/
│ ├── integration/ # Integration tests
│ └── fixtures/ # Test fixtures
└── requirements.txt
# Run all tests
pytest
# Run with coverage
pytest --cov=shadows_mail_tools --cov-report=html
# Run integration tests
pytest tests/integration/
# Run specific test file
pytest tests/test_migration.py# Format code
black .
# Sort imports
isort .
# Lint
flake8 .
pylint shadows_mail_tools
# Type checking
mypy shadows_mail_tools# Activate virtual environment
source .venv/bin/activate # Linux/Mac
.venv\Scripts\activate # Windows
# Run CLI
python -m shadows_mail_tools.cli --help
# Run dashboard
python -m shadows_mail_tools.cli dashboardThe project includes comprehensive test coverage:
- Unit Tests - Individual component testing
- Integration Tests - End-to-end workflow testing
- Mock-based Testing - No external dependencies required
Test coverage includes:
- Configuration loading and validation
- Credential provider (all sources)
- Email client operations
- Domain batch processing
- Backup and restore workflows
- Sync operations
- Checkpoint and resume
- Notification service
- OAuth2 authentication
- Cross-platform compatibility
- CLI (
cli.py) - Command-line interface using Click - Dashboard (
dashboard.py) - Interactive Textual-based UI with multi-mode operation support (migrate, backup, restore, sync) - Email Client (
email_client.py) - IMAP operations with retry logic - Migration (
migration.py) - Core migration logic - Domain Batch Processor (
domain_batch_processor.py) - Parallel batch processing - State Manager (
state_manager.py) - Checkpoint persistence - Credential Provider (
credential_provider.py) - Multi-source credential loading - Notification Service (
notification_service.py) - Discord webhooks - Config Loader (
config_loader.py) - YAML configuration management
- Configuration → Loaded from YAML with validation
- Credentials → Loaded from multiple sources (env, encrypted file, interactive)
- Domain Batching → Accounts grouped by domain
- Migration → Parallel processing with progress tracking
- State Persistence → Checkpoints saved periodically
- Notifications → Discord webhooks for events
- Dashboard → Real-time UI updates with mode-specific views and progress tracking
See requirements.txt for complete list. Key dependencies:
rich>=13.7.0- Terminal formattingtextual>=0.45.0- Terminal UI frameworkclick>=8.1.7- CLI frameworkpyyaml>=6.0.1- YAML configurationcryptography>=41.0.0- Encryptionkeyring>=24.3.0- Windows Credential Managerrequests>=2.31.0- HTTP requestspytest>=7.4.3- Testing framework
MIT
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests
- Run code quality checks
- Submit a pull request
For issues, questions, or contributions, please open an issue on the repository.