Skip to content

About

Python terminal dashboard for email migration, backup, restore, and synchronization - interactive TUI built with Textual. By Nick Loubser.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Shadows Mail Tools

A comprehensive Python terminal dashboard for email migration, backup, restore, and synchronization with a modern, interactive interface built with Textual.

Features

Core Functionality

  • Interactive Terminal Dashboard - Real-time operation monitoring with Textual framework supporting migrate, backup, restore, and sync modes
  • Email Migration - Migrate emails between IMAP servers with folder structure preservation
  • Domain Batch Processing - Process multiple accounts grouped by domain in parallel
  • Backup & Restore - Export to Maildir++ format and restore with full metadata preservation
  • Synchronization - One-way and incremental sync strategies between accounts
  • Checkpoint & Resume - Automatic state persistence with resume capability after interruptions

Security & Credentials

  • Multi-Source Credential Provider - Load credentials from environment variables, encrypted files, or interactive prompts
  • AES-256 Encryption - Secure credential storage using Fernet encryption
  • Windows Credential Manager - Integration with Windows Credential Manager via keyring
  • OAuth2 Authentication - Support for Gmail, Outlook, and Yahoo OAuth2 authentication

Reliability & Monitoring

  • Discord Notifications - Webhook notifications for account completion, batch completion, and errors
  • Rate Limit Detection - Automatic detection and handling of IMAP rate limits
  • Retry Logic - Configurable exponential backoff retry for transient errors
  • Error Classification - Intelligent error classification (transient, permanent, rate_limit)

Configuration & Customization

  • YAML Configuration - Comprehensive YAML-based configuration with validation
  • Folder Mapping Presets - Built-in presets for Gmail, Outlook, and Yahoo folder mappings
  • Command-Line Overrides - Override configuration settings via CLI arguments
  • Cross-Platform Support - Full Windows and Linux compatibility

Installation

Prerequisites

  • Python 3.9 or higher
  • pip

Install Dependencies

pip install -r requirements.txt

Install Package (Development)

pip install -e .

Quick Start

  1. Create a configuration file (see example_config.yaml):

    cp example_config.yaml ~/.shadows-mail-tools/config.yaml
    # Edit with your account details
  2. Launch the dashboard:

    python -m shadows_mail_tools.cli dashboard --config ~/.shadows-mail-tools/config.yaml

    Or use the default config location:

    # Place config at ~/.shadows-mail-tools/config.yaml
    python -m shadows_mail_tools.cli dashboard
  3. Select operation mode from the dropdown (Migrate, Backup, Restore, or Sync)

  4. Configure and start your operation

  5. Monitor progress with interactive controls:

    • p - Pause/Resume
    • s - Stop gracefully
    • r - Retry failed accounts
    • d - Toggle domain/account view
    • q - Quit

Configuration

The dashboard requires a configuration file to function. Without it, you'll see a message to load configuration.

Create your config file at ~/.shadows-mail-tools/config.yaml or specify a custom path with --config.

See example_config.yaml for a complete example with all available options.

Minimum Required Configuration

For the dashboard to work, you need at least:

settings:
  concurrency: 3

accounts:
  - name: "account1"
    source_host: "imap.source.com"
    source_port: 993
    source_username: "user@source.com"
    source_password: "password"
    source_use_ssl: true
    dest_host: "imap.dest.com"
    dest_port: 993
    dest_username: "user@dest.com"
    dest_password: "password"
    dest_use_ssl: true
    domain: "source.com"

Configuration Sections

Settings

settings:
  concurrency: 3              # Number of domain batches to process in parallel
  checkpoint_interval: 30      # Checkpoint save interval in seconds

Accounts

accounts:
  - name: "account1"
    source_host: "imap.source.com"
    source_port: 993
    source_username: "user@source.com"
    source_password: "password"  # Or use credential references
    source_use_ssl: true
    dest_host: "imap.dest.com"
    dest_port: 993
    dest_username: "user@dest.com"
    dest_password: "password"
    dest_use_ssl: true

Notifications

notifications:
  discord_webhook_url: "https://discord.com/api/webhooks/your-webhook-url"

Folder Mappings

folder_mappings:
  "INBOX.Sent": "Sent"
  "INBOX.Trash": "Trash"

Or use presets for Gmail, Outlook, or Yahoo.

Usage

Interactive Dashboard

IMPORTANT: The dashboard requires a configuration file to enable all operation modes.

Launch the dashboard with a configuration file:

python -m shadows_mail_tools.cli dashboard --config config.yaml

Or use the default config location:

# Place config at ~/.shadows-mail-tools/config.yaml
python -m shadows_mail_tools.cli dashboard

Without a config file, the dashboard will show limited functionality with a message to load configuration.

With a config file, you'll see:

  • Operation Mode selector (Migrate, Backup, Restore, Sync)
  • Mode-specific configuration forms
  • Domain batch view
  • Real-time progress tracking
  • Full interactive controls

Dashboard Operation Modes

Once the dashboard is running with a config file, use the Operation Mode dropdown to select your desired operation:

Migrate Mode

  1. Select "Migrate" from the Operation Mode dropdown
  2. The dashboard will use accounts from your config file
  3. Click "Start Migration" to begin
  4. Monitor progress in Domain Batch View or Account View (toggle with 'd' key)

What it does:

  • Migrates emails between source and destination accounts
  • Uses accounts from configuration file
  • Processes accounts in domain batches
  • Shows domain batch progress and per-account status

Backup Mode

  1. Select "Backup" from the Operation Mode dropdown
  2. Enter the backup directory path (e.g., /home/user/email-backups or C:\Users\User\email-backups)
  3. Click "Start Backup" to export emails to Maildir++ format
  4. Monitor files being written in the activity log

What it does:

  • Exports emails to Maildir++ format
  • Backs up all configured accounts
  • Preserves folder structure and IMAP flags
  • Creates organized directory structure per account

Restore Mode

  1. Select "Restore" from the Operation Mode dropdown
  2. Enter the backup directory path containing Maildir++ files
  3. The dashboard will use destination accounts from your config
  4. Click "Start Restore" to import emails back to IMAP
  5. Monitor folder recreation and message uploads

What it does:

  • Imports emails from Maildir++ format
  • Restores to configured destination accounts
  • Recreates folder structure and restores flags
  • Preserves original message metadata

Sync Mode

  1. Select "Sync" from the Operation Mode dropdown
  2. Choose sync strategy (One-way or Incremental)
  3. Click "Start Sync" to synchronize accounts
  4. Monitor only new messages being processed (for incremental)

What it does:

  • Synchronizes emails between accounts
  • Supports One-Way (source → destination) or Incremental (only new messages) strategies
  • Syncs all configured accounts
  • Tracks sync state to prevent duplicates

Dashboard Controls:

  • p - Pause/Resume operation
  • s - Stop operation gracefully
  • r - Retry failed accounts (migrate mode)
  • d - Toggle between domain batch view and account view
  • q - Quit application

Dashboard Features:

  • Real-time progress tracking with mode-appropriate metrics
  • Domain batch view showing progress per domain
  • Account-level status view
  • Live operation log with timestamps
  • Statistics panel (total, completed, failed, remaining, throughput)
  • Discord notifications for operation completion and errors

Migration Command

Migrate emails between accounts:

python -m shadows_mail_tools.cli migrate --config config.yaml

Options:

  • --config, -c - Path to configuration file
  • --concurrency - Override concurrency setting
  • --checkpoint-interval - Override checkpoint interval
  • --resume - Resume from last checkpoint
  • --migration-id - Migration ID for checkpoint resume

Example:

python -m shadows_mail_tools.cli migrate \
  --config config.yaml \
  --concurrency 5 \
  --checkpoint-interval 60

Backup Command

Backup emails to Maildir++ format:

python -m shadows_mail_tools.cli backup \
  --config config.yaml \
  --output-dir /path/to/backup \
  --account account1  # Optional: backup specific account

Restore Command

Restore emails from Maildir++ format:

python -m shadows_mail_tools.cli restore \
  --config config.yaml \
  --input-dir /path/to/backup \
  --account account1  # Optional: restore specific account

Sync Command

Synchronize emails between accounts:

python -m shadows_mail_tools.cli sync \
  --config config.yaml \
  --strategy incremental  # or "one-way" \
  --account account1  # Optional: sync specific account

Troubleshooting

Dashboard shows "No batch processor configured"

Problem: Dashboard launched without configuration file.

Solution:

# Create a config file (see example_config.yaml)
python -m shadows_mail_tools.cli dashboard --config /path/to/config.yaml

Operation mode selector not visible

Problem: Configuration file not loaded or invalid.

Solution:

  1. Verify config file exists and is valid YAML
  2. Check config file has accounts section with at least one account
  3. Run with --config flag explicitly:
    python -m shadows_mail_tools.cli dashboard --config config.yaml

"Please fill in all fields" error

Problem: Required fields missing for selected operation mode.

Solution:

  • Backup mode: Requires backup directory path
  • Restore mode: Requires restore directory path
  • Sync mode: Requires sync strategy selection
  • Migrate mode: Requires valid config file with accounts

Dashboard shows limited functionality

Problem: Dashboard launched without --config flag and no default config found.

Solution:

  1. Create config file at ~/.shadows-mail-tools/config.yaml, or
  2. Use --config flag to specify config file path:
    python -m shadows_mail_tools.cli dashboard --config /path/to/config.yaml

Invalid configuration file

Problem: YAML syntax errors or missing required fields.

Solution:

  1. Validate YAML syntax using an online YAML validator
  2. Check that accounts section exists with at least one account
  3. Verify all required fields are present (see Minimum Required Configuration above)
  4. Compare with example_config.yaml for correct structure

Credential Management

Environment Variables

Set credentials using environment variables:

export SHADOWS_CRED_account1_USER="user@example.com"
export SHADOWS_CRED_account1_PASS="password"

Encrypted Credential File

Store credentials in an encrypted file at ~/.shadows-mail-tools/credentials.enc:

# Credentials are encrypted with AES-256 using a master password
# The master password is prompted once per session

Windows Credential Manager

On Windows, credentials can be stored in Windows Credential Manager automatically.

Credential References in Config

Use credential references in your config:

accounts:
  - name: "account1"
    source_password: "${account1}"  # References credential provider
    dest_password: "${account1_dest}"

Features in Detail

Domain Batch Processing

Accounts are automatically grouped by source domain and processed in parallel batches. This helps:

  • Respect rate limits per domain
  • Process related accounts together
  • Track progress per domain

Checkpoint & Resume

Migration state is automatically saved every 30 seconds (configurable). If migration is interrupted:

  • Resume from last checkpoint
  • No duplicate messages
  • Continue from exact point of interruption

Folder Mapping

Built-in presets for common email providers:

  • Gmail: Maps [Gmail]/Sent Mail → Sent
  • Outlook: Maps Sent Items → Sent
  • Yahoo: Maps provider-specific folders to standard names

Custom folder mappings can be defined per account or globally.

OAuth2 Authentication

Support for OAuth2 authentication:

  • Gmail (Google OAuth2)
  • Outlook (Microsoft OAuth2)
  • Yahoo (Yahoo OAuth2)

OAuth2 tokens are securely stored and automatically refreshed.

Rate Limit Handling

Automatic detection and handling of IMAP rate limits:

  • Detects rate limit errors
  • Pauses affected domain batches
  • Sends Discord notifications
  • Resumes after cooldown period

Notification Service

Discord webhook notifications for:

  • Account completion
  • Domain batch completion
  • Critical errors
  • Rate limit events

Notifications include retry logic with exponential backoff.

Development

Project Structure

shadows-mail-tools/
├── shadows_mail_tools/
│   ├── cli.py                    # Command-line interface
│   ├── dashboard.py              # Interactive terminal dashboard
│   ├── config_loader.py          # YAML configuration loader
│   ├── config_schema.py          # Configuration data models
│   ├── credential_provider.py   # Multi-source credential provider
│   ├── crypto_utils.py           # Encryption utilities
│   ├── domain_batch_processor.py # Domain batch processing
│   ├── email_client.py           # IMAP email client
│   ├── folder_mapping.py         # Folder mapping presets
│   ├── maildir_operations.py    # Maildir backup/restore
│   ├── migration.py              # Email migration logic
│   ├── notification_service.py  # Discord notifications
│   ├── oauth2_handler.py        # OAuth2 authentication
│   ├── platform_utils.py        # Cross-platform utilities
│   ├── state_manager.py         # Checkpoint state management
│   └── sync_operations.py       # Sync operations
├── tests/
│   ├── integration/             # Integration tests
│   └── fixtures/                # Test fixtures
└── requirements.txt

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=shadows_mail_tools --cov-report=html

# Run integration tests
pytest tests/integration/

# Run specific test file
pytest tests/test_migration.py

Code Quality

# Format code
black .

# Sort imports
isort .

# Lint
flake8 .
pylint shadows_mail_tools

# Type checking
mypy shadows_mail_tools

Running Locally

# Activate virtual environment
source .venv/bin/activate  # Linux/Mac
.venv\Scripts\activate      # Windows

# Run CLI
python -m shadows_mail_tools.cli --help

# Run dashboard
python -m shadows_mail_tools.cli dashboard

Testing

The project includes comprehensive test coverage:

  • Unit Tests - Individual component testing
  • Integration Tests - End-to-end workflow testing
  • Mock-based Testing - No external dependencies required

Test coverage includes:

  • Configuration loading and validation
  • Credential provider (all sources)
  • Email client operations
  • Domain batch processing
  • Backup and restore workflows
  • Sync operations
  • Checkpoint and resume
  • Notification service
  • OAuth2 authentication
  • Cross-platform compatibility

Architecture

Components

  1. CLI (cli.py) - Command-line interface using Click
  2. Dashboard (dashboard.py) - Interactive Textual-based UI with multi-mode operation support (migrate, backup, restore, sync)
  3. Email Client (email_client.py) - IMAP operations with retry logic
  4. Migration (migration.py) - Core migration logic
  5. Domain Batch Processor (domain_batch_processor.py) - Parallel batch processing
  6. State Manager (state_manager.py) - Checkpoint persistence
  7. Credential Provider (credential_provider.py) - Multi-source credential loading
  8. Notification Service (notification_service.py) - Discord webhooks
  9. Config Loader (config_loader.py) - YAML configuration management

Data Flow

  1. Configuration → Loaded from YAML with validation
  2. Credentials → Loaded from multiple sources (env, encrypted file, interactive)
  3. Domain Batching → Accounts grouped by domain
  4. Migration → Parallel processing with progress tracking
  5. State Persistence → Checkpoints saved periodically
  6. Notifications → Discord webhooks for events
  7. Dashboard → Real-time UI updates with mode-specific views and progress tracking

Requirements

See requirements.txt for complete list. Key dependencies:

  • rich>=13.7.0 - Terminal formatting
  • textual>=0.45.0 - Terminal UI framework
  • click>=8.1.7 - CLI framework
  • pyyaml>=6.0.1 - YAML configuration
  • cryptography>=41.0.0 - Encryption
  • keyring>=24.3.0 - Windows Credential Manager
  • requests>=2.31.0 - HTTP requests
  • pytest>=7.4.3 - Testing framework

License

MIT

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests
  5. Run code quality checks
  6. Submit a pull request

Support

For issues, questions, or contributions, please open an issue on the repository.

About

Python terminal dashboard for email migration, backup, restore, and synchronization - interactive TUI built with Textual. By Nick Loubser.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages