Virtual Mouse - Gesture Control System

Transform your webcam into a sophisticated AI-powered input device
Desktop-grade cursor control with intuitive hand gestures

Features • Gestures • Installation • Usage • Configuration • Architecture

---

📖 Overview

HandGestureDetector is a premium, AI-powered virtual mouse system that replaces your physical mouse with natural hand gestures. Built with MediaPipe's advanced hand tracking and high-performance interpolation algorithms, it delivers smooth, responsive cursor control that rivals traditional input devices.

Perfect for:

• 🎓 Presentations - Control slides hands-free
• ♿ Accessibility - Alternative input method for users with mobility challenges
• 🎮 Interactive Displays - Touch-free interaction for public installations
• 🚀 Innovation - Explore the future of human-computer interaction

---

✨ Key Features

🎯 Palm-Based Tracking Uses the palm's center of mass instead of fingertips for rock-solid, jitter-free cursor movement.	🤝 Dual-Hand Control Independently detects left and right hands, enabling complex multi-hand gestures and interactions.
🚀 120Hz Interpolation High-frequency interpolation layer ensures buttery-smooth cursor movement comparable to premium gaming mice.	🌊 Smart Scrolling Velocity-based scroll with acceleration curves and dead-zones for natural, precise document navigation.
🖥️ Dual-Dashboard System Real-time telemetry window + dedicated hand skeleton visualization for complete system transparency.	🛡️ Toggle Protection Accidental input prevention with 3-second hold gesture to enable/disable the entire system.

---

🎮 Gesture Reference

Single-Hand Gestures (Right Hand)

Gesture	Visual	Hand Configuration	Action	Notes
Cursor Move	☝️	Index finger extended Other fingers curled	Move cursor Palm-center tracking	Primary control gesture 120Hz interpolation
Left Click	🤏	Thumb + Index pinch Distance < 40px	Single click 300ms cooldown	Quick pinch = click Hold for drag
Drag & Drop	🤏⏱️	Pinch held > 500ms	Click + Hold Drag mode active	Release pinch to drop Visual indicator shown
Vertical Scroll	✌️	Index + Middle up Other fingers down	Scroll up/down Velocity-based	Fast hand = fast scroll Acceleration enabled
Toggle System	✊⏱️	Closed fist Hold for 3 seconds	Enable/Disable System control	Safety feature Visual feedback shown

Single-Hand Gestures (Left Hand)

Gesture	Visual	Hand Configuration	Action	Notes
Right Click	🤏	Thumb + Index pinch (Left hand)	Context menu Right-click action	Independent of right hand 300ms cooldown

Dual-Hand Gestures (Both Hands Required)

Gesture	Visual	Hand Configuration	Action	Notes
Minimize Window	🤏🤏	Both hands pinch Simultaneously	Hide active app Minimize to taskbar	500ms timing tolerance Works across all apps
Open Terminal	✊✊⚡⚡	Both hands closed fist Pinch twice rapidly	Launch terminal Windows Terminal/cmd	Double-pinch gesture Quick succession

Gesture Summary

6 Single-Hand Gestures + 2 Dual-Hand Gestures = 8 Total Commands

---

🏗️ Architecture

HandGestureDetector follows a 3-layer self-annealing architecture for maximum stability, maintainability, and extensibility:

┌─────────────────────────────────────────────────────────────┐
│                    1. DIRECTIVES LAYER                       │
│         Standard Operating Procedures (Markdown)             │
│                  Defines WHAT to do                          │
└────────────────────────┬────────────────────────────────────┘
│
┌────────────────────────▼────────────────────────────────────┐
│                   2. ORCHESTRATION LAYER                     │
│          Core Logic (main.py + /modules)                     │
│      Manages data flow between sensors & executors           │
└────────────────────────┬────────────────────────────────────┘
│
┌────────────────────────▼────────────────────────────────────┐
│                    3. EXECUTION LAYER                        │
│        Deterministic Scripts (/execution)                    │
│     Environment-specific tasks & OS-level calls              │
└─────────────────────────────────────────────────────────────┘

Layer Details

Layer	Location	Purpose	Example
Directives	/directives/*.md	Define system behavior and requirements	Gesture definitions, timing thresholds
Orchestration	/modules/*.py, main.py	Coordinate between components	Hand detection → Gesture recognition → Action
Execution	/execution/*.py	Perform platform-specific operations	Window management, terminal launch

This architecture ensures:

• ✅ Separation of Concerns - Each layer has a single responsibility
• ✅ Easy Maintenance - Change directives without touching code
• ✅ Cross-Platform - Swap execution scripts per OS
• ✅ Testability - Mock layers independently

---

🛠️ Tech Stack

Python 3.10+ Core Runtime

MediaPipe Hand Tracking AI

OpenCV Computer Vision

PySide6 (Qt) GUI Framework

Additional Libraries:

• pynput - Cross-platform mouse control
• pygetwindow - Window management (Windows)
• screeninfo - Multi-monitor support
• numpy - Numerical operations

---

🚀 Installation

Prerequisites

• Python 3.8+ (Download here)
• Webcam (built-in or USB)
• Windows/Linux/macOS (cross-platform support)

Quick Start

Option 1: Automated Setup (Windows)

# Clone the repository
git clone https://github.com/yourusername/HandGestureDetector.git
cd HandGestureDetector

# Run the launcher (automatically sets up environment)
run.bat

Option 2: Manual Installation (All Platforms)

# Clone the repository
git clone https://github.com/yourusername/HandGestureDetector.git
cd HandGestureDetector

# Create virtual environment
python -m venv venv

# Activate virtual environment
# Windows:
venv\Scripts\activate
# Linux/macOS:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Run the application
python main.py

Linux Additional Setup

For window management features on Linux:

sudo apt-get update
sudo apt-get install wmctrl

---

💻 Usage

Starting the Application

1. Launch the application using run.bat (Windows) or python main.py (all platforms)
2. Allow webcam access when prompted
3. Two windows will appear:
   • Status Dashboard - System telemetry and controls
   • Hand Visualization - Live hand skeleton overlay

Basic Workflow

• Show your RIGHT hand to the camera
• System detects hand and displays green skeleton
• Extend INDEX finger only
• Cursor begins tracking palm movement
• Pinch THUMB + INDEX together
• Performs left click
• Hold pinch for 500ms
• Enters drag mode (move hand while pinching)
• Show BOTH hands and pinch simultaneously
• Minimizes active window

Toggling Control

To disable virtual mouse control (prevent accidental inputs):

1. Make a closed fist with right hand
2. Hold for 3 seconds
3. System displays "DISABLED" status

To re-enable:

• Repeat the same gesture

Tips for Best Performance

• ✅ Lighting - Ensure good lighting on your hands
• ✅ Background - Plain backgrounds improve detection
• ✅ Distance - Keep hands 1-2 feet from camera
• ✅ Angle - Face palms toward camera for best tracking
• ✅ Calibration - First-time users: adjust sensitivity in settings

---

⚙️ Configuration

All settings are located in config/settings.py. Customize the system to your preferences:

Core Settings

CONFIG = {
    # Detection
    'MIN_DETECTION_CONFIDENCE': 0.7,    # Hand detection threshold (0-1)
    'MIN_TRACKING_CONFIDENCE': 0.5,     # Tracking stability threshold
    'MAX_NUM_HANDS': 2,                 # Enable dual-hand detection
    
    # Cursor Control
    'CURSOR_SMOOTHING_ALPHA': 0.3,      # Lower = smoother (0.1-0.5)
    'CURSOR_DEAD_ZONE_PX': 5,           # Minimum movement threshold
    'INTERPOLATION_RATE_HZ': 120,       # Cursor refresh rate
    
    # Gestures
    'CLICK_THRESHOLD_PX': 40,           # Pinch distance for click
    'DRAG_HOLD_TIME_MS': 500,           # Hold duration to start drag
    'CLICK_COOLDOWN_MS': 300,           # Prevent double-clicks
    
    # Scroll
    'SCROLL_SENSITIVITY': 1.5,          # Scroll speed multiplier
    'SCROLL_ACCELERATION_ENABLED': True,
    'SCROLL_SMOOTHING_FRAMES': 10,
    
    # Toggle
    'TOGGLE_HOLD_DURATION_MS': 3000,    # Fist hold time to toggle
    
    # Display
    'CAMERA_WIDTH': 640,
    'CAMERA_HEIGHT': 480,
    'MIRROR_CAMERA': True,              # Mirror for natural movement
}

Performance Tuning

For slower computers:

'INTERPOLATION_RATE_HZ': 60,          # Reduce to 60Hz
'CAMERA_WIDTH': 320,                   # Lower resolution
'CAMERA_HEIGHT': 240,

For ultra-smooth experience:

'INTERPOLATION_RATE_HZ': 240,         # Increase to 240Hz
'CURSOR_SMOOTHING_ALPHA': 0.2,        # More aggressive smoothing

---

📂 Project Structure

HandGestureDetector/
│
├── 📁 config/                    # Configuration files
│   └── settings.py               # System settings and constants
│
├── 📁 directives/                # Standard Operating Procedures
│   ├── gesture_definitions.md    # Gesture specifications
│   └── system_behavior.md        # System logic definitions
│
├── 📁 execution/                 # Platform-specific executors
│   ├── window_manager.py         # Window management (minimize, etc.)
│   └── terminal_launcher.py      # Terminal/cmd launcher
│
├── 📁 modules/                   # Core application logic
│   ├── init.py
│   ├── hand_detector.py          # MediaPipe hand detection wrapper
│   ├── gesture_recognizer.py    # Gesture identification logic
│   ├── cursor_smoother.py        # Cursor smoothing algorithms
│   ├── mouse_controller.py       # Mouse action executor
│   ├── state_manager.py          # System state machine
│   └── gui_dashboard.py          # PySide6 GUI components
│
├── 📁 utils/                     # Helper utilities
│   ├── init.py
│   ├── helpers.py                # Utility functions
│   ├── error_handler.py          # Error management
│   ├── calibration.py            # Calibration wizard
│   └── visualization.py          # OpenCV drawing functions
│
├── 📄 main.py                    # Application entry point
├── 📄 run.bat                    # Windows launcher script
├── 📄 requirements.txt           # Python dependencies
├── 📄 README.md                  # This file
├── 📄 LICENSE                    # MIT License
└── 📄 .gitignore                 # Git ignore rules

MIT License

Copyright (c) 2024 HandGestureDetector Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: [Full license text...]

---

🙏 Acknowledgments

• MediaPipe Team - For the incredible hand tracking ML model
• OpenCV Community - For robust computer vision tools
• Qt/PySide6 - For the professional GUI framework
• Contributors - Everyone who has helped improve this project

---

📧 Contact

Project Maintainer: Your Name

• GitHub: @yourusername
• Email: your.email@example.com
• Twitter: @yourhandle

Project Link: https://github.com/yourusername/HandGestureDetector

---

⭐ Show Your Support

If this project helped you, please consider giving it a ⭐️!

---

Made with ❤️ and ☕ for a more intuitive desktop experience

---

_{Built with Python • Powered by MediaPipe • Designed for Everyone}