A hands-free communication board for people with limited mobility — controlled entirely by looking up, down, left, or right.
Use Chrome or Edge, allow camera access, and keep your face well lit. Camera processing stays on your device.
Live gaze-control demo · click the animation to watch the higher-quality MP4
The problem • Features • How it works • Getting started • Roadmap • Contributing • References
Maintainers and contributors can review the conservative safe auto-merge policy. The automation is currently in dry-run mode and cannot merge pull requests.
Eye Voice turns a webcam into an assistive communication device. It tracks the user's iris position in real time, maps it to four directions on screen, and lets the user build and speak short messages using nothing but eye movement and dwell time — no mouse, keyboard, or touch required.
It's a static, dependency-free web app: open index.html (or serve the folder) and it runs. Face tracking happens fully client-side via MediaPipe Face Landmarker; video is never uploaded anywhere.
Losing the ability to speak or move rarely means losing the ability to think or to look. The conditions below all can leave eye movement as one of the few — sometimes the only — channels a person has left, which is exactly the gap gaze-based AAC (Augmentative and Alternative Communication) tries to close.
| Condition / metric | Figure | Source |
|---|---|---|
| People worldwide who cannot rely on natural speech to communicate | ~97 million | ASHA |
| People who actually use an AAC device or system today | ~2 million (a small fraction of those who could benefit) | ASHA |
| Americans who cannot rely on natural speech | ~5 million | ASHA |
| Global ALS (motor neurone disease) prevalence | 5.05 per 100,000, projected to rise ~25% by 2040 as populations age | Vasta et al., Annals of Clinical and Translational Neurology, 2026 |
| ALS patients who develop dysarthria (impaired speech) | 80–95%, of whom ~60% eventually need AAC | Target ALS · ALS News Today |
| Bulbar-onset ALS patients who lose functional speech within ~3 years | 60–70% | ALS News Today |
| Americans living with some form of paralysis | ~5.36 million (1.7% of the population) | Christopher & Dana Reeve Foundation |
| People living with spinal cord injury in the U.S. (end of 2024) | ~308,620, with ~18,000 new cases/year | Reeve Foundation / MSKTC |
| Leading cause of locked-in syndrome (near-total paralysis with intact cognition and eye movement) | Stroke — 86.4% of cases; traumatic brain injury accounts for most of the rest | PMC systematic review, 2021 |
| Children under 5 living with cerebral palsy worldwide | ~8.1 million (1.2%) | Lancet Global Health, 2025 |
| Typical price of a commercial eye-gaze speech-generating device | US $3,000–$15,000+ for new hardware | Inclusive Technology · Tobii Dynavox |
The access gap is the core problem Eye Voice targets: dedicated infrared eye-gaze speech-generating devices are clinically excellent but cost thousands of dollars, often require funding approval, and can take weeks to arrive. Eye Voice trades some tracking precision (see Scientific background) for zero hardware cost — it runs on any laptop or phone with a standard RGB webcam — as a stopgap, a low-stakes way to try gaze control before committing to clinical hardware, or a fallback when a dedicated device isn't available.
- 👁️ Real-time eye tracking — 478-point face mesh via MediaPipe, iris position sampled at ~30 fps with GPU acceleration (CPU fallback if unavailable)
- 🎯 9-point calibration — fits a personal affine gaze-to-screen model so accuracy adapts to each user's face and camera position
- ⏱️ Dwell-to-select input — hold a gaze on a direction to trigger it, with a visual radial progress ring and configurable dwell time (700 ms – 1500 ms)
- 🧭 Four-direction control scheme —
↑Listen ·↓Select & Speak ·←Previous word ·→Next word - 🗣️ Text-to-speech output — selected words/phrases are spoken aloud via the Web Speech API, with voice and language selection
- 🎙️ Voice input — dictate a need or question via speech recognition, transcribed live on screen
- ✨ AI-assisted suggestions — sends the transcript to the Gemini API for 4–6 contextual reply suggestions, with an offline keyword-matching fallback if no key is set or the request fails
- ♿ Accessibility-first UI — large touch targets, high-contrast focus states,
prefers-reduced-motionsupport, live regions for screen readers, keyboard-operable fallback for every gaze target - ⚙️ Adjustable sensitivity & dwell time — tunable per user in Settings, persisted in
localStorage - 🔒 Camera-local processing — video frames never leave the device; only a typed transcript (if you choose to use Listen) is sent to Gemini
Eye Voice sits at the intersection of two established research areas: appearance-based gaze estimation (inferring where someone is looking from an ordinary camera image, without dedicated infrared hardware) and dwell-time gaze interaction (using sustained fixation as a selection mechanism in place of a mouse click). Neither is novel research on its own — the contribution here is combining freely available, browser-native building blocks into a device-free, zero-install AAC tool.
Gaze estimation. Classic eye trackers use infrared illumination and a corneal-reflection ("glint") model to achieve sub-degree accuracy, but that requires dedicated hardware. Eye Voice instead uses appearance-based estimation from a plain RGB webcam: MediaPipe's Face Landmarker locates 478 3D face landmarks per frame — 468 from its Attention Mesh face model plus 10 dedicated iris landmarks (5 per eye) refined by an attention mechanism specifically over the eye regions. A published review of 40 studies on visible-light, low-cost-camera eye tracking found an average accuracy of 2.69° of visual angle; more recent appearance-based methods report 1.1–1.4° in controlled conditions, and the underlying iris model can estimate camera-to-face distance to within <10% error with no special hardware. Consumer webcam gaze estimation is therefore roughly an order of magnitude less precise than infrared trackers — which is precisely why Eye Voice avoids fine-grained cursor control and instead classifies gaze into five coarse zones (UP/DOWN/LEFT/RIGHT/CENTER), a task far more tolerant of a few degrees of error.
Reducing noise. Raw per-frame iris ratios are noisy, so gazeEstimator.js applies an exponential moving average: smoothed = previous + (raw - previous) * α, with α (0.18 / 0.28 / 0.42) tunable via the Sensitivity setting — lower α favors stability over responsiveness, higher α reacts faster but admits more jitter. This is the same low-pass-filter approach commonly used to stabilize noisy pointer signals in HCI systems, traded off against latency.
Personal calibration. Because face geometry, camera angle, and lighting vary per user, calibration.js collects gaze samples at 9 fixed screen points and solves a least-squares affine fit (screen = A·gaze_x + B·gaze_y + C, solved separately for x and y via Gaussian elimination on the 3×3 normal-equations matrix) — the same class of model classic 5–9 point eye-tracker calibration routines use to correct for per-user offset and scale before any classification happens.
Dwell-time selection. Selecting a target by holding your gaze on it (rather than blinking, or an external switch) is a well-studied AAC input technique. Foundational HCI research on gaze typing found that a fixed dwell time yields roughly 5–10 words per minute for novices, while an adjustable dwell time that shortens with practice raised throughput from 6.9 → 19.9 WPM over ten sessions (dwell time falling from 876 ms → 282 ms, error rate from 1.28% → 0.36%) — with more recent work confirming gaze typing tends to plateau around ~20 WPM once dwell thresholds drop to roughly 240–340 ms. This is also a hard mathematical ceiling: since one selection cannot complete faster than the dwell threshold t_d, throughput is bounded by 1 / t_d. Eye Voice's default 1000 ms dwell (adjustable 700–1500 ms in Settings) sits deliberately on the conservative, error-resistant end of that published range, favoring reliability for a coarse 5-target layout over the raw speed of a full on-screen keyboard.
| Design parameter | Eye Voice default | Range offered | Grounded in |
|---|---|---|---|
Dwell time (js/settings.js) |
1000 ms | 700–1500 ms | Majaranta & Räihä dwell-time / speed-accuracy tradeoff research |
Smoothing factor α (gazeEstimator.js) |
0.28 (medium) | 0.18 (low) – 0.42 (high) | EMA / low-pass filtering of noisy appearance-based gaze signal |
Direction threshold (app.js) |
0.11 (medium) | 0.07 (high sens.) – 0.16 (low sens.) | Compensates for ~2–3° typical webcam gaze error by using coarse zones, not a cursor |
Calibration points (calibration.js) |
9-point grid | — | Standard affine/polynomial calibration grid size in eye-tracking literature |
- Capture — the webcam feed is read locally via
getUserMedia. - Landmark detection —
EyeTrackerruns MediaPipe's Face Landmarker on each video frame (GPU delegate, falling back to CPU) to extract 478 face/iris landmarks, roughly every 33 ms (~30 fps). - Gaze estimation —
gazeEstimator.jscomputes each iris center relative to its eye-corner bounding box ((iris − min) / (max − min)in x and y), averages both eyes into one normalized gaze vector, then applies the exponential smoothing described above. - Calibration —
calibration.jswalks the user through 9 on-screen points, holds each for ~1.7 s, averages at least 8 gaze samples per point, and fits the affine model described above. The model is cached inlocalStorageso returning users skip recalibration. - Direction + dwell selection — the calibrated gaze point is classified into
UP/DOWN/LEFT/RIGHT/CENTERagainst the sensitivity threshold, with hysteresis (a smaller "hold" band) so natural micro-jitter near a boundary doesn't repeatedly flicker the active direction. Holding a direction fills a dwell ring (dwellSelection.js) and fires the action once it completes. - Output — a selected word is spoken via
speechSynthesis; lookingUPstarts speech recognition, and the resulting transcript is sent to the Gemini API for reply suggestions (or handled by a local rule-based fallback insuggestions.jsif no key is configured or the network call fails).
UI preview — recreated from the live app for illustration. Run it locally to see your own camera feed and tracking overlay.
Eye Voice has no build step and no dependencies to install — it's plain HTML/CSS/JS loaded as ES modules, with MediaPipe fetched from a CDN at runtime.
- A modern browser with camera access and Web Speech API support (Chrome or Edge recommended)
- A webcam
- Any static file server (ES modules can't be loaded from
file://)
git clone https://github.com/MostafaMedhat19/EyeTracking.git
cd EyeTracking
python -m http.server 4173Then open http://localhost:4173 and click Start Eye Tracking.
A ready-made launch config for this exact command is already in
.claude/launch.jsonif you're using Claude Code.
Any other static server works too, e.g. npx serve . or the VS Code "Live Server" extension.
-
Grant camera access on the start screen — video is processed locally and never recorded or uploaded.
-
Calibrate — look at each of the 9 dots as they appear and hold steady; this builds your personal gaze model.
-
Navigate with your eyes:
Direction Action ↑ Up Start / stop voice listening ↓ Down Select the current quick word and speak it ← Left Previous quick word → Right Next quick word -
Speak your need faster — look
UP, say what you need, lookUPagain to stop. Eye Voice transcribes it and (if a Gemini key is set) offers short, relevant reply suggestions in the quick-words carousel. -
Adjust settings — dwell time, sensitivity, voice, language, and your own Gemini API key are all in the Settings panel and persist across sessions (the API key persists only for the current browser tab/session).
-
Recalibrate anytime from Settings if tracking drifts (e.g. you moved, or lighting changed).
Suggestions work fine without any setup — you'll get sensible offline keyword-based suggestions out of the box. To enable smarter, context-aware suggestions:
- Get a free API key from Google AI Studio.
- Paste it into the Gemini API key field in Settings.
- The key is kept only in
sessionStoragefor that browser tab — it is never written to disk or committed to the repo.
EyeTracking/
├── .github/ # Issue forms, PR template & quality checks
├── CONTRIBUTING.md # Contributor setup and quality checklist
├── GOVERNANCE.md # Project roles and decision process
├── ROADMAP.md # Public priorities and contribution ideas
├── SECURITY.md # Responsible vulnerability reporting
├── LICENSE # MIT license
├── index.html # App shell & markup
├── css/
│ ├── style.css # Layout, theme, camera panel, carousel
│ └── accessibility.css # Focus states, reduced-motion, a11y helpers
├── js/
│ ├── app.js # Wires everything together, main frame loop
│ ├── camera.js # getUserMedia webcam setup
│ ├── eyeTracking.js # MediaPipe Face Landmarker integration
│ ├── gazeEstimator.js # Iris ratio → gaze vector, smoothing, direction
│ ├── calibration.js # 9-point calibration & affine gaze mapping
│ ├── dwellSelection.js # Dwell-timer based selection logic
│ ├── settings.js # Dwell/sensitivity persistence
│ ├── speechRecognition.js # Web Speech API (STT) wrapper
│ ├── textToSpeech.js # Web Speech API (TTS) wrapper
│ ├── suggestions.js # Offline keyword-based suggestion fallback
│ ├── geminiSuggestions.js # Gemini API request + response parsing
│ └── geminiConfig.js # Local dev key placeholder (empty by default)
└── assets/screenshots/ # README images
- Camera video is processed entirely in the browser and is never recorded, stored, or transmitted.
- The gaze calibration model is stored only in your browser's
localStorage, scoped to that browser/device. - If you use Listen, the transcribed text (not audio or video) is sent to Google's Gemini API solely to generate reply suggestions. This only happens if you've provided your own API key.
- No analytics, tracking, or third-party scripts are included beyond the MediaPipe model files (fetched from Google's CDN) and, optionally, the Gemini API call described above.
The public roadmap lives in ROADMAP.md. It includes approachable starter tasks as well as longer-term accessibility, localization, testing, and privacy improvements. If you want to help but are unsure where to begin, open a discussion or choose an issue labeled good first issue.
Contributions are welcome — code, accessibility testing, translations, documentation, clinical-context feedback, and UX research all help.
- Read
CONTRIBUTING.md. - Check the
ROADMAP.mdand existing issues. - Open an issue before a large change so we can align on scope.
- Submit a focused pull request using the included template.
Participation is governed by our CODE_OF_CONDUCT.md. Project roles and decision-making are described in GOVERNANCE.md.
Uses public GitHub profile locations when available · optional corrections live in contributors-locations.json
Every contribution matters — code, documentation, testing, translation, accessibility feedback, and design.
Eye Voice was conceived, designed, and initially implemented by Mostafa Medhat, Founder and Lead Maintainer. Community contributions are welcomed and remain credited through Git history and release notes.
The official project name, roadmap, and releases are stewarded under the process in GOVERNANCE.md. Academic or research use can cite the project using CITATION.cff.
Distributed under the MIT License. Copyright and attribution notices must be preserved in copies and substantial portions of the software; see NOTICE for project attribution.
The problem / statistics
- ASHA — Augmentative and Alternative Communication (AAC)
- Vasta et al., "Amyotrophic Lateral Sclerosis Prevalence Projection in 2040", Annals of Clinical and Translational Neurology, 2026
- ALS News Today — ALS prevalence projected to rise sharply worldwide by 2040
- Target ALS — How ALS Affects Speech
- ALS News Today — ALS speech problems
- Christopher & Dana Reeve Foundation — Spinal Cord Injury Prevalence in the U.S.
- PMC — Locked-In Syndrome: A Systematic Review of Long-Term Management and Prognosis (2021)
- The Lancet Global Health — Cerebral palsy in young children: bridging the global data gap (2025)
- Inclusive Technology — Eye Gaze Technology & Eye Tracking Devices
- Tobii Dynavox — Assistive technology devices for AAC
Gaze estimation science
- Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications (arXiv:1901.10906)
- Frontiers in Robotics and AI — Webcam-based gaze estimation for computer screen interaction (2024)
- Democratizing eye-tracking? Appearance-based gaze estimation with improved attention branch, ScienceDirect
- MediaPipe Iris — Real-time Iris Tracking & Depth Estimation
Dwell-time interaction research
- Majaranta, Ahola & Špakov, "Fast Gaze Typing with an Adjustable Dwell Time", CHI 2009
- Majaranta, MacKenzie et al., "Effects of feedback and dwell time on eye typing speed and accuracy"
- ACM CHI 2025 — There Is More to Dwell Than Meets the Eye: Toward Better Gaze-Based Text Entry Systems