Skip to content

About

Open-source, browser-based gaze-controlled assistive communication for people with limited mobility.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Eye Voice — gaze-controlled assistive communication

Made with JavaScript MediaPipe Face Landmarker No build step Runs 100% in-browser License Contributions welcome Code of Conduct

A hands-free communication board for people with limited mobility — controlled entirely by looking up, down, left, or right.

Use Chrome or Edge, allow camera access, and keep your face well lit. Camera processing stays on your device.



Eye Voice demo showing real-time iris tracking and gaze-controlled navigation between quick words

Live gaze-control demo · click the animation to watch the higher-quality MP4

The problem • Features • How it works • Getting started • Roadmap • Contributing • References

Maintainers and contributors can review the conservative safe auto-merge policy. The automation is currently in dry-run mode and cannot merge pull requests.


About

Eye Voice turns a webcam into an assistive communication device. It tracks the user's iris position in real time, maps it to four directions on screen, and lets the user build and speak short messages using nothing but eye movement and dwell time — no mouse, keyboard, or touch required.

It's a static, dependency-free web app: open index.html (or serve the folder) and it runs. Face tracking happens fully client-side via MediaPipe Face Landmarker; video is never uploaded anywhere.

The problem, by the numbers

Losing the ability to speak or move rarely means losing the ability to think or to look. The conditions below all can leave eye movement as one of the few — sometimes the only — channels a person has left, which is exactly the gap gaze-based AAC (Augmentative and Alternative Communication) tries to close.

Condition / metric Figure Source
People worldwide who cannot rely on natural speech to communicate ~97 million ASHA
People who actually use an AAC device or system today ~2 million (a small fraction of those who could benefit) ASHA
Americans who cannot rely on natural speech ~5 million ASHA
Global ALS (motor neurone disease) prevalence 5.05 per 100,000, projected to rise ~25% by 2040 as populations age Vasta et al., Annals of Clinical and Translational Neurology, 2026
ALS patients who develop dysarthria (impaired speech) 80–95%, of whom ~60% eventually need AAC Target ALS · ALS News Today
Bulbar-onset ALS patients who lose functional speech within ~3 years 60–70% ALS News Today
Americans living with some form of paralysis ~5.36 million (1.7% of the population) Christopher & Dana Reeve Foundation
People living with spinal cord injury in the U.S. (end of 2024) ~308,620, with ~18,000 new cases/year Reeve Foundation / MSKTC
Leading cause of locked-in syndrome (near-total paralysis with intact cognition and eye movement) Stroke — 86.4% of cases; traumatic brain injury accounts for most of the rest PMC systematic review, 2021
Children under 5 living with cerebral palsy worldwide ~8.1 million (1.2%) Lancet Global Health, 2025
Typical price of a commercial eye-gaze speech-generating device US $3,000–$15,000+ for new hardware Inclusive Technology · Tobii Dynavox

The access gap is the core problem Eye Voice targets: dedicated infrared eye-gaze speech-generating devices are clinically excellent but cost thousands of dollars, often require funding approval, and can take weeks to arrive. Eye Voice trades some tracking precision (see Scientific background) for zero hardware cost — it runs on any laptop or phone with a standard RGB webcam — as a stopgap, a low-stakes way to try gaze control before committing to clinical hardware, or a fallback when a dedicated device isn't available.

Features

  • 👁️ Real-time eye tracking — 478-point face mesh via MediaPipe, iris position sampled at ~30 fps with GPU acceleration (CPU fallback if unavailable)
  • 🎯 9-point calibration — fits a personal affine gaze-to-screen model so accuracy adapts to each user's face and camera position
  • ⏱️ Dwell-to-select input — hold a gaze on a direction to trigger it, with a visual radial progress ring and configurable dwell time (700 ms – 1500 ms)
  • 🧭 Four-direction control scheme — ↑ Listen · ↓ Select & Speak · ← Previous word · → Next word
  • 🗣️ Text-to-speech output — selected words/phrases are spoken aloud via the Web Speech API, with voice and language selection
  • 🎙️ Voice input — dictate a need or question via speech recognition, transcribed live on screen
  • ✨ AI-assisted suggestions — sends the transcript to the Gemini API for 4–6 contextual reply suggestions, with an offline keyword-matching fallback if no key is set or the request fails
  • ♿ Accessibility-first UI — large touch targets, high-contrast focus states, prefers-reduced-motion support, live regions for screen readers, keyboard-operable fallback for every gaze target
  • ⚙️ Adjustable sensitivity & dwell time — tunable per user in Settings, persisted in localStorage
  • 🔒 Camera-local processing — video frames never leave the device; only a typed transcript (if you choose to use Listen) is sent to Gemini

Scientific background

Eye Voice sits at the intersection of two established research areas: appearance-based gaze estimation (inferring where someone is looking from an ordinary camera image, without dedicated infrared hardware) and dwell-time gaze interaction (using sustained fixation as a selection mechanism in place of a mouse click). Neither is novel research on its own — the contribution here is combining freely available, browser-native building blocks into a device-free, zero-install AAC tool.

Gaze estimation. Classic eye trackers use infrared illumination and a corneal-reflection ("glint") model to achieve sub-degree accuracy, but that requires dedicated hardware. Eye Voice instead uses appearance-based estimation from a plain RGB webcam: MediaPipe's Face Landmarker locates 478 3D face landmarks per frame — 468 from its Attention Mesh face model plus 10 dedicated iris landmarks (5 per eye) refined by an attention mechanism specifically over the eye regions. A published review of 40 studies on visible-light, low-cost-camera eye tracking found an average accuracy of 2.69° of visual angle; more recent appearance-based methods report 1.1–1.4° in controlled conditions, and the underlying iris model can estimate camera-to-face distance to within <10% error with no special hardware. Consumer webcam gaze estimation is therefore roughly an order of magnitude less precise than infrared trackers — which is precisely why Eye Voice avoids fine-grained cursor control and instead classifies gaze into five coarse zones (UP/DOWN/LEFT/RIGHT/CENTER), a task far more tolerant of a few degrees of error.

Reducing noise. Raw per-frame iris ratios are noisy, so gazeEstimator.js applies an exponential moving average: smoothed = previous + (raw - previous) * α, with α (0.18 / 0.28 / 0.42) tunable via the Sensitivity setting — lower α favors stability over responsiveness, higher α reacts faster but admits more jitter. This is the same low-pass-filter approach commonly used to stabilize noisy pointer signals in HCI systems, traded off against latency.

Personal calibration. Because face geometry, camera angle, and lighting vary per user, calibration.js collects gaze samples at 9 fixed screen points and solves a least-squares affine fit (screen = A·gaze_x + B·gaze_y + C, solved separately for x and y via Gaussian elimination on the 3×3 normal-equations matrix) — the same class of model classic 5–9 point eye-tracker calibration routines use to correct for per-user offset and scale before any classification happens.

Dwell-time selection. Selecting a target by holding your gaze on it (rather than blinking, or an external switch) is a well-studied AAC input technique. Foundational HCI research on gaze typing found that a fixed dwell time yields roughly 5–10 words per minute for novices, while an adjustable dwell time that shortens with practice raised throughput from 6.9 → 19.9 WPM over ten sessions (dwell time falling from 876 ms → 282 ms, error rate from 1.28% → 0.36%) — with more recent work confirming gaze typing tends to plateau around ~20 WPM once dwell thresholds drop to roughly 240–340 ms. This is also a hard mathematical ceiling: since one selection cannot complete faster than the dwell threshold t_d, throughput is bounded by 1 / t_d. Eye Voice's default 1000 ms dwell (adjustable 700–1500 ms in Settings) sits deliberately on the conservative, error-resistant end of that published range, favoring reliability for a coarse 5-target layout over the raw speed of a full on-screen keyboard.

Design parameter Eye Voice default Range offered Grounded in
Dwell time (js/settings.js) 1000 ms 700–1500 ms Majaranta & Räihä dwell-time / speed-accuracy tradeoff research
Smoothing factor α (gazeEstimator.js) 0.28 (medium) 0.18 (low) – 0.42 (high) EMA / low-pass filtering of noisy appearance-based gaze signal
Direction threshold (app.js) 0.11 (medium) 0.07 (high sens.) – 0.16 (low sens.) Compensates for ~2–3° typical webcam gaze error by using coarse zones, not a cursor
Calibration points (calibration.js) 9-point grid — Standard affine/polynomial calibration grid size in eye-tracking literature

How it works

Eye Voice pipeline: webcam → MediaPipe Face Landmarker → gaze estimation → calibration → dwell selection → speak or listen, with Gemini-assisted suggestions feeding back into the quick-words carousel
  1. Capture — the webcam feed is read locally via getUserMedia.
  2. Landmark detection — EyeTracker runs MediaPipe's Face Landmarker on each video frame (GPU delegate, falling back to CPU) to extract 478 face/iris landmarks, roughly every 33 ms (~30 fps).
  3. Gaze estimation — gazeEstimator.js computes each iris center relative to its eye-corner bounding box ((iris − min) / (max − min) in x and y), averages both eyes into one normalized gaze vector, then applies the exponential smoothing described above.
  4. Calibration — calibration.js walks the user through 9 on-screen points, holds each for ~1.7 s, averages at least 8 gaze samples per point, and fits the affine model described above. The model is cached in localStorage so returning users skip recalibration.
  5. Direction + dwell selection — the calibrated gaze point is classified into UP / DOWN / LEFT / RIGHT / CENTER against the sensitivity threshold, with hysteresis (a smaller "hold" band) so natural micro-jitter near a boundary doesn't repeatedly flicker the active direction. Holding a direction fills a dwell ring (dwellSelection.js) and fires the action once it completes.
  6. Output — a selected word is spoken via speechSynthesis; looking UP starts speech recognition, and the resulting transcript is sent to the Gemini API for reply suggestions (or handled by a local rule-based fallback in suggestions.js if no key is configured or the network call fails).

Preview

Eye Voice — communication beyond touch using browser-based eye tracking



Eye Voice interface: live tracking panel with four directional targets around the camera view, current message bar, Listen action, and a quick-words carousel

UI preview — recreated from the live app for illustration. Run it locally to see your own camera feed and tracking overlay.

Getting started

Eye Voice has no build step and no dependencies to install — it's plain HTML/CSS/JS loaded as ES modules, with MediaPipe fetched from a CDN at runtime.

Prerequisites

  • A modern browser with camera access and Web Speech API support (Chrome or Edge recommended)
  • A webcam
  • Any static file server (ES modules can't be loaded from file://)

Run locally

git clone https://github.com/MostafaMedhat19/EyeTracking.git
cd EyeTracking
python -m http.server 4173

Then open http://localhost:4173 and click Start Eye Tracking.

A ready-made launch config for this exact command is already in .claude/launch.json if you're using Claude Code.

Any other static server works too, e.g. npx serve . or the VS Code "Live Server" extension.

Using Eye Voice

  1. Grant camera access on the start screen — video is processed locally and never recorded or uploaded.

  2. Calibrate — look at each of the 9 dots as they appear and hold steady; this builds your personal gaze model.

  3. Navigate with your eyes:

    Direction Action
    ↑ Up Start / stop voice listening
    ↓ Down Select the current quick word and speak it
    ← Left Previous quick word
    → Right Next quick word
  4. Speak your need faster — look UP, say what you need, look UP again to stop. Eye Voice transcribes it and (if a Gemini key is set) offers short, relevant reply suggestions in the quick-words carousel.

  5. Adjust settings — dwell time, sensitivity, voice, language, and your own Gemini API key are all in the Settings panel and persist across sessions (the API key persists only for the current browser tab/session).

  6. Recalibrate anytime from Settings if tracking drifts (e.g. you moved, or lighting changed).

Using AI suggestions (optional)

Suggestions work fine without any setup — you'll get sensible offline keyword-based suggestions out of the box. To enable smarter, context-aware suggestions:

  1. Get a free API key from Google AI Studio.
  2. Paste it into the Gemini API key field in Settings.
  3. The key is kept only in sessionStorage for that browser tab — it is never written to disk or committed to the repo.

Project structure

EyeTracking/
├── .github/                   # Issue forms, PR template & quality checks
├── CONTRIBUTING.md            # Contributor setup and quality checklist
├── GOVERNANCE.md              # Project roles and decision process
├── ROADMAP.md                 # Public priorities and contribution ideas
├── SECURITY.md                # Responsible vulnerability reporting
├── LICENSE                    # MIT license
├── index.html                 # App shell & markup
├── css/
│   ├── style.css              # Layout, theme, camera panel, carousel
│   └── accessibility.css      # Focus states, reduced-motion, a11y helpers
├── js/
│   ├── app.js                 # Wires everything together, main frame loop
│   ├── camera.js              # getUserMedia webcam setup
│   ├── eyeTracking.js         # MediaPipe Face Landmarker integration
│   ├── gazeEstimator.js       # Iris ratio → gaze vector, smoothing, direction
│   ├── calibration.js         # 9-point calibration & affine gaze mapping
│   ├── dwellSelection.js      # Dwell-timer based selection logic
│   ├── settings.js            # Dwell/sensitivity persistence
│   ├── speechRecognition.js   # Web Speech API (STT) wrapper
│   ├── textToSpeech.js        # Web Speech API (TTS) wrapper
│   ├── suggestions.js         # Offline keyword-based suggestion fallback
│   ├── geminiSuggestions.js   # Gemini API request + response parsing
│   └── geminiConfig.js        # Local dev key placeholder (empty by default)
└── assets/screenshots/        # README images

Privacy & data handling

  • Camera video is processed entirely in the browser and is never recorded, stored, or transmitted.
  • The gaze calibration model is stored only in your browser's localStorage, scoped to that browser/device.
  • If you use Listen, the transcribed text (not audio or video) is sent to Google's Gemini API solely to generate reply suggestions. This only happens if you've provided your own API key.
  • No analytics, tracking, or third-party scripts are included beyond the MediaPipe model files (fetched from Google's CDN) and, optionally, the Gemini API call described above.

Roadmap

The public roadmap lives in ROADMAP.md. It includes approachable starter tasks as well as longer-term accessibility, localization, testing, and privacy improvements. If you want to help but are unsure where to begin, open a discussion or choose an issue labeled good first issue.

Contributing

Contributions are welcome — code, accessibility testing, translations, documentation, clinical-context feedback, and UX research all help.

  1. Read CONTRIBUTING.md.
  2. Check the ROADMAP.md and existing issues.
  3. Open an issue before a large change so we can align on scope.
  4. Submit a focused pull request using the included template.

Participation is governed by our CODE_OF_CONDUCT.md. Project roles and decision-making are described in GOVERNANCE.md.

Community around the world

World map showing EyeTracking contributor counts and colors by country
Uses public GitHub profile locations when available · optional corrections live in contributors-locations.json

Star history

EyeTracking GitHub star history
Generated from live GitHub data every day and after changes to main · click to view stargazers

Contributors

EyeTracking contributors with GitHub avatars and contribution counts
Every contribution matters — code, documentation, testing, translation, accessibility feedback, and design.

Project leadership & attribution

Eye Voice was conceived, designed, and initially implemented by Mostafa Medhat, Founder and Lead Maintainer. Community contributions are welcomed and remain credited through Git history and release notes.

The official project name, roadmap, and releases are stewarded under the process in GOVERNANCE.md. Academic or research use can cite the project using CITATION.cff.

License

Distributed under the MIT License. Copyright and attribution notices must be preserved in copies and substantial portions of the software; see NOTICE for project attribution.

References

The problem / statistics

Gaze estimation science

Dwell-time interaction research


Built to give a voice back to people who can't use their hands or speech to communicate.

About

Open-source, browser-based gaze-controlled assistive communication for people with limited mobility.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Sponsor this project

Packages

Contributors

Languages