Cue

🙌 Cue 🙌

Video calls have become the default way people meet, but they're built entirely around spoken language. For someone who is deaf or hard of hearing and doesn't have a hearing interpreter on the call, a Google Meet is functionally unusable. Captions only work one direction, and there's no way to sign back and be understood. Cue is a real-time translation layer for video meetings. It uses computer vision and machine learning to read a user's sign language through their camera and speak it aloud to hearing participants, while simultaneously converting hearing participants' speech into on-screen captions for the signer. Unlike existing sign-recognition tools that require specialized hardware or are built purely for teaching, Cue is designed to sit invisibly inside a call people are already having, turning a one-way captioning feature into an actual two-way conversation.

---

MVP ✅

• Meeting Integration → Connection to Google Meet
• Sign Recognition Pipeline → Gesture classification using the pretrained Pose-TGCN model
• Speech Output → ASL converted to Speech
• Type-To-Voice → Alternative Type to Speech
• Signer-Facing Feedback → Display of the currently detected sign in real time
• Captions for the Signer → Captions that show for other people talking

---

Stretch Goals 💪

• Phrase Predictions → Predictive text/phrase suggestions
• Multiple Languages → Support sign language dialects beyond ASL (e.g., BSL, LSF)

---

Tech Stack & Resources 💻

📱 Frontend

• React Documentation
• Tailwind CSS Documentation
• Chrome Extensions Documentation
• Chrome Extensions: Get Started Tutorial
• VIDEO: React JS Full Course for Beginners
• VIDEO: Chrome Extension Development Tutorial for Beginners
• VIDEO: Tailwind CSS Full Course for Beginners

🔐 Auth

• Google OAuth with FastAPI
• Passport.js
• Firebase Authentication
• VIDEO: Code with Me: Learning FastAPI + Google OAuth2
• VIDEO: FastAPI Authentication Example With OAuth2, JSON Web Tokens and Tortoise ORM
• VIDEO: Node.js Passport Login System Tutorial
• VIDEO: Express JS #15 - Passport.js Authentication

👀 Vision & Recognition

• MediaPipe (Hollistic Landmarker)
• Sign Classification: Pose-TGCN
• PyTorch + scikit-learn

💬 Speech Layer

• Speech-to-Text
• Virtual Audio Cable
• PyTorch + scikit-learn
• OBS Virtual Camera

🗄️ Database

• Firebase Firestore
• PostgreSQL Docs
• Supabase
• AWS S3
• VIDEO: Firebase Firestore Tutorial #1 - Introduction (full series)
• VIDEO: PostgreSQL Tutorial Full Course 2022
• VIDEO: AWS S3 Tutorial for Beginners 2023 | AWS S3 Overview | Cloud Computing

🎨 Design

• Figma
• LottieFiles
• VIDEO: Figma Tutorial for Beginners

🛠️ Dev Tools

• Node.js:
  Download: Node.js LTS
  Tutorial: Installing Node.js and npm

• VS Code:
  Download: Visual Studio Code
  Tutorial: VS Code Tips for React Native

• Postman (API Testing):
  Download: Postman
  Tutorial: Postman API Testing for Beginners

• Git:
  Download: Git
  Tutorial: Git Handbook

• Python:
  Download: Python
  Tutorial: FastAPI Quickstart

---

Roadmap 📅

Week	Frontend	Backend
1	Decide roles, discuss project plan, design day Decide integration approach (browser extension vs. virtual camera/mic) and UI layout	Set up repo Load the pretrained Pose-TGCN (PyTorch) checkpoint and confirm it runs on a static test clip's precomputed keypoints (not webcam yet).
2	Build meeting-overlay UI: detected sign display, confidence score bar	Build the MediaPipe model keypoint remapping/normalization layer, and a rolling buffer that samples frames into the model's expected input shape.
3	Build caption display panel for incoming speech	Integrate a speech-to-text API (e.g., Web Speech API / Google Cloud Speech-to-Text) Build sign-boundary segmentation (pause-based or sliding-window) so continuous signing gets split into individual signs before classification.
4	Style the sentence-builder UI (running transcript of assembled signs)	Build sentence-assembly logic: buffer recognized signs into phrases with basic grammar rules
5	Wire TTS audio into UI controls (mute/replay last phrase)	Integrate TTS engine; route synthesized audio into a virtual audio output device
6	Build Google Meet integration UI (join flow, camera/mic device selection)	Implement virtual camera/mic injection so Cue's outputs appear as native call audio/video
7	Polish caption/sign overlay positioning and readability START PRESENTATION PREP and freeze new features	Tune model confidence thresholds; add a "low confidence please repeat" prompt Tune model confidence thresholds; add a "low confidence please repeat" prompt
8	Polish UI colors/fonts/accessibility (contrast, font size for captions) Presentation Prep.	Fix critical bugs; test full round-trip latency (sign → speech, speech → caption) Presentation Prep.
9	✨ Presentation Prep ✨

---

GitHub Cheat Sheet 💬

Command	Description
cd	Change directories over to our repository
git branch	Lists branches for you
git branch "branch name"	Makes new branch
git checkout "branch name"	Switch to branch
git checkout -b "branch name"	Same as 2 previous commands together
git add .	Finds all changed files
git commit -m "Testing123"	Commit with message
git push origin "branch"	Push to branch
git pull origin "branch"	Pull updates from a specific branch
git commit hash (find on GitHub or run git log --oneline in the terminal), then run git revert <commit-hash> --no-edit	Undo a commit that has been pushed
git reset --soft HEAD~	Undo commit (not pushed) but keep the changes
get commit hash (run git log --oneline), then git reset --hard <commit-hash>	Undo a commit that has not been pushed and remove its changes

The Team 🎉

🎊Developers🎊

Bratesh Gupta

Julian Castellanos

Karina Gonzalez-Martinez

Laya Mallina

Safa Ahmed

🎊Project Manager🎊

Hadent Dang

🎊Industry Mentor🎊

Abis Naqvi