A chess-playing robotic arm that sees the board through cameras, thinks with the Stockfish engine, and physically moves pieces using the LeRobot SO-ARM101 arm.
About Us • Overview • Features • Hardware • Wiring Diagram • Credits
Team Name: TPC
Team Number: 18
GitHub Usernames:
- @TaniWanKenobi
- @techn1-cal
- @Ckacha
Training the arm:
training.mp4
Demo runs:
demo.mp4
demo1.mp4
Bill of Materials:
| Component | Purpose |
|---|---|
| Hugging Face LeRobot SO-ARM101 | Robotic arm that physically moves chess pieces |
| AMD PC | Runs all vision, chess engine, and arm control software |
| USB Cameras x3 | Wrist-mounted, top-down, and side-angle board views |
| Chess Board (3D printed) | Custom board sized and colored for vision detection |
| Chess Pieces (3D printed) | Custom pieces with consistent brightness for detection |
Robby Fischer is a fully autonomous chess-playing robot. A human sits down, makes their move on the physical board, and the robot handles everything else: it sees what happened, calculates the best response, and physically moves its own piece.
How it works end-to-end:
-
Board Detection: Three USB cameras capture the board simultaneously. The top-down camera is the primary source; it's warped via a 4-point perspective transform (homography) to produce a clean bird's-eye view of all 64 squares.
-
Move Detection: After the human moves, the system compares the current board image against the previous state. Each square is analyzed using a voting system across three methods: brightness threshold, edge density, and texture variance. If two of three methods agree a square changed, it's flagged, giving the system the "from" and "to" squares of the human's move.
-
Chess Engine: The detected move is validated against a tracked
chess.Boardstate using thepython-chesslibrary. If the move is legal, it's applied and Stockfish calculates the best response at depth 15, or with a 1-second time limit. -
Arm Execution: The LeRobot SO-ARM101 arm executes the engine's chosen move on the physical board, completing the loop.
Game Flow:
Camera Input → Perspective Warp → Square Occupancy Detection → Move Validation → Stockfish → Arm Execution → Repeat
Human plays White and moves first. The robot plays Black.
- Triple-Camera System: Wrist, top-down, and side-angle cameras run in parallel threads for continuous board monitoring.
- Perspective Calibration: A saved 4-point calibration (
board_calib_4pt.json) maps the physical board to a corrected 800x800 pixel top-down view on every run. - Robust Piece Detection: Occupancy uses a 3-method voting system (brightness, edge density, variance) so a single noisy reading doesn't cause false moves.
- Stockfish Integration: Full chess engine analysis with move suggestions, position evaluation, and legal move validation through
python-chess. - Live Board Visualization: OpenCV windows display the warped board with per-square occupancy indicators and an arrow overlay showing the suggested move.
- Manual Override: Keyboard controls let the operator confirm moves, enter moves manually, request hints, or start a new game at any time.
(coming soon)
This project was built with the following open-source tools and sponsors:
- Hack Club & AMD: provided the hardware and resources that made this project possible
- LeRobot (Hugging Face): SO-ARM101 robotic arm platform and control software
- Stockfish: open-source chess engine used for move analysis and decision-making
- OpenCV: computer vision library used for camera capture, perspective transforms, and board visualization
- python-chess: chess move validation and board state tracking
- Onshape: used to CAD and design the custom chess board and pieces