Real-time, fully local, privacy-first spatial awareness for assistive navigation, running entirely on consumer edge hardware.
Existing assistive-vision tools emit flat, context-free labels — "Chair. Door. Bottle." — that tell you an object exists but nothing actionable about it. Knowing a chair is in the room is useless without knowing whether it is 0.8 m to your left or directly blocking your path.
Spatial AI closes that gap. From a single RGB camera stream it builds a live, metric 3D scene graph by fusing open-vocabulary object detection (YOLO-World) with dense monocular depth (Depth Anything V2), reasons about each object's position relative to the user, and uses a quantised on-device language model (Phi-3-mini INT4) to speak one short, useful navigation sentence:
"Backpack 0.9 meters directly ahead blocking your path — step right. Chair 1.4 meters on your left."
Everything runs locally. No cloud, no streaming your camera anywhere, no network dependency.
- Architecture
- How It Works
- Features
- Project Structure
- Requirements
- Installation
- Model Weights
- Usage
- Configuration
- Scene Graph Format
- Phased Roadmap
- Performance
- Known Limitations
- Testing
- Future Work
- License & Citation
┌─────────────────────────────────────────────────────────┐
│ LIVE VIDEO FRAME │
│ (Webcam / IP camera / phone) │
└──────────────────────────┬──────────────────────────────┘
│
┌─────────────┴─────────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────────┐
│ SEMANTIC BRANCH │ │ GEOMETRY BRANCH │
│ YOLO-World │ │ Depth Anything V2 │
│ (boxes + classes) │ │ (dense depth map) │
└──────────┬──────────┘ └────────────┬────────────┘
│ (parallel worker threads)│
└─────────────┬─────────────┘
▼
┌────────────────────────────────┐
│ SPATIAL SYNTHESIS ENGINE │
│ Back-projects 2D boxes into │
│ 3D camera space: │
│ P = d · K⁻¹ · [u, v, 1]ᵀ │
└────────────────┬───────────────┘
▼
┌────────────────────────────────┐
│ SCENE GRAPH ENGINE │
│ IOU tracking + EMA smoothing │
│ + dwell-filtered spatial │
│ predicates (blocking_path …) │
└────────────────┬───────────────┘
▼
┌────────────────────────────────┐
│ EDGE SLM (Phi-3-mini INT4) │
│ llama.cpp / GGUF, few-shot │
│ graph → navigation sentence │
└────────────────┬───────────────┘
▼
┌────────────────────────────────┐
│ SPATIAL AUDIO TTS │
│ Kokoro-82M / pyttsx3 │
│ + semantic repetition gate │
└────────────────────────────────┘
The two perception branches run in parallel daemon threads sharing one device pool; their latencies overlap rather than add. The SLM runs in its own thread (dispatched every N frames) so autoregressive decode never stalls the capture loop, and TTS plays non-blocking in the background.
| Stage | Module | What it does |
|---|---|---|
| 1. Capture | src/video_capture.py |
Threaded OpenCV VideoCapture; accepts a webcam index or an IP-camera/phone URL. |
| 2. Detection | src/detector.py |
YOLO-World with a runtime open-vocabulary class set (model.set_classes(...)) — zero-shot, no retraining. Auto-selects MPS → CUDA → CPU. |
| 3. Depth | src/depth_estimator.py |
Depth Anything V2 (Small) via Hugging Face transformers. Disparity output is inverted and multiplied by a calibratable --depth-scale factor to produce pseudo-metric depth. |
| 4. 3D projection | src/spatial_engine.py |
Pinhole back-projection using camera intrinsics. Depth is sampled robustly — the 20th percentile over the central 50 % of each box, so a centroid landing on background (gap in a chair, between legs) doesn't corrupt the estimate. |
| 5. Tracking | src/tracker.py |
IOU tracker with Hungarian-algorithm matching (scipy) for stable track_ids, plus EMA smoothing of 3D positions to fight depth jitter. |
| 6. Scene graph | src/scene_graph.py |
Rule-based spatial predicates relative to the user, stabilised by a dwell filter (a predicate must persist N frames before it commits) so the audio doesn't flicker. |
| 7. Language | src/slm_engine.py |
Phi-3-mini-4k-instruct INT4 GGUF via llama-cpp-python, primed with few-shot examples and a fixed system prompt. Output is sanitised and length-capped. |
| 8. Speech | src/tts_engine.py, src/change_gate.py |
Kokoro-82M neural TTS (pyttsx3 fallback), gated by a semantic-similarity check (all-MiniLM-L6-v2) so identical guidance isn't repeated within a cooldown window. |
The 3D convention is camera-centric: x = right (+), y = down (+), z = forward (+), with the user at the origin.
- Open-vocabulary detection — change the object set in a YAML file, no retraining.
- 3D positioning — every object gets
(x, y, z), distance, and bearing. - Safety-first reasoning —
blocking_pathis evaluated before all other predicates so hazards surface first. - Temporal stability — IOU tracking + EMA smoothing + predicate dwell filter eliminate flicker.
- Natural-language guidance — a small LLM turns the graph into one spoken sentence, with hallucination guards on sparse scenes.
- Non-repetitive audio — semantic change gate suppresses near-identical repeats.
- Fully local & private — no cloud calls; runs on Apple Silicon (MPS), NVIDIA (CUDA), or CPU.
- Flexible input — laptop webcam, USB camera, or a phone streaming over your LAN.
Spatial AI/
├── run_phase1.py # Phase 1 — detection only (flat labels)
├── run_phase2.py # Phase 2 — + depth & 3D projection
├── run_phase3.py # Phase 3 — + scene graph
├── run_phase4.py # Phase 4 — full pipeline (detection→SLM→TTS)
├── run_pipeline.py # convenience entry point
├── config/
│ ├── detection_classes.yaml # open-vocabulary object set
│ ├── scene_graph.yaml # predicate thresholds + tracker/dwell config
│ ├── system_prompt.txt # fixed SLM navigation prompt
│ └── camera_intrinsics.json # fx, fy, cx, cy, distortion coeffs
├── src/
│ ├── video_capture.py detector.py depth_estimator.py
│ ├── spatial_engine.py tracker.py scene_graph.py
│ ├── slm_engine.py tts_engine.py change_gate.py
│ ├── pipeline.py display.py latency_logger.py
├── tools/
│ └── calibrate_camera.py # checkerboard camera calibration
├── tests/
│ ├── test_predicates.py # spatial predicate unit tests
│ └── test_phase4.py # SLM post-processing / integration tests
├── checkpoints/slm/ # GGUF SLM weights live here
├── requirements.txt
└── spatialAI.md # full design spec / technical deep-dive
Software
- Python 3.10 or 3.11 (3.12 not supported by the pinned
llama-cpp-python) cmake+ a C/C++ toolchain (for buildingllama-cpp-python)- See
requirements.txt— PyTorch, Ultralytics, OpenCV, Transformers, llama-cpp-python, Kokoro, sentence-transformers, etc.
Hardware (any one of)
| Platform | Notes |
|---|---|
| Apple Silicon Mac (M-series) | Primary dev/test target — uses the Metal (MPS) backend. |
| NVIDIA GPU (RTX 3060+ / Jetson Orin) | CUDA backend for perception + llama.cpp. |
| x86 CPU-only | Works as a fallback; expect noticeably higher latency. |
A camera is required: built-in webcam, USB camera, or a phone running an IP-webcam app on the same network.
# 1. Clone
git clone https://github.com/RamInTech/SpatialAI.git
cd SpatialAI
# 2. Create an environment (conda or venv)
conda create -n spatial-ai python=3.11 -y
conda activate spatial-ai
# 3. Install dependencies
pip install --upgrade pip
pip install -r requirements.txtInstall llama-cpp-python with the right acceleration for your platform:
# Apple Silicon (Metal)
CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall llama-cpp-python
# NVIDIA (CUDA)
CMAKE_ARGS="-DLLAMA_CUDA=on" pip install --force-reinstall llama-cpp-python
# CPU only
pip install llama-cpp-pythonYOLO-World is fetched automatically by Ultralytics on first run (the yolov8s-worldv2.pt weights are also bundled in the repo root).
Depth Anything V2 (depth-anything/Depth-Anything-V2-Small-hf) downloads automatically from Hugging Face the first time the depth estimator runs.
SLM — download a quantised GGUF model into checkpoints/slm/:
mkdir -p checkpoints/slm
# Phi-3-mini-4k-instruct INT4 (~2.4 GB)
wget -O checkpoints/slm/Phi-3-mini-4k-instruct-q4.gguf \
https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.ggufKokoro TTS weights (~300 MB) download automatically on first use. If you'd rather skip the dependency, run with --tts-engine pyttsx3.
The full pipeline lives in run_phase4.py. Minimal run:
python run_phase4.py --slm-model checkpoints/slm/Phi-3-mini-4k-instruct-q4.ggufFull-featured run (neural TTS, scene-graph printout, latency logging, IP camera):
python run_phase4.py \
--slm-model checkpoints/slm/Phi-3-mini-4k-instruct-q4.gguf \
--tts-engine kokoro --tts-voice af_heart --tts-speed 1.1 \
--camera-id "http://192.168.1.36:8080/video" \
--imgsz 1280 --depth-scale 20.0 \
--show-graph --log-latency logs/latency.csvPress q (display window) or Ctrl-C to stop.
Earlier phases run standalone for debugging a single layer: run_phase1.py (detection), run_phase2.py (+ depth/3D), run_phase3.py (+ scene graph).
| Flag | Default | Description |
|---|---|---|
--camera-id |
0 |
Webcam index (int) or IP-camera URL (str). |
--model |
yolov8s-worldv2.pt |
YOLO-World weights. |
--imgsz |
640 |
Inference resolution for YOLO. |
--confidence |
0.35 |
Minimum detection confidence. |
--intrinsics |
config/camera_intrinsics.json |
Camera calibration JSON. |
--depth-model |
(HF default) | Override the Depth Anything V2 model ID. |
--depth-scale |
10.0 |
Multiplier converting relative depth to pseudo-metres (calibrate this!). |
--scene-config |
config/scene_graph.yaml |
Predicate thresholds + tracker config. |
--dwell-frames |
4 |
Frames a predicate must persist before committing. |
--ema-alpha |
0.35 |
EMA smoothing coefficient for 3D positions. |
--slm-model |
(none) | Path to the GGUF SLM. Omit to run perception-only. |
--slm-n-gpu-layers |
-1 |
GPU layers offloaded (-1 = all). |
--tts-engine |
pyttsx3 |
kokoro or pyttsx3. |
--tts-voice / --tts-speed |
af_heart / 1.1 |
Kokoro voice + speech rate. |
--log-latency |
(none) | CSV file for per-frame module timings. |
--show-depth |
off | Overlay a colourised depth map. |
--show-graph |
off | Print the serialised scene graph to stdout. |
--no-display |
off | Run headless (no OpenCV window). |
config/detection_classes.yaml— the open-vocabulary object set passed to YOLO-World. Edit freely; it already includes navigation hazards (stairs, curb, wet floor sign, cable on floor, glass door, …).config/scene_graph.yaml— predicate thresholds (all in metres) plus tracker (iou_threshold,max_lost,ema_alpha) anddwellsettings.config/system_prompt.txt— the fixed SLM navigation prompt (rules: name every object with its distance, give an avoidance action for blockers, end "Path ahead is clear." when nothing blocks).config/camera_intrinsics.json—fx, fy, cx, cyand distortion coefficients.
The shipped intrinsics are generic fallbacks (fx = fy = 460). For accurate distances, calibrate your camera with a checkerboard:
python tools/calibrate_camera.py \
--board-size 9x6 --square-size 0.025 \
--output config/camera_intrinsics.json --camera-id 0Predicates are assigned by rule from each object's 3D position relative to the user, in priority order (blocking_path always wins):
| Predicate | Condition (metres) |
|---|---|
blocking_path |
z < 1.5, ` |
overhead_hazard |
y < -0.5, z < 1.5 |
floor_obstacle |
y > 0.3, z < 2.5 |
to_left_of / to_right_of |
x < -0.4 / x > 0.4 |
in_front_of |
z < 2.0, ` |
near / far |
0.5 < dist < 1.5 / dist > 3.0 |
The graph is serialised to a compact, LLM-friendly block (closest object first), where each object's user-relation is derived from the same thresholds as the edge predicates — so the text can never contradict the computed graph:
SCENE_GRAPH [frame=45, t=1.50s]
OBJECTS (closest first):
backpack | 0.8m | BLOCKING PATH (centre ahead)
chair | 1.3m | clear, left side
table | 2.1m | clear, right side
END_SCENE
The SLM turns that into:
"Backpack 0.8 meters directly ahead blocking your path, step right. Chair 1.3 meters on your left, table 2.1 meters on your right."
The project was built in four phases, each with a runnable entry point:
- Phase 1 — Detection. Webcam → YOLO-World → flat labels + boxes to stdout. (
run_phase1.py) - Phase 2 — Spatial synthesis. Add parallel depth estimation and 3D back-projection. (
run_phase2.py) - Phase 3 — Scene graph. Add IOU tracking, EMA smoothing, dwell-filtered predicates, serialisation. (
run_phase3.py) - Phase 4 — SLM + TTS. Add edge LLM guidance, neural TTS, semantic change gate, latency profiling. (
run_phase4.py)
The system targets a navigation update rate of a few Hz on consumer hardware. Because the perception branches run in parallel and the SLM/TTS run off the capture thread, end-to-end latency is dominated by SLM decode rather than the sum of all stages. Per-module timings are written to CSV with --log-latency so you can profile on your own device. See spatialAI.md for detailed target benchmarks across M-series, Jetson Orin, and RTX 3060.
Note on depth: the default Depth Anything V2 Small checkpoint produces relative depth, which this pipeline scales to pseudo-metres via
--depth-scale. Absolute distances are approximate — calibrate--depth-scaleagainst a known reference, or swap in a metric checkpoint / stereo rig for true metric accuracy.
- Transparent / reflective surfaces — monocular depth fails silently on glass and mirrors, reporting the scene behind them.
- Depth scale drift — textureless walls and corridors weaken depth estimates; absolute error grows with distance.
- Low light / backlighting — detection confidence drops below ~50 lux and against bright windows.
- Dynamic occlusion — someone walking between camera and object can reset that object's track ID and smoothing state.
- SLM hallucination on sparse scenes — guarded by gating dispatch on confident detections, with a canned "path appears clear" fallback.
Full discussion and mitigations are in spatialAI.md, §8.
pytest tests/ -vtests/test_predicates.py validates spatial-predicate assignment across canonical configurations; tests/test_phase4.py covers SLM output post-processing (sanitisation, truncation) and integration.
- Stereo depth (RealSense / ZED) for true metric accuracy without scale ambiguity.
- IMU fusion to compensate the coordinate frame for head tilt.
- Fine-tuning the SLM on synthetic navigation dialogue to cut hallucination further.
- CoreML export for on-device iOS/Android deployment with spatial-audio output.
- Kalman-filter trajectory prediction to warn about approaching obstacles.
Released under the MIT License.
@software{spatial_ai_navigation,
title = {Spatial AI: Navigation for the Visually Impaired},
author = {Ramkumar M and Contributors},
url = {https://github.com/RamInTech/SpatialAI},
note = {Open-vocabulary detection (YOLO-World) + monocular depth
(Depth Anything V2) + edge SLM (Phi-3-mini INT4, llama.cpp)
+ neural TTS (Kokoro-82M). Fully local, no cloud dependency.}
}For the complete design rationale, mathematical derivations, and benchmark tables, see spatialAI.md.