# 🛰️ Nexus Vision: Autonomous Terrain Intelligence **BigRock Exchange Hackathon - Official Submission** Nexus Vision is a high-performance **Dual-Engine Perception System** engineered for autonomous rover navigation. By fusing **Transformers (SegFormer)** for terrain analysis and **CNNs (YOLOv8)** for obstacle detection, the system provides a redundant, safety-critical solution for off-road robotics. --- ## 🛠️ Core Technology Stack - **Backbone A (Terrain):** SegFormer MiT-B1/B2 (Transformers) - **Backbone B (Objects):** YOLOv8 (Real-time Object Detection) - **Framework:** PyTorch, HuggingFace, & Ultralytics - **Augmentation:** Albumentations (Domain-Hardening) - **Optimization:** Vectorized NumPy Look-Up Table (LUT) - **Deployment:** Dual-Model Fusion Stream with Adaptive Path-Vector Calculation --- ## 🚀 Key Innovations ### 1. Dual-Engine Perception Fusion We implemented a multi-modal approach where SegFormer provides the "world context" (Sky, Sand, Landscape) while YOLOv8 provides the "hazard context" (Rocks, Obstacles). This redundancy ensures the robot never misses a hazard. ### 2. Transformer-Based Global Context Our SegFormer backbone utilizes global self-attention to segment the horizon with **96.01% precision**, providing a stable navigational anchor for the rover's visual odometry. ### 3. Edge-Ready Vectorized Pipeline Our inference pipeline is optimized for edge deployment using a pre-computed LUT for mask remapping, reducing latency and enabling smooth, high-FPS dual-model inference on robotics hardware. --- ## 📊 Final Performance Metrics | Class Name | IoU Score | Impact on Navigation | | :--- | :--- | :--- | | **Sky** | **96.01%** | Horizon & IMU Stabilization | | **Trees** | **65.46%** | Rigid Vertical Hazard Detection | | **Dry Grass** | **61.19%** | Navigable Free-Space Mapping | | **Landscape** | **59.28%** | Base Surface Detection | **Mean IoU:** 43.37% | **Pixel Accuracy:** 80.94% --- ## 📂 Project Structure - `live_demo.py`: **[RUN THIS]** Dual-Model Fusion demo with pathfinding. - `train.py`: Local "Super Accuracy" training script. - `evaluate_performance.py`: Generates the technical audit table. - `video_inference.py`: Batch video processing tool for demos. --- ## 💻 Setup & Usage ```bash # 1. Install dependencies pip install torch transformers albumentations opencv-python tqdm ultralytics # 2. Run the Dual-Engine Robot Intelligence Demo python3 live_demo.py ``` --- **Developed by Team Nexus Vision**# Track2_Tech-Nexus