README.md

🏜️ Off-Road Semantic Segmentation — Track 2 | Loop Legends

A semantic segmentation pipeline for off-road autonomous navigation, built on top of a DINOv2 vision backbone with a custom lightweight segmentation head. Developed as part of a hackathon challenge focused on robust perception for off-road environments.

---

📌 Overview

This project tackles the challenge of pixel-wise semantic segmentation in unstructured off-road terrain — environments with highly variable lighting, vegetation density, and terrain types. The model classifies every pixel in an image into one of 11 semantic classes critical for off-road autonomy.

---

🗂️ Project Structure

track_2_loop_legends/
├── Offroad_Segmentation_Scripts/
│   ├── train_segmentation.py       # Main training script
│   ├── test_segmentation.py        # Evaluation script (computes per-class & mean IoU)
│   ├── visualize.py                # Colorizes segmentation masks for visualization
│   ├── best_model.pth              # Best model checkpoint (by val IoU)
│   └── checkpoint_epoch_6.pth      # Mid-training checkpoint for resume
│
├── Offroad_Segmentation_Training_Dataset/
│   ├── train/
│   │   ├── Color_Images/           # RGB training images
│   │   └── Segmentation/           # Ground truth masks
│   └── val/
│       ├── Color_Images/
│       └── Segmentation/
│
└── Offroad_Segmentation_testImages/
    ├── Color_Images/               # Test RGB images
    └── Segmentation/               # Test ground truth masks

---

🧠 Model Architecture

Component	Details
Backbone	DINOv2 (ViT-based, self-supervised pre-trained)
Segmentation Head	Custom lightweight MLP decoder
Backbone Unfreezing	Last 4 transformer blocks fine-tuned
Loss Function	Combined CrossEntropy + Dice Loss (α=0.3)
Optimizer	AdamW
Scheduler	CosineAnnealingLR

---

🏷️ Segmentation Classes

ID	Class	Notes
0	Background
1	Trees
2	Lush Bushes
3	Dry Grass
4	Dry Bushes
5	Ground Clutter
6	Flowers	Rare class
7	Logs	Rare class
8	Rocks
9	Landscape	Dominant class
10	Sky	Dominant class

Class imbalance is handled with weighted loss — rare classes (Flowers, Logs) receive up to 5× weight, while dominant classes (Sky, Landscape) receive reduced weight.

---

⚙️ Setup

Prerequisites

• Python 3.10+
• PyTorch with CUDA support
• Conda (recommended)

Installation

# Create and activate environment
conda create -n offroad_seg python=3.10 -y
conda activate offroad_seg

# Install PyTorch with CUDA
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# Install dependencies
pip install tqdm opencv-python matplotlib pillow timm

---

🚀 Training

python train_segmentation.py

To resume from a checkpoint (e.g., epoch 6):

The script automatically detects checkpoint_epoch_6.pth and resumes training. Set n_epochs in the script to control total epochs.

Key hyperparameters:

lr            = 1e-4       # Segmentation head learning rate
backbone_lr   = 1e-5       # Backbone learning rate (lower to preserve pre-trained features)
n_epochs      = 8
alpha         = 0.3        # Weight of CrossEntropy in combined loss (Dice gets 0.7)

---

🧪 Evaluation

python test_segmentation.py \
  --model_path "Offroad_Segmentation_Scripts/best_model.pth" \
  --data_dir "Offroad_Segmentation_testImages" \
  --output_dir "./predictions"

This outputs:

• evaluation_metrics.txt — mean IoU and per-class IoU breakdown
• Predicted mask images saved to --output_dir

Results on validation set:

Metric	Score
Mean IoU (val)	~0.46
Sky IoU	0.93
Landscape IoU	0.53
Mean IoU (test)	0.17

Note on domain gap: The test set images originate from a visually distinct desert environment compared to training data. The gap between val IoU (0.46) and test IoU (0.17) is a known challenge in off-road perception — models trained on one environment can struggle to generalize to unseen terrain. Future work includes domain randomization and environment-agnostic feature learning.

---

🎨 Visualization

To generate colorized segmentation masks:

python visualize.py

Set input_folder inside the script to point to your segmentation mask directory. Colorized outputs are saved to a colorized/ subfolder.

---

📊 Training Strategy

• Phase 1: Train segmentation head only (backbone frozen) for initial epochs
• Phase 2: Unfreeze last 4 backbone blocks, fine-tune end-to-end with lower backbone LR
• Checkpointing: Best model saved whenever validation IoU improves; epoch checkpoints saved for resume

---

👥 Team

Loop Legends — VectorFlow, VVCE

---

📄 License

This project was developed for hackathon purposes. Dataset and base model weights belong to their respective owners.