SeniorVis

Drone-based intelligent monitoring system combining two independent computer vision pipelines in a single Android application:

• Solar panel anomaly detection — YOLOv8s, 7 fault classes, thermal aerial imagery
• Tree counting — YOLO detection + SAHI panorama inference + SIFT Re-ID deduplication across video frames

The user uploads imagery or video from the mobile app → inference runs automatically on the edge server → an LLM (Gemma via Ollama) generates a Turkish-language analysis report → results stream back to the app in real time via Firebase.

---

System Architecture

Mobile App (Android)
        │
        │  Upload image / video
        ▼
Firebase Storage  ──►  Firestore (status: "pending")
                                │
                                ▼
                        listener.py  (edge server)
                                │
                        master_pipeline.py
                         ┌──────┴──────┐
                   Solar Model     Tree Algorithms
                   (anomaly)       (SAHI + SIFT)
                         └──────┬──────┘
                                │
                         Ollama LLM (Gemma)
                         Turkish report generation
                                │
                                ▼
                   Firebase Storage + Firestore
                                │
                                ▼
                   Mobile App (real-time update)

---

Models

Model	Task	mAP50	mAP50-95	Precision	Recall
YOLOv8s	Solar panel anomaly (7 classes)	0.730	0.528	0.702	0.749
yolo26s	Tree detection (1 class)	0.994	0.826	0.990	0.986

Solar Fault Classes

Class	Annotations	Share
Cell	3,625	5.7%
Cell (Foreign Object)	453	0.7%
Multi-Cell	1,722	2.7%
Hot Spot	17,500	27.6%
Multi Hot Spot	5,305	8.4%
Open Circuit String	12,856	20.3%
Diode	22,014	34.7%

---

Project Structure

SeniorDesignProject/
├── src/
│   ├── train.py                             # Tree model training
│   ├── optimized_solar_anomaly_detector.py  # Solar inference pipeline
│   ├── optimized_sahi_counter.py            # SAHI panorama tree counting
│   ├── optimized_sift_reid.py               # SIFT Re-ID tree counting
│   ├── sahi_counter.py                      # SAHI base implementation
│   ├── counter_updated.py                   # Line-crossing video counter
│   ├── master_pipeline.py                   # Main orchestrator
│   ├── listener.py                          # Firestore-triggered listener
│   └── split_dataset.py                     # Dataset split utility
├── App/                                     # Android app (Kotlin / Compose)
├── results/
│   ├── solar_yolov8s/                       # Solar model training outputs
│   │   ├── weights/best.pt
│   │   ├── results.csv
│   │   ├── args.yaml
│   │   └── *.png / *.jpg
│   └── tree_detection_v2/                   # Tree model training outputs
│       ├── weights/best.pt
│       ├── results.csv
│       ├── args.yaml
│       └── *.png / *.jpg
├── report.tex                               # IEEE-format technical paper (LaTeX)
└── presentation.md                          # Jury presentation package (4 speakers)

---

Setup

Requirements

pip install ultralytics opencv-python firebase-admin numpy requests

Firebase Configuration

Place serviceAccountKey.json in the src/ directory (Firebase Console → Project Settings → Service Accounts).

Start the Server Listener

cd src
python listener.py

While listener.py is running it automatically picks up upload triggers from Firestore and invokes master_pipeline.py.

Manual Invocation

# Solar only
python master_pipeline.py --solar

# Tree counting only
python master_pipeline.py --tree

# Both
python master_pipeline.py --solar --tree

Model Weights

Model	Path
Solar	results/solar_yolov8s/weights/best.pt
Tree	results/tree_detection_v2/weights/best.pt

---

Tree Counting Algorithms

SAHI (Panorama-Based)

1. Drone U-turns are detected via SIFT homography odometry (threshold: 135° accumulated yaw)
2. Frames from each straight-flight segment are stitched into a panorama
3. YOLO runs on 1024×1024 sliding windows (25% overlap) over the panorama
4. Cross-patch NMS (IoU: 0.3) removes duplicate detections
5. Per-segment counts are summed

SIFT Re-ID (Video-Based)

1. Trees are tracked per-frame with ByteTrack
2. Every 10 frames, 3,000 SIFT features are extracted per detected crop
3. After a U-turn, each new detection is matched against pre-turn keyframes using Lowe's ratio test (r < 0.75) with a minimum of 8 good matches required to merge
4. A Union-Find structure maintains the unique tree count

---

Mobile Application

Language: Kotlin · UI: Jetpack Compose · Architecture: MVVM + StateFlow

Screens:

• LoginScreen / RegisterScreen — Firebase Auth (email/password)
• DashboardScreen — Solar tab (heatmaps + detection list) and Tree tab (SIFT video + SAHI panoramas)
• ReportScreen — LLM-generated Markdown report
• SettingsScreen

Upload flow: User selects file → uploaded to Firebase Storage → AnalysisTrigger doc written to Firestore with status: "pending" → listener.py picks it up → results written back to Firestore → app updates automatically.

---

LLM Report Generation

After inference completes, master_pipeline.py sends the numerical results to a locally running Ollama instance (http://localhost:11434). Model: gemma4:e4b, language: Turkish, format: Markdown.

Report sections:

• If tree counting ran: comparison of SIFT and SAHI method results
• If solar analysis ran: panels scanned, anomalies detected, maintenance recommendations
• Conclusion and action plan

---

Documentation

• report.tex — Complete IEEE two-column conference paper
• presentation.md — 4-speaker, 16-slide jury presentation package with speaker scripts