Skip to content

Commit 5506700

Browse files
author
Felix
committed
initial commit
Update README.md include submodule for nuscenes remove NRF from readme remove nf and fish eye data leftovers add subfolder for network move submodule update readme Update README.md Create LICENSE Add files via upload Update README.md Fixes for Docker usage Update readme
1 parent 979bb5f commit 5506700

74 files changed

Lines changed: 18788 additions & 166 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
2+
############## MEDIA AND DATA FILES ##############
3+
*.pdf
4+
*.png
5+
*.jpg
6+
*.json
7+
*.csv
8+
*.mp4
9+
10+
############## COMPILED FILES ##############
11+
*.o
12+
*.so
13+
*.pyc
14+
*.pyd
15+
16+
############## MACHINE SPECIFIC ##############
17+
.vscode/
18+
.idea/
19+
.eggs/
20+
*.egg-info/
21+
.DS_Store
22+
23+
############## LOCAL FILES ##############
24+
crfnet/playground/
25+
crfnet/snapshots/
26+
crfnet/out/
27+
crfnet/feature_visualization/
28+
crfnet/tb_logs/*
29+
crfnet/saved_models/*
30+
crfnet/data/tools/anchor_data/
31+
crfnet/build/
32+
crfnet/archive/
33+
crfnet/experiments/
34+
35+
############## CONFIG FILES ############
36+
crfnet/configs/*
37+
crfnet/configs/neural_filter/*
38+
39+
############## Exceptions ##############
40+
!crfnet/configs/neural_filter
41+
!crfnet/configs/default.cfg
42+
!crfnet/configs/neural_filter/default.json.example
43+
!crfnet/configs/README.md
44+
!crfnet/images/*.png
45+
!crfnet/tb_logs/.gitkeep
46+
!crfnet/saved_models/.gitkeep
47+
*.code-workspace

‎.gitmodules‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "nuscenes-devkit"]
2+
path = crfnet/nuscenes-devkit
3+
url = https://github.com/Fellfalla/nuscenes-devkit.git

‎Dockerfile‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
2+
# Build the docker with
3+
# > docker build . -t crfnet
4+
# You can start your docker with
5+
# > nvidia-docker run -it --net=host -e NVIDIA_VISIBLE_DEVICES=0 -e USERID=$UID -v ($HOME)/data:/data --name crfnet_gpu0 crfnet
6+
7+
FROM nvidia/cuda:10.0-cudnn7-runtime-ubuntu16.04
8+
# Setting locale to US prevents apt package server connection error during python installation. Taken from: https://github.com/CPJKU/madmom/issues/390
9+
# only necessary if not done on machine before for other software
10+
RUN export LC_ALL=en_US.UTF-8
11+
12+
RUN apt-get update && apt-get install -y apt-transport-https
13+
RUN apt-get install python3.5 python3-pip python3-tk python-opencv git -y --no-install-recommends apt-utils python3-dev build-essential
14+
RUN apt-get install -y graphviz
15+
RUN alias python="python3.5"
16+
RUN alias pip="pip3"
17+
18+
# pulling nuscenes fork
19+
RUN pip3 install --upgrade pip
20+
RUN pip3 install --upgrade setuptools
21+
RUN pip3 install --upgrade Cython
22+
23+
# Clone and install the crfnet repository
24+
COPY . /CRFN
25+
WORKDIR /CRFN/crfnet
26+
RUN pip3 install -e .
27+
RUN python3 setup.py build_ext --inplace

‎LICENSE‎

Lines changed: 201 additions & 165 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 121 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,121 @@
1-
# CameraRadarFusionNet
1+
# CRF-Net for Object Detection (Camera and Radar Fusion Network)
2+
![Radar Augmented Image Example](crfnet/images/imageplus_format.png)
3+
4+
This repository provides a neural network for object detection based on camera and radar data. It builds up on the work of [Keras RetinaNet](https://github.com/fizyr/keras-retinanet).
5+
The network performs a multi-level fusion of the radar and camera data within the neural network.
6+
The network can be tested on the [nuScenes](https://www.nuscenes.org/) dataset, which provides camera and radar data along with 3D ground truth information.
7+
8+
## Requirements
9+
- Linux Ubuntu (tested on versions 16.04 and 18.04)
10+
- CUDA 10.0
11+
- Python 3.5
12+
- Docker 19.03 (only if usage in Docker is desired)
13+
- NVIDIA Container Toolkit (nvidia docker)
14+
15+
## Installation
16+
Install system wide packages:
17+
* `sudo apt install graphviz`
18+
19+
All required python packages can be installed with the crfnet pip package. We recommend to install the package in its own virtual environment. To install, run the following commands in the repository folder (crfnet).
20+
21+
* `pip install -e .`
22+
* `python setup.py build_ext --inplace`
23+
24+
# CRF-Net Usage
25+
The network uses camera and radar inputs to detect objects. It can be used with the nuScenes dataset and extended to other radar and camera datasets. The nuScenes dataset can be downloaded [here](https://www.nuscenes.org/download).
26+
Pretrained weights are provided [here](https://syncandshare.lrz.de/dl/fi9RrjqLXyLZFuhwjk9KiKjc/crf_net.h5 ) (270MB).
27+
## Start Training
28+
1. Create your desired [configuration](crfnet/configs/README.md) for the CRF-Net. Start by making a copy of the [default_config](crfnet/configs/default.cfg)
29+
2. Execute `python train_crfnet.py`. This will train a model on a given dataset specified in the configs. The result will be stored in [saved_models](crfnet/saved_models) and the logs in [tb_logs](crfnet/tb_logs).
30+
* `--config <path to your config>` to use your config. Per default the config file found at [./configs/local.cfg](crfnet/configs/local.cfg) is used.
31+
32+
Example usage:
33+
```bash
34+
python train_crfnet.py --config configs/crf_net.cfg
35+
```
36+
37+
## Evaluate Model
38+
1. Execute `python evaluate_crfnet.py` to calculate the precision and recall values for a model on the data specified
39+
in the config file. The values and curves are saved onto the hard drive.
40+
* `--config <path to your config>` show the path of training config.
41+
* `--model <path to model>` model file saved from prior training
42+
* `--st <score trehshold>` select a custom threshold at which predictions are considered as positive.
43+
* `--render` to show images with predicted bounding boxes during execution
44+
* `--eval_from_detection_pickle` to load saved detection files from the hard drive instead of running the model to
45+
evaluate it.
46+
47+
Example usage:
48+
```bash
49+
python evaluate_crfnet.py --model saved_models/crf_net.h5 --config configs/crf_net.cfg --st 0.5
50+
```
51+
52+
## Test Model
53+
1. Execute `python test_crfnet.py` to run a inference model on the data specified in the config file.
54+
* `--config <path to your config>` show the path of training config.
55+
* `--model <path to model>` model file saved from prior training
56+
* `--st <score trehshold>` select a custom threshold at which predictions are considered as positive.
57+
* `--render` to show images with predicted bounding boxes during execution
58+
* `--no_radar_visualization` suppresses the radar data in the visualization
59+
* `--inference` to run the network on all samples (not only the labeled ones, only TUM camra dataset)
60+
61+
Example usage:
62+
```bash
63+
python test_crfnet.py --model saved_models/crf_net.h5 --config configs/crf_net.cfg --st 0.5
64+
```
65+
66+
## Folder Structure
67+
68+
All main scripts depend on the following subfolders:
69+
70+
1. [configs](crfnet/configs/README.md) folder contains the config files. All the settings for the main scripts are described in this file. A default.cfg (configs/default.cfg) shows an exemplary config file. It is recommended to give each configuration a unique name, as models and tensorboard logs are also named after it.
71+
72+
2. [data_processing](crfnet/data_processing/README.md) folder contains all functions for preprocessing before the neural network. Data fusion functions are placed in this folder. VIsualization of the generated radar augmented image (RAI) is provided by the generator script for the corresponding data set.
73+
74+
3. [model](crfnet/model/README.md) folder contains all the neural network models that can be used. Based on RetinaNet, architectures that fuse camera and radar data at the feature extractor or the FPN are stored here.
75+
76+
4. [utils](crfnet/utils/README.md) utils folder contains helper functions that are needed in many places in this repository.
77+
78+
79+
## Files
80+
| File | Description |
81+
|----|----|
82+
train_crfnet.py | Used to train the CRF-Net.
83+
evaluate_crfnet.py | Used to evaluate a trained CRF-Net model on the validation set.
84+
test_crfnet.py | Used to test a trained CRF-Net model on the test set. This script can be used to record videos.
85+
Dockerfile | Dockerfile to setup the environment.
86+
requirements.txt | Contains the requirements for the scripts in this repository
87+
setup.py | Installs the requirements for this repository and registers this repository in the python modules
88+
89+
## Usage Docker Image
90+
The use of the docker image is optional. It is tested on a Ubuntu 16.04 host system.
91+
92+
1. Build the docker container via
93+
94+
`docker build --tag crfnet .`
95+
96+
2. Start the docker using GPU 0 and the container name `crfnet_gpu0` via
97+
98+
1. Specify dataset dir: `export HOST_DATA_DIR=<HOST_DATA_DIR>`
99+
2. Specify log dir: `export HOST_LOG_DIR=<HOST_LOG_DIR>`
100+
3. Specify saved models dir: `export HOST_MODEL_DIR=<HOST_MODEL_DIR>`
101+
5. Start container: `docker run -it --net=host --gpus '"device=0"' -e USERID=$UID -v $HOST_DATA_DIR:/data/nuscenes -v $HOST_LOG_DIR:/CRFN/crfnet/tb_logs -v $HOST_MODEL_DIR:/CRFN/crfnet/saved_models -e DISPLAY=$DISPLAY --name crfnet_gpu0 crfnet`
102+
103+
104+
3. The repository is located at `/CRFN/crfnet` inside the docker and already installed when building the docker. Inside the docker, you start the training with python3 and specify your config file as usual: `python3 train_crfnet.py --config configs/crf_net.cfg`
105+
Additionally you can connect to it via `docker attach crfnet_gpu0`
106+
107+
# Contributions
108+
[1] M. Geisslinger, "Autonomous Driving: "Object Detection using Neural Networks for Radar and Camera Sensor Fusion," Master's Thesis, Technical University of Munich, 2019
109+
110+
[2] M. Weber, "Autonomous Driving: Radar Sensor Noise Filtering and Multimodal Sensor Fusion for Object Detection with Artificial Neural Networks," Master’s Thesis, Technical University of Munich, 2019.
111+
112+
If you find our work useful in your research, please consider citing:
113+
114+
@INPROCEEDINGS{nobis19crfnet,
115+
author={Nobis, Felix and Geisslinger, Maximilian and Weber, Markus and Betz, Johannes and Lienkamp, Markus},
116+
title={A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection},
117+
booktitle={2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF)},
118+
year={2019},
119+
doi={10.1109/SDF.2019.8916629},
120+
ISSN={2333-7427}
121+
}

‎crfnet/LICENSE‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Copyright 2019 Chair of Automotive Technology, Technical University of Munich
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.

‎crfnet/__init__.py‎

Whitespace-only changes.

‎crfnet/configs/README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
## Configs
2+
This config folder holds the configuration files that are needed for training, testing and evaluation.
3+
A description of all configuration parameters if given below.
4+
5+
| Parameter | Description | Example / Range|
6+
|---|---|---|
7+
| **Path** |
8+
|save_model | Path to save the model | ./saved_models/ |
9+
|load_model | Path to load a model to continue training on it| empty or ./saved_models/model1.h5 e.g. |
10+
| **Data** |
11+
| data_set | Whether nuscenes, camra or coco for the specific data sets | nuscenes |
12+
| data_path | Path to the data set | ~/data/nuscenes |
13+
| save_val_img_path | Path so save evaluated validation images after every epoch | True / False |
14+
| n_sweeps | Number of radar time steps used | 1 - 26 |
15+
| radar_projection_height | Height of projected radar lines in m | 0.01 -> points, 1000 -> "barcode", or meters in between |
16+
| noise_filter_perfect | Perfect noise filter based on ground truth | True / False|
17+
| radar_filter_dist | Filter radar data from a certain distance in m | 100 |
18+
| scene_selection | Define validtion and test set| default or debug |
19+
| **Tensorboard** |
20+
| tensorboard | True if tensorboard logs should be saved | True/False |
21+
| logdir | Path to save tensorboard log files | ./tb_logs/ |
22+
| **Computing** |
23+
| seed | Random seed to perform training | e.g. 0 |
24+
| gpu | Integer that specifies the system's GPU to run the training on | e.g. 0 |
25+
| gpu_mem_usage | Proportion (between 0 and 1)of GPU memory that should be used | e.g. 0.5 |
26+
| workers | Number of threads for generating data during training and evaluation | e.g. 4 |
27+
| **Preprocessing** |
28+
| normalize_radar | True if radar data should be normalized | True/False |
29+
| random_transform | True for extended data augmentation with rotation, shear etc. | True/False|
30+
| sample_selection | True to exclude samples without any objects from training | True/False|
31+
| only_radar_annotated | Only keep bounding boxes that have according radar points| 1 for nuScenes method, 2 for points_in_box method |
32+
| noisy_image_method | Generate noisy image in data generator| poisson, gauss, s&p-perpixel, blurr|
33+
| noise_factor | Degree of noise, highly depends on the method | e.g. 0.2 or 1e-4 |
34+
| **Hyperparameters**|
35+
| learning_rate | Learning rate for training | e.g. 1e-4 |
36+
| batchsize | Batch size used for training | e.g. 1|
37+
| epochs | Number of epochs for training | e.g. 50 |
38+
| weighted_map | True if mAP should be calculated weighted | True/False |
39+
| category_mapping | Categories (classes) to be used or merged | see default.cfg|
40+
| class_weights | Class weights for imbalanced classes | see default.cfg |
41+
| **CRF-Net** |
42+
| channels | Input channels (RGB + Radar) according to nuscenes| e.g. 0,1,2,18 (encoding below)|
43+
| image_height | Height of the input image in pixels | e.g. 360 |
44+
| image_width | Width of the input image in pixels | e.g. 640 |
45+
| dropout_radar | Chance that a sample has no radar data during training | 0 - 1 e.g. 0.2|
46+
| dropout_image | Chance that a sample has no image data during training | 0 - 1 e.g. 0.2|
47+
| network | Feature extractor network | e.g. vgg-max-fpn or resnet101 |
48+
| network width | Width factor of neural network to adapt number of kernels | e.g. 0.5 or 1.5 |
49+
| pooling | Pooling in radar branch | max, min or conv |
50+
| anchor params | default or small for different anchor sizes | default or small |
51+
| pretrain_basenet | True if feature extractor should initialized with ImageNet weights | True/False |
52+
| distance_detection | True if distances should be predicted (by an extra loss function) | True/False |
53+
| distance_alpha | Weight factor for distance loss | e.g. 10 |
54+
| class_specific_nms | True if NMS should be specific to classes | True/False |
55+
| score_thresh_train | Score trehsold, from which detections count as positive | e.g. 0.05 |
56+
57+
58+
59+
60+
61+
### Radar Augmented Image Channels
62+
63+
| Channel ID | Description |
64+
|---|---|
65+
|0 | R-Channel (Image)|
66+
|1 | G-Channel (Image)|
67+
|2 | B-Channel (Image)|
68+
|3 | dyn_prop|
69+
|4 | id|
70+
|5 | rcs|
71+
|6 | vx|
72+
|7 | vy|
73+
|8 | vx_comp|
74+
|9 | vy_comp |
75+
|10 | is_quality_valid |
76+
|11 | ambig_state |
77+
|12 | x_rms |
78+
|13 | y_rms |
79+
|14 | invalid_state |
80+
|15 | pdh0 |
81+
|16 | vx_rms |
82+
|17 | vy_rms |
83+
|18 | distance |

‎crfnet/configs/default.cfg‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
; config.ini
2+
[PATH]
3+
save_model = ./saved_models/
4+
load_model =
5+
6+
[DATA]
7+
data_set = nuscenes
8+
data_path = /data/nuscenes
9+
save_val_img_path =
10+
11+
n_sweeps = 13
12+
radar_projection_height = 3
13+
14+
noise_filter_model =
15+
noise_filter_cfg =
16+
noise_filter_threshold =
17+
noise_filter_perfect = False
18+
radar_filter_dist =
19+
20+
scene_selection = default
21+
22+
[TENSORBOARD]
23+
tensorboard = True
24+
logdir = ./tb_logs/
25+
histogram = False
26+
27+
[COMPUTING]
28+
seed = 0
29+
gpu = 0
30+
gpu_mem_usage =
31+
workers = 6
32+
33+
[PREPROCESSING]
34+
normalize_radar = False
35+
random_transform = False
36+
37+
sample_selection = False
38+
only_radar_annotated = 0
39+
40+
noise_factor = 0.0
41+
noisy_image_method =
42+
43+
[HYPERPARAMETERS]
44+
learning_rate = 2e-5
45+
batchsize = 1
46+
epochs = 20
47+
weighted_map = True
48+
49+
50+
[CATEGORY_MAPPING]
51+
vehicle.car = vehicle.car
52+
vehicle.motorcycle = vehicle.motorcycle
53+
vehicle.bicycle = vehicle.bicycle
54+
vehicle.bus = vehicle.bus
55+
vehicle.truck = vehicle.truck
56+
vehicle.emergency = vehicle.truck
57+
vehicle.trailer = vehicle.trailer
58+
human = human
59+
60+
[CLASS_WEIGHTS]
61+
vehicle.car = 1
62+
vehicle.motorcycle = 39
63+
vehicle.bicycle = 42
64+
vehicle.bus.rigid = 30
65+
vehicle.truck = 6
66+
vehicle.trailer = 20
67+
human.pedestrian.adult = 2
68+
69+
[CRF-Net]
70+
channels = [0,1,2,5,18]
71+
image_height = 360
72+
image_width = 640
73+
74+
dropout_radar = 0.0
75+
dropout_image = 0.2
76+
77+
network = vgg-max-fpn
78+
network_width = 1.0
79+
pooling = max
80+
81+
anchor_params = small
82+
83+
pretrain_basenet = True
84+
85+
distance_detection = False
86+
distance_alpha = 10.0
87+
88+
class_specific_nms = False
89+
score_thresh_train = 0.05

0 commit comments

Comments
 (0)