MultiBodyCuboids: Multi-body Segmentation and Motion Evaluation

MultiBodyCuboids: Multi-body Segmentation and Motion Evaluation by Unsupervised Cuboid Shape Abstraction Master Thesis — National Yang Ming Chiao Tung University (NYCU), 2022 Advisor: Prof. I-Chen Lin

---

🔥 Highlights

• 📦 Unsupervised cuboid-based shape abstraction — no manual part annotation needed
• 🔄 Handles arbitrary number of disordered point cloud sets
• ⚡ Outperforms MultiBodySync on SAPIEN benchmark (+2.3% mIoU, +0.04 RI)
• 🏎️ Full 4-iteration pipeline runs in 3.4s (vs. 3.5s MultiBodySync, 11.5s DeepPart)
• 🧩 Supports both articulated objects and solid rigid bodies

---

📊 Results

SAPIEN Dataset (Articulated Objects)

Method	Multi-Scan mIoU	Multi-Scan RI	Single-Scan mIoU	Single-Scan RI
PointNet++	47.5	0.62	51.2±12.1	0.65±0.09
DeepPart	49.2	0.64	53.0±8.9	0.67±0.06
MeteorNet	43.7	0.59	45.7±5.4	0.60±0.03
NPP	48.2	0.63	51.5±6.6	0.66±0.05
MultiBodySync	66.7	0.76	67.3±4.3	0.77±0.03
Ours (3 views)	68.5	0.78	68.0±3.3	0.77±0.03
Ours (4 views)	69.0	0.80	68.2±4.5	0.78±0.03

DynLab Dataset (Solid Objects)

Method	Multi-Scan mIoU	Multi-Scan RI	Single-Scan mIoU	Single-Scan RI
PointNet++	37.2	0.53	39.4±7.1	0.54±0.03
MultiBodySync	90.7	0.95	94.0±3.1	0.96±0.02
Ours (4 views)	90.8	0.97	94.2±4.3	0.96±0.03

---

🏗️ Architecture

Our network consists of two stages:

Stage 1 — Flow Evaluation Subnet (netflow + netconf)

• Estimates pairwise 3D scene flow between point clouds
• Confidence-weighted permutation synchronization for multi-view consistency
• Inspired by OANet for inlier probability estimation

Stage 2 — Motion Segmentation Subnet (netcub)

• Decomposes point cloud pairs into segments covered by compact cuboids
• Variational autoencoder (VAE) with iterative depth declaration network
• Modified Kuhn-Munkres (KM) weighted loss for robust synchronization
• Weighted Kabsch algorithm for absolute motion estimation

---

⚙️ Setup

# Create environment
conda create -n mbs python=3.8
conda activate mbs

# Install PyTorch
conda install pytorch==1.6.0 cudatoolkit=10.2 -c pytorch

# Install dependencies
pip install -r requirements.txt

For domestic users: consider using a mirror if there are connection problems.

---

📁 Dataset

Each dataset is organized in the following structure:

<dataset-name>/
    ├ meta.json
    └ data/
        ├ 000000.npz
        ├ 000001.npz
        └ ...

After downloading, set paths in the corresponding config/*.yaml files.

Split	Type	Download
Train+Val	Articulated (mbs-shapepart)	Google Drive
Test	Articulated (mbs-sapien)	Google Drive
Train+Val	Solid (mbs-shapewhole)	Google Drive
Test	Solid (mbs-dynlab)	Google Drive

---

🚀 Training

We suggest training the flow and mot networks simultaneously, then training the conf network after flow is fully converged.

# Stage 1: Flow evaluation
python train.py config/articulated-flow.yaml

# Stage 2: Motion segmentation
python train.py config/articulated-mot.yaml

# Stage 3: Confidence network (after flow converges)
python train.py config/articulated-conf.yaml

# End-to-end fine-tuning
python train.py config/articulated-full.yaml

For solid objects: replace articulated → solid in all config paths.

---

🧪 Testing

python test.py config/articulated-full.yaml

Pre-trained Models

Download weights and extract to ./ckpt/articulated-full/best.pth.tar:

Object Type	Download
Articulated	Google Drive
Solid	Google Drive

For solid objects: %s/articulated/solid/g

---

🔑 Key Contributions

1. End-to-end trainable architecture for movement segmentation through cuboid-based shape abstraction
2. Confidence-weighted loss functions to improve accuracy and robustness over baseline Hungarian algorithm
3. Cross-dataset generalization: trained on SAPIEN, generalizes to ShapeNet and Deformable-3D-Reconstruction datasets without retraining
4. Multi-scan consistency: weighted synchronization ensures label consistency across all input point clouds

---

📎 Citation

@mastersthesis{wu2022multibodycuboids,
  title  = {MultiBodyCuboids: Multi-body Segmentation and Motion Evaluation
            by Unsupervised Cuboid Shape Abstraction},
  author = {Wu, Cheng-Rung},
  school = {National Yang Ming Chiao Tung University},
  year   = {2022},
  month  = {November},
  advisor = {Lin, I-Chen}
}

---

👤 Author

ChengRung Wu (吳承融) 📧 wu840407@gmail.com 🔗 LinkedIn 🐙 GitHub