Multi-scenario Overlapping Text Segmentation with Depth Awareness

Official PyTorch implementation of the ICCV paper "Overlapping Text Segmentation and Recognition". This codebase is built on top of the MMSegmentation framework.

Introduction

Overlapping text presents severe challenges for open-scene text perception tasks, while existing methods are mostly limited to document scenarios. To address this, we propose a multi-scenario overlapping text segmentation task and build a real English-Chinese dataset covering diverse scenes. We further introduce a hierarchical training data synthesis strategy to improve model generalization. Moreover, we utilize depth maps to provide 3D relative position cues and design a depth-guided decoder that fuses image and depth features to capture complex overlapping interactions between text instances.

News

• 2026.6.23 🚀MonkeyOCRv2-AS is extended to Overlapping Text Segmentation based on our framework.

Installation

We recommend using Conda to set up the environment, Please refer to MMSegmentation Install Guide for more detailed instruction.

pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cu121
pip install -U openmim
mim install mmengine
pip install --no-build-isolation mmcv==2.2.0 -f https://download.openmmlab.com/mmcv/dist/cu121/torch2.5.0/index.html
pip install -v -e .
pip install transformers==4.51.0 accelerate
pip install mmpretrain mmdet
pip install ftfy

MOT Dataset and Weights

• Download MOT dataset from BaiduNetCloud.

• Download MonkeyOCRv2-AS-MOTS from BaiduNetCloud.

Traning

Training on a single GPU, please use

python tools/train.py  ${CONFIG_FILE} [optional arguments]

Training on multiple GPUs, please use

sh tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]

For example, we use this script to train the model:

sh tools/dist_train.sh configs/overlap/mots_overlaptext.py 8

To train or evaluate MonkeyOCRv2-AS based on our framework. place its weight path in the configs/overlap/mots_monkeyvit_overlaptext.py.

Evaluation

Testing on a single GPU, please use

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Training on multiple GPUs, please use

sh tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [optional arguments]

For example, we use this script to train the model:

sh tools/dist_test.sh configs/overlap/mots_overlaptext.py 8

Visualization

Citation

Please cite the following paper when using the MOT dataset or this repo.

@inproceedings{liu2025multi,
  title={Multi-scenario Overlapping Text Segmentation with Depth Awareness},
  author={Liu, Yang and Xie, Xudong and Liu, Yuliang and Bai, Xiang},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={17454--17463},
  year={2025}
}

Acknowledgement

This repo is based on MMSegmentation. We appreciate this wonderful open-source toolbox.