RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems

📰 NEWS

---

• [Coming Soon] Updated version data/weights — Full RoleRMBench datasets and trained checkpoints will be released soon.
• [Released] code/results — Core experimental results and evaluation scripts are now available.

📖 Overview

Most existing reward models are tuned for relatively objective tasks (reasoning, coding, factual QA). When used to judge profile-based role play, they struggle with nuances like narrative flow, persona fidelity, and engagement, often behaving little better than random choice. This makes them unreliable both as evaluators and as optimization targets for role-playing agents.

This repository provides:

• RoleRMBench – A systematic benchmark for role-play reward modeling, covering seven fine-grained capabilities (narrative, role consistency, safety, multi-turn coherence, attractiveness, etc.).
• RoleRM – A specialized reward model trained on human-ranked role-play data with Continuous Implicit Preferences (CIP), using structured pairwise supervision to better capture nuanced, persona-grounded preferences.
• Code & Evaluation Pipeline – Scripts to reproduce our main results and to plug RoleRM / RoleRMBench into your own role-playing scenario and alignment workflows.

Results

Task Definition

🚀 Quickstart

📦 Installation

Eval

Create and activate the conda environment:

conda create -n eval python=3.10
pip install -r requirements.txt
conda activate eval

Evaluate your Reward Model with scripts from /eval_code .

Train

We follow the classic RM training method provided in OpenRLHF

✍️ Citation

If you use our work or are inspired by our work, please consider cite us:

@misc{ding2025rolermbenchrolermreward,
      title={RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems}, 
      author={Hang Ding and Qiming Feng and Dongqi Liu and Qi Zhao and Tao Yao and Shuo Wang and Dongsheng Chen and Jian Li and Zhenye Gan and Jiangning Zhang and Chengjie Wang and Yabiao Wang},
      year={2025},
      eprint={2512.10575},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.10575}, 
}