Question about reproducing LabUtopia results

[qpww-ic]

Hi, thanks for sharing your work! I would like to include LabVLA as a baseline and align my experimental settings with yours for a fair comparison on LabUtopia.

I tried fine-tuning [LabVLA-5B-Base](https://huggingface.co/zjunlp/LabVLA-5B-Base) using the data from [LEAP-LAB-KUS](https://huggingface.co/LEAP-LAB-KUS) with:

bash launch/finetune/train_labutopia.sh

However, after 80,000 training iterations, I obtained a 0% success rate on both Level 1 and Level 3. Could you please share the data generation procedure and experimental setup to obtain your reported results, so I can align mine with yours?

Thanks for your help!

[BaochangRen]

Hi, thank you very much for your interest in our work and for taking the time to reproduce the LabUtopia results! We really appreciate your detailed feedback and for bringing this issue to our attention!

To provide a reference for reproduction, we recently re-ran the Level 3 TransportBeaker task on LabUtopia. Under the is_test_material_False setting, we obtained a 94.17% success rate (113/120 episodes).

We have also uploaded the complete evaluation logs and inference results here for reference:

https://github.com/zjunlp/LabVLA/tree/main/outputs/infer/2026.09.09/04.28.55_labvla_level3_TransportBeaker_40000_120

We hope these logs can help with checking and aligning the evaluation setup. There may be some differences in the experimental or evaluation configurations that lead to the discrepancy, and we are also happy to investigate them together.

Please feel free to reach out at any time if you have any further questions or encounter any other issues! We would be very happy to communicate and help with the reproduction. Thanks again for your interest!

[zxlzr]

hi do you have any other questions？

[qpww-ic]

Thanks for your quick reply!

I do have two follow-up questions.

(1) Also, I noticed that the [LabUtopia](https://github.com/Rui-li023/LabUtopia) repository currently has both main and Dev branches, so I would like to make sure I am using the same branch and evaluation setup as yours.

(2) Is it sufficient to fine-tune LabVLA-5B-Base using the provided LabUtopia training script and data to reproduce the results reported in the paper, or are there any additional training or data-processing steps required?

Thanks again for your help!

[BaochangRen]

Thanks for the follow-up questions!

For our LabUtopia experiments, we used the Isaac Sim 5.1 version of the benchmark, and the same codebase was used consistently for data collection, training, and evaluation. To make reproduction easier, we have uploaded the LabUtopia code used in our experiments to the labutopia branch here: https://github.com/zjunlp/LabVLA/tree/labutopia. We did not apply any additional processing to the collected training data before fine-tuning LabVLA-5B-Base.

When we first worked with this benchmark, we also encountered several issues, which were later resolved with the help and coordination of the LabUtopia authors, along with some corresponding code adjustments. We suspect that you may be facing similar environment- or implementation-related issues, and we hope the version we uploaded can help resolve them. Please feel free to reach out anytime if you have any further questions or encounter any other problems—we would be very happy to discuss them and help with the reproduction!

[zxlzr]

hi do you have any other questions?

[qpww-ic]

Thanks for your detailed clarification and for uploading the LabUtopia branch.

I am trying to reproduce the results with this setup. I also hope we may have a chance to communicate more and potentially collaborate in the future.

Thanks again for your help!

[BaochangRen]

So excited! We are also looking forward to staying in touch and hopefully having opportunities to collaborate in the future.

Please feel free to reach out to me at baochang.ren@zju.edu.cn if you would like to discuss anything in more detail. I would be very happy to communicate further!