I've noticed that in both grpo_trainer_mmp.py and grpo_trainer_aid.py, there is an operation of inserting gt into completions. I'm wondering if gt here refers to the standard ground truth, and if it includes the "think" part. Additionally, what's the purpose of this operation? Can it contribute to the model update? In my understanding, simply performing a replacement might not have a proper effect on gradient backpropagation.
I've noticed that in both
grpo_trainer_mmp.pyandgrpo_trainer_aid.py, there is an operation of insertinggtintocompletions. I'm wondering ifgthere refers to the standard ground truth, and if it includes the "think" part. Additionally, what's the purpose of this operation? Can it contribute to the model update? In my understanding, simply performing a replacement might not have a proper effect on gradient backpropagation.