Seeking training recipe/advice for SDPO on Math tasks

[AzureStarz]

Could you please share a training recipe for applying SDPO to math tasks?

Currently, I am training Qwen2.5-3B-Instruct on a Math training split using your SDPO implementation, but the val-core metric keeps degrading throughout the training progress. I have already tried swapping in other models and different datasets, but the training still isn't working as expected.

I would love to know if you have any empirical experience, recommended hyperparameters, or general advice for adapting your method successfully to math-heavy tasks. Thanks!

[JianghaoWu]

I’m facing the same issue and would also appreciate any guidance or practical suggestions.

[euclidgame]

I think this is somewhat expected... I ran a lot of math training with SDPO as well, so the training rewards went up because the model started memorizing the correct demonstrations for the training problems. The model can't learn to reason through SDPO, and thus the improvement can't transfer to the validation set.

[joejajo]

Hi ,
Im trying one shot rlvr paper on SDPO , is this a good idea ?

[joejajo]

did any one try on math problems , the model after training, give any improvment in generalisation ?

[joejajo]

I think this is somewhat expected... I ran a lot of math training with SDPO as well, so the training rewards went up because the model started memorizing the correct demonstrations for the training problems. The model can't learn to reason through SDPO, and thus the improvement can't transfer to the validation set.

The paper does prove generalisation on tasks right i think math training should work i guess , did you find any succes after trying something ?

[joejajo]

I’m facing the same issue and would also appreciate any guidance or practical suggestions.

did you get any workaround were you able to successfully implement math traning on SDPO ?

[cnsdqd-dyb]

I have a similar concern. I trained SDPO with Qwen3-1.7B and Qwen3-4B on NuminaMath, and evaluated on MATH500 every 5 training steps using batch size 16 and three seeds. The responses became shorter, but the MATH500 performance did not really improve. It slightly increased at first and then slightly degraded, with roughly ±2% fluctuation in mean@4 over 40 steps. This behavior is very different from GRPO, where I observed around +3%~5% improvement, although the results were still somewhat unstable.

[cnsdqd-dyb]

I think this is somewhat expected... I ran a lot of math training with SDPO as well, so the training rewards went up because the model started memorizing the correct demonstrations for the training problems. The model can't learn to reason through SDPO, and thus the improvement can't transfer to the validation set.

I also visualized the token-level signals during SDPO training. Based on this visualization, my understanding is that SDPO tends to encourage the model to closely fit the teacher solution at the token level. However, in math reasoning, multiple solution paths can be valid. In many cases, the model may only change the wording or formatting, or follow a different but still correct reasoning trajectory. Such diversity is important for RL exploration, but it may still be penalized by the SDPO objective.

This issue may be more serious for small models in the 1B–8B range. The token-level visualization suggests that the self-teacher may not reliably identify which tokens or reasoning steps are truly wrong, even when guided by hints from the reference solution during rollout. As a result, the training signal may suppress useful exploration and discourage the model from discovering alternative reasoning strategies. This could explain why SDPO does not improve math performance as clearly as GRPO in my experiments.

[jameszhu-pixel-time]

I think this is somewhat expected... I ran a lot of math training with SDPO as well, so the training rewards went up because the model started memorizing the correct demonstrations for the training problems. The model can't learn to reason through SDPO, and thus the improvement can't transfer to the validation set.

I also visualized the token-level signals during SDPO training. Based on this visualization, my understanding is that SDPO tends to encourage the model to closely fit the teacher solution at the token level. However, in math reasoning, multiple solution paths can be valid. In many cases, the model may only change the wording or formatting, or follow a different but still correct reasoning trajectory. Such diversity is important for RL exploration, but it may still be penalized by the SDPO objective.

This issue may be more serious for small models in the 1B–8B range. The token-level visualization suggests that the self-teacher may not reliably identify which tokens or reasoning steps are truly wrong, even when guided by hints from the reference solution during rollout. As a result, the training signal may suppress useful exploration and discourage the model from discovering alternative reasoning strategies. This could explain why SDPO does not improve math performance as clearly as GRPO in my experiments.

Maybe training from scratch with small models is not that practical, since small models have weaker prior and fail to produce valid corrections. I tried doing inference instead of forwarding based on previous attempts and see whether it produce valid responses in math tasks, but most rollouts are found coping instead of reasoning. Recent works have implemented SFT and advanced models that gain improvement in math.I suppose that the model's prior matters the most. Appreciate for your valuable training insights!

[cnsdqd-dyb]

I think this is somewhat expected... I ran a lot of math training with SDPO as well, so the training rewards went up because the model started memorizing the correct demonstrations for the training problems. The model can't learn to reason through SDPO, and thus the improvement can't transfer to the validation set.

I also visualized the token-level signals during SDPO training. Based on this visualization, my understanding is that SDPO tends to encourage the model to closely fit the teacher solution at the token level. However, in math reasoning, multiple solution paths can be valid. In many cases, the model may only change the wording or formatting, or follow a different but still correct reasoning trajectory. Such diversity is important for RL exploration, but it may still be penalized by the SDPO objective.
This issue may be more serious for small models in the 1B–8B range. The token-level visualization suggests that the self-teacher may not reliably identify which tokens or reasoning steps are truly wrong, even when guided by hints from the reference solution during rollout. As a result, the training signal may suppress useful exploration and discourage the model from discovering alternative reasoning strategies. This could explain why SDPO does not improve math performance as clearly as GRPO in my experiments.

Maybe training from scratch with small models is not that practical, since small models have weaker prior and fail to produce valid corrections. I tried doing inference instead of forwarding based on previous attempts and see whether it produce valid responses in math tasks, but most rollouts are found coping instead of reasoning. Recent works have implemented SFT and advanced models that gain improvement in math.I suppose that the model's prior matters the most. Appreciate for your valuable training insights!

Yes, I agree with "model's prior matters the most.", especially for small models.