Reproduce issue

[Harry-mic]

Hi, thanks for this awesome work!
I encounter a reproduce issue that the performance I reproduce on some tasks(such as GuessTheNumber, MineSweeper, Mastermind) doesn't reach the performance in the blog: https://axon-rl.notion.site/gem#238dba9c0091800bb5eedaf66517d840.

Here is the reproduced performance of GuessTheNumber:

MineSweeper:

Mastermind:

The parameters I used is :

python examples/train_oat.py \
    --env_id game:**(three game )** \
    --prompt_template qwen3_game \
    --wrappers concat \
    --gamma 0.9 \
    --norm_return \
    --gpus 8 \
    --gradient-checkpointing \
    --num_samples 1 \
    --rollout_batch_size 128 \
    --num_env 2 \
    --rollout_batch_size_per_device 16 \
    --pi_buffer_maxlen_per_device 16 \
    --pretrain  Qwen3-1.7B-Base \
    --enable_prefix_caching \
    --collocate \
    --vllm_sleep \
    --vllm_gpu_ratio 0.45 \
    --rnd-seed \
    --learning_rate 0.000001 \
    --lr_scheduler constant \
    --lr_warmup_ratio 0 \
    --num_ppo_epochs 2 \
    --train_batch_size 128 \
    --train_batch_size_per_device 1 \
    --beta 0 \
    --max_model_len 12800 \
    --generate_max_length 4096 \
    --temperature 1.0 \
    --top_p 1 \
    --eval_steps -1 \
    --save_steps 450 \
    --eval_temperature 0.6 \
    --eval_top_p 0.95 \
    --eval_generate_max_length 4096 \
    --max_train 65000 \
    --max_save_num 1 \
    --use-wb \
    --wb-run-name oat-qwen3-1.7b-base-game:**(three game )** \
    --wb_project gem

[Harry-mic]

Report in issue