Reproducing Qwen2.5-3B performance

[zhuqiangLu]

Hi there,

I just tried running the LV-Eval -128k subset on the Qwen2.5-3B-Instruct model(no AHN enabled). The performance is terribly low and can't match the number reported in paper.

So here is the screenshot. I ran the evaluation on a single A6000Ada 48G VRAM with single process mode. Could you please share the model prediction (the json files) so I can verify what goes wrong.

[zhuqiangLu]

also, could you please confirm which transformer version is used for evaluation? The config.json in your model shows transformers==4.49.0.

(Based on my own experience, different transformers CAN indeed get very different performance potentially due to the default sampling parameters)

https://huggingface.co/ByteDance-Seed/AHN-DN-for-Qwen-2.5-Instruct-3B/blob/main/config.json

[yuweihao]

Hi @zhuqiangLu , for Qwen2.5 (full attn) prediction results on LV-Eval, please see this link. Also, make sure your GPU's memory size is sufficient to evaluate 128k.