TRL's GRPOTrainer now supports multimodal VLMs (Qwen2.5-VL). Our 793-line custom implementation duplicates advantage computation, KL divergence, reference policy management, policy gradient loss, and checkpointing.
The one genuine gap is multi-turn rollout support — TRL is single-turn. Keep GRPORolloutCollector, replace GRPO math with TRL.
References: TRL v0.20+ VLM support, WebAgent-R1.
TRL's GRPOTrainer now supports multimodal VLMs (Qwen2.5-VL). Our 793-line custom implementation duplicates advantage computation, KL divergence, reference policy management, policy gradient loss, and checkpointing.
The one genuine gap is multi-turn rollout support — TRL is single-turn. Keep
GRPORolloutCollector, replace GRPO math with TRL.References: TRL v0.20+ VLM support, WebAgent-R1.