Skip to content

feat(grpo): replace custom training loop with TRL GRPOTrainer #35

Description

@abrichr

TRL's GRPOTrainer now supports multimodal VLMs (Qwen2.5-VL). Our 793-line custom implementation duplicates advantage computation, KL divergence, reference policy management, policy gradient loss, and checkpointing.

The one genuine gap is multi-turn rollout support — TRL is single-turn. Keep GRPORolloutCollector, replace GRPO math with TRL.

References: TRL v0.20+ VLM support, WebAgent-R1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions