fix(grpo): VLM processor metadata misalignment after token concatenation

[abrichr]

Problem

In _compute_rollout_loss (trainer.py:519-525), after concatenating action token IDs to prompt input_ids, only input_ids and attention_mask are updated. VLM-specific keys like pixel_values, image_grid_thw (Qwen2.5-VL) retain their original shapes tied to the prompt-only length. This may cause image token position misalignment during the forward pass.

Additionally, line 525 does {k: v.to(device) for k, v in full_inputs.items()} which fails if any value is not a tensor (e.g., integer metadata in processor output). Need an isinstance(v, torch.Tensor) guard.

[abrichr]

Closing: resolved by replacing the custom GRPO trainer with a minimal TRL bridge (see PR #34). The custom token concatenation logic in _compute_rollout_loss that caused VLM processor metadata misalignment (pixel_values, image_grid_thw shape mismatch) is eliminated. TRL handles tokenization and forward pass construction correctly, including proper VLM-specific key management for Qwen2.5-VL.