Skip to content

feat(training): add SFT demo conversion pipeline#29

Merged
abrichr merged 16 commits into
mainfrom
feat/sft-training-pipeline
Feb 24, 2026
Merged

feat(training): add SFT demo conversion pipeline#29
abrichr merged 16 commits into
mainfrom
feat/sft-training-pipeline

Conversation

@abrichr

@abrichr abrichr commented Feb 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • Add convert_demos.py for converting VLM-annotated demo JSON files into ms-swift SFT training format (JSONL)
  • Add train_from_jsonl() to TRL trainer for training directly from JSONL bundles
  • Add --jsonl flag to train.py for JSONL-based training
  • Add --bundle flag to Lambda Labs CLI for uploading training bundles
  • Consolidate duplicated SFTTrainer setup into shared _run_sft_training()
  • Add plateau-based early stopping (min_delta + plateau_patience)
  • Fix callback MRO (OpenAdaptCallback before TrainerCallback)
  • Fix code sync: git archive replaces rsync (1.8GB → 10MB)

Training Results (Qwen3-VL-2B)

First fine-tuning run completed on Lambda Labs A10 24GB:

Metric Value
Loss 27.24 → 9.77 (64% reduction)
Duration 6.8 hours (50 steps, 10 epochs)
Cost ~$5.10 ($0.75/hr)
Method QLoRA 4-bit NF4, LoRA rank 16
Data 20 annotated WAA demo steps (3 demos)
Checkpoint 12MB adapter (see checkpoints/qwen3vl2b_demo_lora/)

Key observations:

  • ~70% of loss reduction happens in epochs 2-4
  • Plateau-based early stopping would have saved ~2 hours (stop at step 35, loss 10.10 vs 9.77)
  • 33% GPU utilization suggests CPU-side bottleneck with batch_size=1
  • Full analysis: docs/training_results_qwen3vl2b_demo.md

Files Changed

File Change
openadapt_ml/training/convert_demos.py Demo → JSONL conversion with bundle support
openadapt_ml/training/trl_trainer.py JSONL loader, consolidated training, plateau stopping, MRO fix
openadapt_ml/scripts/train.py --jsonl flag, plateau stopping config
openadapt_ml/cloud/lambda_labs.py --bundle upload, git archive sync
configs/qwen3vl_demo.yaml Training config for demo data
docs/training_results_qwen3vl2b_demo.md Detailed training results analysis
docs/gpu_hosting_options.md GPU hosting options for open-source
docs/training_pipeline_gaps.md Pipeline gap analysis

Test plan

  • Conversion produces 20 training samples from 3 annotated demos, all with screenshots
  • Verified training format matches inference prompt structure
  • Coordinate values correctly converted (e.g., 0.294 → 294)
  • Training completed on Lambda Labs A10 (50/50 steps, 6.8 hours)
  • Checkpoint downloaded and verified (28MB adapter)
  • Plateau early stopping would trigger at step ~35 with min_delta=0.1, patience=5

🤖 Generated with Claude Code

abrichr and others added 16 commits February 23, 2026 14:21
@abrichr @claude
feat(training): add demo conversion pipeline for ms-swift SFT format
b7700b8 
Convert annotated demo JSON files to JSONL training data compatible with
ms-swift for Qwen3-VL fine-tuning. Handles coordinate conversion from
[0,1] to [0,1000] range, generates <think> blocks from observation and
intent fields, and accumulates action history across steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
feat(training): add screenshot linking via mapping file
ed2b277 
Support --mapping flag for pre-computed screenshot mapping JSON that
maps task_id -> {step_index -> screenshot_path}. This correctly handles
the coalesced step-to-raw-capture mapping (where step indices skip due
to merging). Also adds --captures-dir with DB-based fallback and
DOUBLE_CLICK/RIGHT_CLICK parsing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
fix(training): align SFT format with inference prompt and add validation
3e08273 
- Add system role message to training conversations matching
  qwen3vl_agent.SYSTEM_PROMPT
- Add "Output exactly one action" / thinking instruction to user message
  matching _build_prompt() output
- Add coordinate range validation warning for values outside [0, 1]
- Add input schema validation for required demo/step fields
- Remove broken _resolve_screenshots_from_db() and
  _resolve_screenshots_direct() fallbacks that produced silently wrong
  mappings for coalesced demos
- Remove --screenshot-dir CLI arg (unreliable for coalesced demos)
- Keep --mapping (recommended) and capture API as screenshot resolution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
feat(training): add JSONL training pipeline with bundle support
de34267 
Align convert_demos output with internal SFT format (images + messages),
add train_from_jsonl() loader, --jsonl flag to train.py, --bundle flag
to convert_demos and Lambda Labs train command. Enables training on
annotated demo data without Episode objects.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
fix(training): add TRL callback, 4-bit quantization, early stopping
28cef56 
- Add OpenAdaptCallback for training_log.json output + early stop on loss
- Fix _load_standard_model to use BitsAndBytesConfig for 4-bit quantization
- Use AutoModelForImageTextToText (supports Qwen3-VL) instead of Qwen2VL class
- Switch demo config to 2B model for fast iteration on A10
- Hide Azure ML Jobs panel when cloud_provider is not azure
- Fix Lambda setup: remove uv.sources before uv sync on remote

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
fix(cloud): use git archive for code sync, fix callback MRO
22c8682 
Replace rsync with `git archive HEAD | ssh tar` in sync_local_code() to
send only committed tracked files (~10MB vs ~1.8GB with binary artifacts).

Fix callback class MRO: _OpenAdaptCallback must precede TrainerCallback
so our on_log/on_train_begin override the no-op base implementations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
refactor(training): consolidate duplicated SFTTrainer setup
659e11e 
Extract _run_sft_training() shared by train_with_trl() and
train_from_jsonl(), eliminating ~80 lines of duplicated SFTConfig,
SFTTrainer instantiation, and training loop code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
feat(training): add plateau-based early stopping
156a6b4 
Add early_stop_min_delta and early_stop_plateau_patience to stop
training when loss stops improving by at least min_delta for N
consecutive steps. Works alongside the existing absolute threshold.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
docs: add GPU hosting options and training pipeline gap analysis
154b8d2 
GPU hosting options covers 24 platforms ranked by value for open-source
projects needing free/credited GPU compute for VLM fine-tuning.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
docs: add Qwen3-VL-2B training results analysis
5973a15 
Detailed analysis of first fine-tuning run: 27.24 → 9.77 loss (64%
reduction) over 50 steps on 20 annotated WAA demo samples. Includes
per-epoch breakdown, compute efficiency metrics, and recommendations
for future training runs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
fix: remove absolute paths from repo, fix LoRA task_type, add tests
04b477e 
- Remove screenshot_mapping.json (has absolute local paths), add to
  .gitignore, add screenshot_mapping.example.json instead
- Fix LoRA task_type: always use CAUSAL_LM (Qwen-VL is decoder-only,
  not encoder-decoder like T5/BART that needs SEQ_2_SEQ_LM)
- Add 57 tests for convert_demos (action parsing, coordinate conversion,
  step conversion, validation, bundle creation) and training callback
  (log writing, threshold early stopping, plateau detection)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
fix: address self-review issues (config wiring, security, tests)
55d49fe 
- Wire lr_scheduler_type, weight_decay, max_grad_norm, target_modules
  from YAML config through to SFTConfig (were silently ignored)
- Fix command injection in lambda_labs.py via shlex.quote()
- Fix callback writing loss=0 on non-loss log events (track _last_loss)
- Fix WAIT() mapping to wait() instead of finished() in convert_demos
- Fix CI: add --no-sources to uv sync for uv.sources compatibility
- Add test for non-loss log event callback behavior
- Update SYSTEM_PROMPT comment (remove stale cross-reference)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
ci: fix uv.sources with UV_NO_SOURCES env var, skip integration tests
267be63 
UV_NO_SOURCES=1 covers uv sync, uv run ruff, and uv run pytest.
Integration tests require openadapt_evals which is not a dependency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
style: format annotate.py with ruff
df605b5 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
feat: auto-generate training plots, persist data with checkpoint
058a834 
- Add plot_training.py: generates loss curve, LR schedule, and combined
  plots from training_log.json using matplotlib
- Copy training_log.json + plots into checkpoint directory after training
  so artifacts are self-contained and never lost
- Add periodic rsync of training_log.json during Lambda training (every
  5 min) so data survives instance interruption
- Replace ASCII loss curve in training results doc with real PNG plots
- Add reconstructed training_log.json from Qwen3-VL-2B demo run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr @claude
test: add tests for plot generation and checkpoint co-location
75b3cb5 
11 tests covering:
- Loss plot generation (with/without LR data)
- Output directory creation and defaults
- Empty data handling
- Epoch boundary rendering
- Real training log validation
- Checkpoint co-location of log + plots

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr
abrichr merged commit e8baa69 into main Feb 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@abrichr