Skip to content

refactor: make openadapt-ml a leaf; break ml<->evals import cycle#65

Merged
abrichr merged 3 commits into
mainfrom
refactor/ml-leaf-benchmark-types
Jul 13, 2026
Merged

refactor: make openadapt-ml a leaf; break ml<->evals import cycle#65
abrichr merged 3 commits into
mainfrom
refactor/ml-leaf-benchmark-types

Conversation

@abrichr

@abrichr abrichr commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

Phase 1 of the openadapt evals→ml refactor. Makes openadapt-ml a clean model-library leaf by removing every module-level import of openadapt-evals.

Depends on openadapt-types PR #5 (adds openadapt_types.benchmark).

Fork A — Benchmark* vocabulary from openadapt-types

The one hard, unconditional module-level ml → evals import was openadapt_ml/benchmarks/agent.py importing BenchmarkAction/BenchmarkAgent/BenchmarkObservation/BenchmarkTask from openadapt_evals. These types now live in openadapt-types; agent.py (and the lazy BenchmarkAction imports in trainer.py / runner.py) import them from there.

Fork B — thin RolloutEnv Protocol in openadapt-ml

New openadapt_ml/training/grpo/rollout_env.py defines a minimal RolloutEnv Protocol (reset/step/observe/collect_rollout/screen_size) — the surface the GRPO trainer/collector actually drive. The concrete WAA envs/adapters (RLEnvironment, WAALiveAdapter, WAADesktopEnv) stay in openadapt-evals and implement it structurally. ml depends on the interface; evals provides the implementation.

Cleanup — remaining module-level evals imports made lazy

Converted the guarded module-level openadapt-evals imports to lazy in-function imports:

  • rollout_collector.py — env/adapter imported inside __init__ (TYPE_CHECKING for annotations)
  • verl_backend.pyWAADesktopEnv behind _load_waa_desktop_env()
  • trainer.pyTaskConfig inside _load_task_configs

Verification

$ grep -rn "import.*openadapt_evals\|from openadapt_evals" openadapt_ml/ --include=*.py

All remaining hits are lazy (in-function), TYPE_CHECKING-only, or docstrings — no module-level ml → evals import remains. import openadapt_ml.benchmarks.agent now succeeds with openadapt-evals not installed.

Tests: pytest tests/ --ignore=tests/integration451 passed, 5 skipped. (tests/integration requires a live openadapt-evals install and is excluded in CI, as before.)

⚠️ Release sequencing (IMPORTANT — do not merge before types)

  1. openadapt-types PR fix: lower Python requirement from 3.12 to 3.10 #5 merges → cut an openadapt-types release (adds openadapt_types.benchmark).
  2. Update the pin in pyproject.toml here: currently openadapt-types>=0.2.0 with a local editable [tool.uv.sources] entry (../openadapt-types) so imports resolve pre-release. Replace the editable source with the published version once released.
  3. Then merge this PR.

Pre-existing junk (left untouched)

This working tree has pre-existing untracked junk NOT part of this change and deliberately not committed: .git-backup-*.bundle, test_artifacts/, demo .txt files under experiments/waa_demo/annotated_demos/, and a self-referential openadapt-ml symlink at repo root (the symlink causes recursive test collection if you run bare pytest from the root — run pytest tests/ as CI does).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ

abrichr and others added 3 commits July 13, 2026 12:19
Phase 1 of the evals->ml refactor.

Fork A: import the Benchmark* vocabulary (Task/Observation/Action/Agent)
from openadapt-types instead of openadapt-evals. Removes the one hard,
unconditional module-level ml->evals import in benchmarks/agent.py.

Fork B: add a thin RolloutEnv Protocol (reset/step/observe/collect_rollout
/screen_size) in training/grpo/rollout_env.py. The GRPO collector/trainer
type against this interface; the concrete WAA env/adapters stay in
openadapt-evals and implement it structurally.

Also convert the remaining guarded module-level openadapt-evals imports
(rollout_collector, verl_backend, trainer TaskConfig) to lazy in-function
imports so openadapt-ml has zero module-level openadapt-evals imports.

Adds openadapt-types as a dependency (>=0.2.0) with a local editable
uv source until the openadapt-types release is published.

Tests: tests/ (excl. integration) 451 passed, 5 skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
@abrichr
abrichr marked this pull request as ready for review July 13, 2026 16:44
@abrichr
abrichr merged commit 7f62c18 into main Jul 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant