Skip to content

Align locomotion defaults with validated DexUni training - #1130

Merged
acrlw merged 2 commits into
mainfrom
codex/locomotion-action-clip-config
Nov 11, 2026
Merged

acrlw merged 2 commits into
mainfrom
codex/locomotion-action-clip-config

Conversation

@acrlw

@acrlw acrlw commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR makes the six official flat-ground velocity tasks default to the DexUni → CUDA MuJoCo Warp configuration used by the validated Unitree training runs. The Newton environments explicitly set the solver and its MuJoCo options, and their catalogs and README commands select the Newton deployment. Both backend configurations also preserve unbounded policy actions with clip: null.

The existing Newton YAMLs select MuJoCo Warp directly with 10 solver iterations. The validated runs instead use DexUni with dynamic joints, automatic contact selection, 20 solver iterations and 50 line-search iterations, with CPU MuJoCo disabled. Recording these values in the task configuration removes the need for training-script solver overrides.

The action parameters are also currently empty, so they inherit the default clip of 1.0. A policy output of 2.0 therefore becomes 1.0 before the task's offset-and-scale mapping. This also changes the action history used by observations and action-rate rewards. Explicitly setting clip: null restores the unbounded actions used by the successful historical training runs.

The default changed during the action migration in #680. Explicit null support was added in #698, but the task YAMLs were left empty. Several task JSONs already declare actions.joint_pos.clip: null; the Velocity environment does not forward that field. The issue is present on main and was identified while validating #829.

  • Set env.actions.joint_position.params.clip: null in both env.yaml and env.newton.yaml for Go1, Go2, G1, H1_2, ANYmal-C and MicroDuck: 12 configurations.
  • Set all six env.newton.yaml files to solver_type: dexuni, joint_mode: dynamic, contact_mode: auto, and explicit mujoco_options: iterations: 20, ls_iterations: 50, use_mujoco_cpu: false, integrator: implicitfast, cone: pyramidal, impratio: 1.0, ls_parallel: false, separate_worlds: true, tolerance: 1e-8, njmax: 1024 and nconmax: 256.
  • Select newton as the six task catalogs' default deployment and update their README training commands and solver description.
  • Extend the existing joint-binding test to load all 12 configurations through the production decoder, initialize the task binding and process positive and negative actions outside ±1 through ActionManager. Check joint ordering, unbounded action space, preserved raw actions and calibrated joint targets.
  • Extend existing deployment and catalog tests to check native DexSim solver conversion, the complete MuJoCo option mapping, the preferred deployment and matching PPO references.

Dependencies: none beyond the current main dependencies.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Validation

On this main-based branch, 60 CPU tests pass. The regression checks fail with the original action clipping, solver selection and catalog defaults:

CUDA_VISIBLE_DEVICES='' PYTHONPATH=.:extensions:extensions/embodichain_tasks \
  python -m pytest -q \
  tests/gym/envs/tasks/test_unitree_locomotion.py \
  tests/gym/envs/managers/test_default_joint_position.py \
  tests/gym/envs/tasks/test_locomotion_deployments.py::test_packaged_deployment_and_agent_config_agree \
  tests/test_locomotion_catalog.py

The extended test closes the gap between generic action-term coverage and actual packaged task configuration. Existing action-history and explicit clipping tests are reused. These checks run on CPU without creating a simulation World.

On the current PR head 941c77366, all six Newton tasks also pass actual GPU training smoke runs: 4,096 environments, 24 steps/update, seed 42, 10 PPO updates from scratch (983,040 environment steps per task). The runs load the committed environment YAMLs directly; only the update limit is shortened and the environment file reference is resolved to its absolute path. The runtime is Python 3.11.15 with DexSim 0.6.0rc1 AOT.

Each run verifies DexUni pure_mujoco dispatch, CUDA MuJoCo Warp with CPU MuJoCo disabled, solver iterations 20/50, unbounded actions and NoRender. All 60 PPO updates have finite losses and policy parameters. All six final resets return finite observations, and all six checkpoints save and reload successfully with the expected update/step counters and finite policy tensors.

Previously completed controlled training used #829 at c6e991c2e, whose Velocity binding and action manager match this branch's base 460939c83. Each of the four Unitree tasks completed a separate 10-update startup check and 1,000 updates from scratch using DexUni → CUDA MuJoCo Warp, solver iterations 20/50, 4,096 environments, 24 steps/update and seed 42. The configuration change under comparison was only clip: null. The four updated Unitree environment YAMLs match those recorded null-clip training inputs exactly after YAML decoding. These 1k runs are reused as convergence evidence alongside the current-head GPU smoke runs above.

Episode returns below are medians over updates 951–1,000. The nine-command evaluation uses 128 environments per command, 1,000 control steps and excludes the first 50 steps from error metrics.

Task Default clip return Null clip return Historical return Null clip / historical commands passed
Go1 62.71 90.79 90.99 9/9 / 9/9
Go2 44.58 56.04 55.83 9/9 / 9/9
G1 −4.11 22.32 21.71 9/9 / 9/9
H1_2 −6.40 −4.57 −3.93 2/9 / 2/9

Historical runs resumed at updates 8 and 30; the new runs trained continuously from zero. H1_2 matches the historical 1k pass count and still fails seven commands at that stage. ANYmal-C and MicroDuck pass the current-head 10-update GPU smoke runs; their longer-term training convergence remains unvalidated in this change.

black==26.10.1 formatting and its repository-wide check pass. The public API documentation gate passes (4,434/4,434 exports). Task READMEs describe the new default. The env-framework and rl-learning context was reviewed: it already directs readers to the task-owned configurations, and its loading guidance remains accurate.

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
  • Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py), if applicable. No public API changes; coverage check passed.
  • I verified appropriate behavioral coverage, reviewed the necessity of added or extended tests, and explained when no new tests were needed.
  • Dependencies have been updated, if applicable. No dependency changes required.

@acrlw acrlw added bug Incorrect existing behavior or a fix restoring the intended behavior. task Concrete tasks, deployments, objectives and recipes, including configured Task Program tasks. rl Reinforcement learning algorithms, collectors, trainers and training contracts. labels Nov 9, 2026
@acrlw acrlw changed the title Restore unbounded actions in locomotion task configurations Align locomotion defaults with validated DexUni training Nov 10, 2026
@acrlw
acrlw marked this pull request as ready for review November 10, 2026 02:17
@yuecideng
yuecideng self-requested a review November 11, 2026 02:18
@acrlw
acrlw merged commit bf3c06c into main Nov 11, 2026
16 checks passed
@acrlw
acrlw deleted the codex/locomotion-action-clip-config branch November 11, 2026 02:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Incorrect existing behavior or a fix restoring the intended behavior. rl Reinforcement learning algorithms, collectors, trainers and training contracts. task Concrete tasks, deployments, objectives and recipes, including configured Task Program tasks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants