Skip to content

Add G1 rough locomotion and USD-native dexterous manipulation RL tasks - #837

Merged
yuecideng merged 24 commits into
mainfrom
codex/rl-environments-assets
Nov 19, 2026
Merged

yuecideng merged 24 commits into
mainfrom
codex/rl-environments-assets

Conversation

@acrlw

@acrlw acrlw commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add the 37-DOF Unitree G1 rough-terrain locomotion task (UnitreeG1RoughRL-v1) and five USD-native dexterous manipulation RL tasks (ShadowPenFixedRL-v1, ShadowPenContinuousRL-v1, LeapCubeFaceTurnRL-v1, LeapCubeReorientRL-v1, LeapCubeRotateZAxisRL-v1) together with their registered Hugging Face asset archives (UnitreeG1Rough, G1RoughTerrains, ShadowHandPen, LeapHandCubeFaceTurn, LeapHandCube), task catalogs, PPO configurations, and documentation.

Summary of Changes

  • USD-Native Dexterous Manipulation (EmbodiedEnv):
    • Implement DexterousManipulationEnv(EmbodiedEnv) and five registered Gym tasks (ShadowPenFixedRL-v1, ShadowPenContinuousRL-v1, LeapCubeFaceTurnRL-v1, LeapCubeReorientRL-v1, LeapCubeRotateZAxisRL-v1) that load scene geometry, fixed tendons, joint limits, actuator ctrlrange, PD target_ke/target_kd, site local poses, and collision filter pairs directly from packaged .usdc files via SimulationManager.add_usd() with zero runtime MJCF dependency.
    • Add SimulationManager.remove_ground_plane() and SimulationManager.clear_articulation_dynamics() to support groundless scenes and multi-articulation reset velocity clearing on Newton backends.
  • 37-DOF Unitree G1 Rough-Terrain Locomotion (UnitreeG1RoughRL-v1):
    • Add RoughVelocityEnv(EmbodiedEnv), heightfield terrain curriculum (G1RoughTerrains), 187-point local elevation scanner, and schema-v2 checkpoint curriculum state save/restore (--resume).
  • Single-Deployment DexUni Task Catalogs:
    • Configure each new task directory under extensions/embodichain_tasks/configs/tasks/ with catalog.yaml (default_deployment: default), env.yaml (physics: newton, solver_type: dexuni, use_mujoco_cpu: false, GPU MuJoCo Warp on CUDA), agents/ppo.yaml, and README.md.
  • Runtime correctness:
    • Use processed rough-locomotion actions so configured scale, offset, and encoder bias reach the physics targets.
    • Preserve opted-in sensor sampling on each physics step in the rough and dexterous custom stepping paths.
    • Convert the FaceTurn cube root COM velocity to body-origin velocity for the task freejoint state.
    • Preserve time-limit resets and bootstrapping when dexterous physical terminations are ignored.
  • Validation:
    • CI test isolation: restrict the materialization test path override to its synthetic asset. The original test order reproduced the stale default-cache failure; after the fix, both ordered cases and all 93 tests in tests/gym/envs/task_program/test_asset_materialization.py and tests/data/test_download.py pass.
    • Runtime regression validation: 173 passed, 1 deselected across test_rough_velocity.py, test_dexterous_metrics.py, test_dexterous_usd.py, test_unitree_locomotion.py, and test_rl_resume.py with -m "not gpu and not requires_sim". The targeted regression selection reproduced 14 failing cases before the fixes. Existing test modules cover configured targets, all custom stepping paths, oriented COM offsets, and M1–M5 timeout/GAE behavior.
    • black --check --diff --color ./ (Black 26.10.1), git diff --check, and API docs coverage (4920/4920) pass. Reviewed environment and RL agent context; these fixes restore the documented action, sensor, and episode behavior without changing public APIs or usage.
    • Earlier full-PR validation: unit, catalog layout, API docs, download, RL resume, and CLI suites (tests/gym/envs/tasks/test_dexterous_metrics.py, test_dexterous_usd.py, test_rough_velocity.py, test_rough_velocity_frame.py, test_unitree_locomotion.py, test_locomotion_deployments.py, tests/gym/envs/test_official_task_layout.py, tests/docs/test_check_api_docs.py, tests/data/test_download.py, tests/learning/test_rl_resume.py, tests/cli/): 339 passed, 14 skipped; docs/scripts/check_api_docs.py: 4920/4920 exports aligned.
    • Earlier sustained single-GPU PPO training (4096 parallel environments, solver_type: dexuni, NVIDIA RTX PRO 5000 Blackwell 72 GB), --resume latest (+10 updates), and 64-episode eval-policy:

The training results below predate the runtime correctness fixes. Sustained training and policy evaluation were not repeated for these fixes.

Environment Updates (train + resume) Transitions Steady SPS Train episode_reward_avg_100 (start -> end) eval-policy (64 episodes, --checkpoint latest)
UnitreeG1RoughRL-v1 150 + 10 15,728,640 67,576 -4.71 -> -4.19 avg_reward=-3.40, avg_length=45.25, linear_velocity_error=0.649, root_height=0.789 m
ShadowPenFixedRL-v1 150 + 10 26,214,400 54,106 -50.49 -> +47.94 avg_reward=+74.26, avg_length=500.0, retained=100%, error=0.143 rad, rotation=2.40 rad
ShadowPenContinuousRL-v1 150 + 10 52,428,800 57,265 -39.46 -> +14.47 avg_reward=+9.64, avg_length=485.8, retained=96.9%, error=0.229 rad, rotation=0.82 rad
LeapCubeFaceTurnRL-v1 150 + 10 26,214,400 55,562 -0.47 -> +216.71 avg_reward=+255.53, avg_length=500.0, success_rate=100%, success_count=12.52, rotation=20.56 rad
LeapCubeReorientRL-v1 300 + 10 50,790,400 48,218 -4.58 -> +172.64 avg_reward=+180.98, avg_length=997.6, retained=98.4%, success_rate=3.1%, position_error=0.020 m
LeapCubeRotateZAxisRL-v1 300 + 10 50,790,400 94,216 -5.56 -> +27.40 avg_reward=+29.35, avg_length=497.5, retained=98.4%, rotation=29.43 rad, angular_velocity_z=1.39 rad/s

Type of change

  • New feature (non-breaking change which adds functionality)
  • Documentation update

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
  • Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py), if applicable
  • I verified appropriate behavioral coverage, reviewed the necessity of added or extended tests, and explained when no new tests were needed.
  • Dependencies have been updated, if applicable.

@acrlw acrlw added enhancement Improvements to existing capabilities; use feature for new capabilities. task Concrete tasks, deployments, objectives and recipes, including configured Task Program tasks. rl Reinforcement learning algorithms, collectors, trainers and training contracts. assets Robot and scene assets, meshes, materials, asset catalogs and downloads. labels Oct 22, 2026
@acrlw
acrlw marked this pull request as ready for review October 22, 2026 07:42
@greptile-apps

greptile-apps Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 2/5

[Medium impact] The PR adds new RL workflows and extends the shared training-resume path.

The PR is not ready to merge while the earlier credential leak and flat-task action-limit regression remain unresolved.

Fix All in CodexFindings

  1. P1 Security Downloads expose the API key ▶
  2. P1 Flat tasks lose action limits ▶
  3. P2 Pen failures get timeout bonuses ▶
Fix with agent prompt
### Issue 1
embodichain/gen_sim/services/runtime.py:186-188
A geometry profile with `api_key` puts its bearer token on the shared session. `_download_glb()` uses that session for the server-provided `mesh` URL, while `_mesh_url()` accepts URLs on any host. A response pointing to a separate file host therefore sends the geometry service's secret to that host, even during a normal download.

Keep credentials on requests to the configured service only. Use an unauthenticated session for external downloads or reject external URLs.

**How this was verified:** The geometry download uses the session carrying `Authorization`, and the server-provided absolute URL passes through without a host check.

### Issue 2
extensions/embodichain_tasks/embodichain_tasks/locomotion/velocity/_embodichain.py:undefined-111
Setting `clip` to `None` in the shared parent also disables clipping for existing flat locomotion tasks. Their configurations leave `params` empty, so they previously used `DefaultJointPositionAction`'s limit of 1.0. PPO outputs outside `[-1, 1]` now reach joint targets and action-history rewards unchanged, even though these tasks still declare that bounded action space. This silently changes training and saved-policy behavior.

Set the unbounded default only for `IsaacVelocityEnv` and preserve the existing tasks' clipping.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 3
extensions/embodichain_tasks/embodichain_tasks/manipulation/dexterous/_runtime.py:undefined-681
A pen task that fails on step 500 returns both `terminated` and `truncated` as true. The shared `compute_gae()` adds a value estimate whenever `truncated` is true, so that physical failure incorrectly receives a time-limit bonus. This gives the final transition the wrong training return.

Match the LEAP tasks by excluding terminated rows from `truncated`.

```suggestion
            truncated = (self.elapsed_steps >= 500) & ~terminated
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds G1 rough-terrain locomotion and five pen/cube manipulation tasks.

  • The G1 can train to walk over a rough terrain course.
  • Five hand tasks let trainers spin a pen and turn or rotate cubes.
  • The asset catalog now includes the G1, terrain, pen, and cube archives.
  • RL checkpoints can carry optional per-rank environment state.

The previous-review range was not used because its merge base differs from the current PR's merge base. This review therefore covered the full PR.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Config[Task configuration] --> Assets[Shared asset catalog]
  Assets --> Scene[Robot and terrain or manipulation scene]
  Scene --> Solver[Configured physics solver]
  Solver --> Task[Observations rewards and episode results]
  Task --> PPO[PPO training]
  PPO --> Checkpoint[Schema-v2 checkpoint]
  Task --> State[Per-rank training state]
  State --> Checkpoint
  Checkpoint --> Reset[Restore state before fresh episodes]
Loading

Reviews (8) · Last reviewed commit: "test(tasks): model ground redeclaration ..." · Reviewed by Greptile

@yuecideng yuecideng added feature New user-facing capabilities. docs Documentation, API references and example explanations. physics Physics backends, dynamics, contacts, fluids, deformables and physics solvers. and removed enhancement Improvements to existing capabilities; use feature for new capabilities. labels Oct 25, 2026
@acrlw
acrlw force-pushed the codex/rl-environments-assets branch from a52c3c2 to fc534c0 Compare November 2, 2026 02:52
@acrlw
acrlw force-pushed the codex/rl-rollout-performance branch from b9448d4 to 944715e Compare November 2, 2026 02:52
@acrlw
acrlw marked this pull request as draft November 2, 2026 02:53
@acrlw
acrlw marked this pull request as ready for review November 2, 2026 05:21
@acrlw
acrlw force-pushed the codex/rl-rollout-performance branch from 3649faf to 9ed54ed Compare November 3, 2026 10:50
Comment thread embodichain/gen_sim/services/runtime.py Outdated
Comment on lines +186 to +188
session = requests.Session()
session.headers["Authorization"] = f"Bearer {api_key}"
kwargs["session"] = session

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security Downloads expose the API key

A geometry profile with api_key puts its bearer token on the shared session. _download_glb() uses that session for the server-provided mesh URL, while _mesh_url() accepts URLs on any host. A response pointing to a separate file host therefore sends the geometry service's secret to that host, even during a normal download.

Keep credentials on requests to the configured service only. Use an unauthenticated session for external downloads or reject external URLs.

How this was verified: The geometry download uses the session carrying Authorization, and the server-provided absolute URL passes through without a host check.

Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/gen_sim/services/runtime.py
Line: 186-188

Comment:
**Downloads expose the API key**

A geometry profile with `api_key` puts its bearer token on the shared session. `_download_glb()` uses that session for the server-provided `mesh` URL, while `_mesh_url()` accepts URLs on any host. A response pointing to a separate file host therefore sends the geometry service's secret to that host, even during a normal download.

Keep credentials on requests to the configured service only. Use an unauthenticated session for external downloads or reject external URLs.

**How this was verified:** The geometry download uses the session carrying `Authorization`, and the server-provided absolute URL passes through without a host check.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in #1137, commit 67b696d8a.

Fixed the HTTP profile session so the service Bearer token is attached only when the request's scheme, host and effective port match the configured service origin. External mesh downloads receive no profile Authorization header, and cross-origin redirects strip it, including same-host HTTP-to-HTTPS redirects. Seven transport regression cases cover service-relative URLs, external URLs, port changes and redirects. The affected code is already on main, so this fix is kept separate from the RL task changes.

"preserve_order": True,
"offset": self.velocity_task_config.default_joint_position,
"scale": self.velocity_task_config.action_scale,
"clip": None,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Flat tasks lose action limits

Setting clip to None in the shared parent also disables clipping for existing flat locomotion tasks. Their configurations leave params empty, so they previously used DefaultJointPositionAction's limit of 1.0. PPO outputs outside [-1, 1] now reach joint targets and action-history rewards unchanged, even though these tasks still declare that bounded action space. This silently changes training and saved-policy behavior.

Set the unbounded default only for IsaacVelocityEnv and preserve the existing tasks' clipping.

Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/embodichain_tasks/embodichain_tasks/locomotion/velocity/_embodichain.py
Line: 111

Comment:
**Flat tasks lose action limits**

Setting `clip` to `None` in the shared parent also disables clipping for existing flat locomotion tasks. Their configurations leave `params` empty, so they previously used `DefaultJointPositionAction`'s limit of 1.0. PPO outputs outside `[-1, 1]` now reach joint targets and action-history rewards unchanged, even though these tasks still declare that bounded action space. This silently changes training and saved-policy behavior.

Set the unbounded default only for `IsaacVelocityEnv` and preserve the existing tasks' clipping.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex Fix in Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 528958cf2.

Fixed the shared velocity environment to retain the original clip=1.0 default. Only IsaacVelocityEnv defaults to clip=None, and explicit action-term settings still take precedence. The regression exercises the actual constructor and action-processing path: an input of 2.0 at scale 0.5 produces a target of 0.5 for the flat environment and 1.0 for Isaac. Explicit clipping and unbounded overrides are also covered.

if self.task_id in ("M1", "M2"):
obs, reward, terminated, metrics = self.task.step(state, actions)
self.elapsed_steps += 1
truncated = self.elapsed_steps >= 500

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Pen failures get timeout bonuses

A pen task that fails on step 500 returns both terminated and truncated as true. The shared compute_gae() adds a value estimate whenever truncated is true, so that physical failure incorrectly receives a time-limit bonus. This gives the final transition the wrong training return.

Match the LEAP tasks by excluding terminated rows from truncated.

Suggested change
truncated = self.elapsed_steps >= 500
truncated = (self.elapsed_steps >= 500) & ~terminated
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/embodichain_tasks/embodichain_tasks/manipulation/dexterous/_runtime.py
Line: 681

Comment:
**Pen failures get timeout bonuses**

A pen task that fails on step 500 returns both `terminated` and `truncated` as true. The shared `compute_gae()` adds a value estimate whenever `truncated` is true, so that physical failure incorrectly receives a time-limit bonus. This gives the final transition the wrong training return.

Match the LEAP tasks by excluding terminated rows from `truncated`.

```suggestion
            truncated = (self.elapsed_steps >= 500) & ~terminated
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 528958cf2.

Fixed the M1/M2 adapter to set truncated = (elapsed_steps >= 500) & ~terminated. Physical failure now takes precedence when it coincides with the time limit. The regression runs both tasks through the real step/reset bridge and GAE: with reward 1, successor value 10 and gamma 0.9, failure at step 500 returns 1, while a pure timeout returns 10.

Base automatically changed from codex/rl-rollout-performance to main November 11, 2026 11:46
@acrlw acrlw changed the title Add G1 rough-terrain and five dexterous manipulation RL tasks Add G1 rough locomotion and USD-native dexterous manipulation RL tasks Nov 13, 2026
@acrlw

acrlw commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

@greptileai review this PR again. Please review the full current diff against main at c4a6f79; it has been substantially simplified and updated since the previous review.

@acrlw
acrlw force-pushed the codex/rl-environments-assets branch from c4a6f79 to 2db0ef8 Compare November 19, 2026 06:07
acrlw added 23 commits November 19, 2026 15:44
…diedEnv with DexUni default

- Replace runtime MJCF loading and _dexuni_adapter with USD-native DexterousManipulationEnv(EmbodiedEnv) loading ShadowHandPen, LeapHandCubeFaceTurn, and LeapHandCube .usdc scenes via SimulationManager.add_usd()
- Add SimulationManager.remove_ground_plane() and clear_articulation_dynamics() for groundless and multi-articulation Newton scenes
- Standardize Gym IDs (*RL-v1) and catalog deployments (newton -> solver_type: dexuni default; mjwarp -> solver_type: mujoco_warp) across all 5 dexterous tasks
- Update public API docs, dexterous RL overview with 4096-env DexUni training/resume/eval baseline, and unit/USD parity tests
…ocs, and tests

- Set UnitreeG1RoughRL-v1 default newton deployment to solver_type: dexuni with Newton explicit heightfield contacts and add paired mjwarp deployment
- Remove deprecated env.dexuni.yaml, ppo.dexuni.yaml, and legacy rough locomotion README/extension keys
- Update locomotion RL overview, rl-learning agent context, g1_rough README, and rough velocity tests
- Collapse g1_rough and the five dexterous manipulation tasks to a single
  default deployment (env.yaml + agents/ppo.yaml) running Newton DexUni on
  GPU MJWarp CUDA, removing 12 duplicate *.newton.yaml and *.mjwarp.yaml files.
- Delete unused offline MJCF migration audit helpers (_audit_model.py,
  _audit_tendons.py) and redundant category README.md.
- Replace recorded JSON snapshot fixtures (g1_rough.json,
  terrain_curriculum.json) with deterministic analytical MDP and curriculum
  unit tests in test_rough_velocity.py.
- Update RL Sphinx documentation, agent_context, and leaf task READMEs to
  match the single-deployment layout and 37/310/310 G1 rough dimensions.
@acrlw
acrlw force-pushed the codex/rl-environments-assets branch from 2db0ef8 to 720a5a0 Compare November 19, 2026 07:46
@yuecideng
yuecideng merged commit fb96705 into main Nov 19, 2026
13 checks passed
@yuecideng
yuecideng deleted the codex/rl-environments-assets branch November 19, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

assets Robot and scene assets, meshes, materials, asset catalogs and downloads. docs Documentation, API references and example explanations. feature New user-facing capabilities. physics Physics backends, dynamics, contacts, fluids, deformables and physics solvers. rl Reinforcement learning algorithms, collectors, trainers and training contracts. task Concrete tasks, deployments, objectives and recipes, including configured Task Program tasks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants