Repository navigation
Add G1 rough locomotion and USD-native dexterous manipulation RL tasks - #837
Conversation
[Medium impact] The PR adds new RL workflows and extends the shared training-resume path. The PR is not ready to merge while the earlier credential leak and flat-task action-limit regression remain unresolved.
|
a52c3c2 to
fc534c0
Compare
b9448d4 to
944715e
Compare
3649faf to
9ed54ed
Compare
| session = requests.Session() | ||
| session.headers["Authorization"] = f"Bearer {api_key}" | ||
| kwargs["session"] = session |
There was a problem hiding this comment.
A geometry profile with api_key puts its bearer token on the shared session. _download_glb() uses that session for the server-provided mesh URL, while _mesh_url() accepts URLs on any host. A response pointing to a separate file host therefore sends the geometry service's secret to that host, even during a normal download.
Keep credentials on requests to the configured service only. Use an unauthenticated session for external downloads or reject external URLs.
How this was verified: The geometry download uses the session carrying Authorization, and the server-provided absolute URL passes through without a host check.
Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/gen_sim/services/runtime.py
Line: 186-188
Comment:
**Downloads expose the API key**
A geometry profile with `api_key` puts its bearer token on the shared session. `_download_glb()` uses that session for the server-provided `mesh` URL, while `_mesh_url()` accepts URLs on any host. A response pointing to a separate file host therefore sends the geometry service's secret to that host, even during a normal download.
Keep credentials on requests to the configured service only. Use an unauthenticated session for external downloads or reject external URLs.
**How this was verified:** The geometry download uses the session carrying `Authorization`, and the server-provided absolute URL passes through without a host check.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Implemented in #1137, commit 67b696d8a.
Fixed the HTTP profile session so the service Bearer token is attached only when the request's scheme, host and effective port match the configured service origin. External mesh downloads receive no profile Authorization header, and cross-origin redirects strip it, including same-host HTTP-to-HTTPS redirects. Seven transport regression cases cover service-relative URLs, external URLs, port changes and redirects. The affected code is already on main, so this fix is kept separate from the RL task changes.
| "preserve_order": True, | ||
| "offset": self.velocity_task_config.default_joint_position, | ||
| "scale": self.velocity_task_config.action_scale, | ||
| "clip": None, |
There was a problem hiding this comment.
Setting clip to None in the shared parent also disables clipping for existing flat locomotion tasks. Their configurations leave params empty, so they previously used DefaultJointPositionAction's limit of 1.0. PPO outputs outside [-1, 1] now reach joint targets and action-history rewards unchanged, even though these tasks still declare that bounded action space. This silently changes training and saved-policy behavior.
Set the unbounded default only for IsaacVelocityEnv and preserve the existing tasks' clipping.
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/embodichain_tasks/embodichain_tasks/locomotion/velocity/_embodichain.py
Line: 111
Comment:
**Flat tasks lose action limits**
Setting `clip` to `None` in the shared parent also disables clipping for existing flat locomotion tasks. Their configurations leave `params` empty, so they previously used `DefaultJointPositionAction`'s limit of 1.0. PPO outputs outside `[-1, 1]` now reach joint targets and action-history rewards unchanged, even though these tasks still declare that bounded action space. This silently changes training and saved-policy behavior.
Set the unbounded default only for `IsaacVelocityEnv` and preserve the existing tasks' clipping.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Implemented in 528958cf2.
Fixed the shared velocity environment to retain the original clip=1.0 default. Only IsaacVelocityEnv defaults to clip=None, and explicit action-term settings still take precedence. The regression exercises the actual constructor and action-processing path: an input of 2.0 at scale 0.5 produces a target of 0.5 for the flat environment and 1.0 for Isaac. Explicit clipping and unbounded overrides are also covered.
| if self.task_id in ("M1", "M2"): | ||
| obs, reward, terminated, metrics = self.task.step(state, actions) | ||
| self.elapsed_steps += 1 | ||
| truncated = self.elapsed_steps >= 500 |
There was a problem hiding this comment.
Pen failures get timeout bonuses
A pen task that fails on step 500 returns both terminated and truncated as true. The shared compute_gae() adds a value estimate whenever truncated is true, so that physical failure incorrectly receives a time-limit bonus. This gives the final transition the wrong training return.
Match the LEAP tasks by excluding terminated rows from truncated.
| truncated = self.elapsed_steps >= 500 | |
| truncated = (self.elapsed_steps >= 500) & ~terminated |
Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/embodichain_tasks/embodichain_tasks/manipulation/dexterous/_runtime.py
Line: 681
Comment:
**Pen failures get timeout bonuses**
A pen task that fails on step 500 returns both `terminated` and `truncated` as true. The shared `compute_gae()` adds a value estimate whenever `truncated` is true, so that physical failure incorrectly receives a time-limit bonus. This gives the final transition the wrong training return.
Match the LEAP tasks by excluding terminated rows from `truncated`.
```suggestion
truncated = (self.elapsed_steps >= 500) & ~terminated
```
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Implemented in 528958cf2.
Fixed the M1/M2 adapter to set truncated = (elapsed_steps >= 500) & ~terminated. Physical failure now takes precedence when it coincides with the time limit. The regression runs both tasks through the real step/reset bridge and GAE: with reward 1, successor value 10 and gamma 0.9, failure at step 500 returns 1, while a pure timeout returns 10.
|
@greptileai review this PR again. Please review the full current diff against main at c4a6f79; it has been substantially simplified and updated since the previous review. |
c4a6f79 to
2db0ef8
Compare
…diedEnv with DexUni default - Replace runtime MJCF loading and _dexuni_adapter with USD-native DexterousManipulationEnv(EmbodiedEnv) loading ShadowHandPen, LeapHandCubeFaceTurn, and LeapHandCube .usdc scenes via SimulationManager.add_usd() - Add SimulationManager.remove_ground_plane() and clear_articulation_dynamics() for groundless and multi-articulation Newton scenes - Standardize Gym IDs (*RL-v1) and catalog deployments (newton -> solver_type: dexuni default; mjwarp -> solver_type: mujoco_warp) across all 5 dexterous tasks - Update public API docs, dexterous RL overview with 4096-env DexUni training/resume/eval baseline, and unit/USD parity tests
…ocs, and tests - Set UnitreeG1RoughRL-v1 default newton deployment to solver_type: dexuni with Newton explicit heightfield contacts and add paired mjwarp deployment - Remove deprecated env.dexuni.yaml, ppo.dexuni.yaml, and legacy rough locomotion README/extension keys - Update locomotion RL overview, rl-learning agent context, g1_rough README, and rough velocity tests
- Collapse g1_rough and the five dexterous manipulation tasks to a single default deployment (env.yaml + agents/ppo.yaml) running Newton DexUni on GPU MJWarp CUDA, removing 12 duplicate *.newton.yaml and *.mjwarp.yaml files. - Delete unused offline MJCF migration audit helpers (_audit_model.py, _audit_tendons.py) and redundant category README.md. - Replace recorded JSON snapshot fixtures (g1_rough.json, terrain_curriculum.json) with deterministic analytical MDP and curriculum unit tests in test_rough_velocity.py. - Update RL Sphinx documentation, agent_context, and leaf task READMEs to match the single-deployment layout and 37/310/310 G1 rough dimensions.
2db0ef8 to
720a5a0
Compare
Description
Add the 37-DOF Unitree G1 rough-terrain locomotion task (
UnitreeG1RoughRL-v1) and five USD-native dexterous manipulation RL tasks (ShadowPenFixedRL-v1,ShadowPenContinuousRL-v1,LeapCubeFaceTurnRL-v1,LeapCubeReorientRL-v1,LeapCubeRotateZAxisRL-v1) together with their registered Hugging Face asset archives (UnitreeG1Rough,G1RoughTerrains,ShadowHandPen,LeapHandCubeFaceTurn,LeapHandCube), task catalogs, PPO configurations, and documentation.Summary of Changes
EmbodiedEnv):DexterousManipulationEnv(EmbodiedEnv)and five registered Gym tasks (ShadowPenFixedRL-v1,ShadowPenContinuousRL-v1,LeapCubeFaceTurnRL-v1,LeapCubeReorientRL-v1,LeapCubeRotateZAxisRL-v1) that load scene geometry, fixed tendons, joint limits, actuatorctrlrange, PDtarget_ke/target_kd, site local poses, and collision filter pairs directly from packaged.usdcfiles viaSimulationManager.add_usd()with zero runtime MJCF dependency.SimulationManager.remove_ground_plane()andSimulationManager.clear_articulation_dynamics()to support groundless scenes and multi-articulation reset velocity clearing on Newton backends.UnitreeG1RoughRL-v1):RoughVelocityEnv(EmbodiedEnv), heightfield terrain curriculum (G1RoughTerrains), 187-point local elevation scanner, and schema-v2 checkpoint curriculum state save/restore (--resume).DexUniTask Catalogs:extensions/embodichain_tasks/configs/tasks/withcatalog.yaml(default_deployment: default),env.yaml(physics: newton,solver_type: dexuni,use_mujoco_cpu: false, GPU MuJoCo Warp on CUDA),agents/ppo.yaml, andREADME.md.93tests intests/gym/envs/task_program/test_asset_materialization.pyandtests/data/test_download.pypass.173 passed, 1 deselectedacrosstest_rough_velocity.py,test_dexterous_metrics.py,test_dexterous_usd.py,test_unitree_locomotion.py, andtest_rl_resume.pywith-m "not gpu and not requires_sim". The targeted regression selection reproduced 14 failing cases before the fixes. Existing test modules cover configured targets, all custom stepping paths, oriented COM offsets, and M1–M5 timeout/GAE behavior.black --check --diff --color ./(Black 26.10.1),git diff --check, and API docs coverage (4920/4920) pass. Reviewed environment and RL agent context; these fixes restore the documented action, sensor, and episode behavior without changing public APIs or usage.tests/gym/envs/tasks/test_dexterous_metrics.py,test_dexterous_usd.py,test_rough_velocity.py,test_rough_velocity_frame.py,test_unitree_locomotion.py,test_locomotion_deployments.py,tests/gym/envs/test_official_task_layout.py,tests/docs/test_check_api_docs.py,tests/data/test_download.py,tests/learning/test_rl_resume.py,tests/cli/):339 passed, 14 skipped;docs/scripts/check_api_docs.py:4920/4920exports aligned.4096parallel environments,solver_type: dexuni, NVIDIA RTX PRO 5000 Blackwell 72 GB),--resume latest(+10updates), and64-episodeeval-policy:The training results below predate the runtime correctness fixes. Sustained training and policy evaluation were not repeated for these fixes.
train + resume)episode_reward_avg_100(start -> end)eval-policy(64episodes,--checkpoint latest)UnitreeG1RoughRL-v1150 + 1015,728,64067,576-4.71 -> -4.19avg_reward=-3.40,avg_length=45.25,linear_velocity_error=0.649,root_height=0.789 mShadowPenFixedRL-v1150 + 1026,214,40054,106-50.49 -> +47.94avg_reward=+74.26,avg_length=500.0,retained=100%,error=0.143 rad,rotation=2.40 radShadowPenContinuousRL-v1150 + 1052,428,80057,265-39.46 -> +14.47avg_reward=+9.64,avg_length=485.8,retained=96.9%,error=0.229 rad,rotation=0.82 radLeapCubeFaceTurnRL-v1150 + 1026,214,40055,562-0.47 -> +216.71avg_reward=+255.53,avg_length=500.0,success_rate=100%,success_count=12.52,rotation=20.56 radLeapCubeReorientRL-v1300 + 1050,790,40048,218-4.58 -> +172.64avg_reward=+180.98,avg_length=997.6,retained=98.4%,success_rate=3.1%,position_error=0.020 mLeapCubeRotateZAxisRL-v1300 + 1050,790,40094,216-5.56 -> +27.40avg_reward=+29.35,avg_length=497.5,retained=98.4%,rotation=29.43 rad,angular_velocity_z=1.39 rad/sType of change
Checklist
black .command to format the code base.python docs/scripts/check_api_docs.py), if applicable