Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b1908b9
feat(tasks): add rough-terrain and dexterous RL task candidates
acrlw Oct 21, 2026
f036b59
fix(tasks): preserve G1 contact observations across reset
acrlw Oct 21, 2026
490903d
feat(rl): enable G1 DexUni and v2 curriculum resume
acrlw Oct 21, 2026
a185a20
fix(rl): retain episode success and update portable task assets
acrlw Oct 22, 2026
ae083db
perf(rl): disable unused per-step context snapshots for G1 rough
acrlw Oct 22, 2026
96044b8
fix(rl): reserve face-turn constraint capacity for large batches
acrlw Oct 22, 2026
105b382
perf(tasks): use cubql BVHs for G1 rough terrain queries
acrlw Oct 22, 2026
c376eaf
fix: Match the G1 asset archive with its bundled BSD license
acrlw Oct 22, 2026
89612ba
fix(tasks): explicitly enable G1 terrain heightfield conversion
acrlw Oct 23, 2026
8b60d52
feat(tasks): load dexterous manipulation scenes from USD
acrlw Nov 2, 2026
560c09e
Use existing MuJoCo ownership selectors for USD manipulation tasks
acrlw Nov 2, 2026
dd310d7
docs: add reproducible manipulation asset and task checks
acrlw Nov 3, 2026
6e7f744
Preserve terminal curriculum state and isolate task reset seeds
acrlw Nov 9, 2026
69069ae
Place G1 terrain guidance alongside locomotion controls
acrlw Nov 9, 2026
543f299
test(tasks): model ground redeclaration in the G1 fixture
acrlw Nov 9, 2026
3349c68
fix(tasks): preserve flat action clipping and terminal bootstrap rules
acrlw Nov 10, 2026
c01091d
Align G1 rough locomotion naming with main and remove unused RSL-RL w…
acrlw Nov 13, 2026
bd022b9
feat(tasks): refactor dexterous manipulation tasks to USD-native Embo…
acrlw Nov 13, 2026
a26c004
feat(tasks): default g1_rough to DexUni and align RL task catalogs, d…
acrlw Nov 13, 2026
c0d1b55
fix(sim): preserve default_plane in asset_uids unless remove_ground_p…
acrlw Nov 13, 2026
dcd053d
Unify DexUni task configs and prune offline audit/fixture bloat
acrlw Nov 13, 2026
a59f090
fix(tasks): correct rough and dexterous RL state handling
acrlw Nov 14, 2026
f08d8dc
test: isolate asset materialization path override
acrlw Nov 14, 2026
720a5a0
fix(tasks): align sensor sampling and rollout tests with main
acrlw Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions agent_context/topics/data-assets/data-assets.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,3 +255,10 @@ under the same lock used by preparation. Historical URDF preparation and
model-fidelity metadata are owned by
`embodichain_benchmarks.assembly.robot_reference`, not the dataset resolver or a
combined arm/gripper catalog preset.

The five classes in `assets/rl_assets.py` use the same archive cache. The
`official_rl` entry point registers four robot entries and the `G1RoughTerrains`
scene entry through the shared `embodichain.assets` catalog. CLI discovery and
`get_data_class()` resolve the same class identities. The G1 Rough asset does not replace
`UnitreeG1Locomotion`; their joint counts and provenance differ. Publication
status is external to code registration and must be verified before release.
17 changes: 17 additions & 0 deletions agent_context/topics/rl-learning/rl-learning.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,3 +211,20 @@ Use `tests/learning/` for routing, algorithms, buffers, gradients and checkpoint
contracts; run native/distributed suites only for those resource boundaries.
Missing names point to registries/discovery, disappearing gradients to early
detachment, and reset/terminal issues to the selected environment protocol.

## Rough terrain and dexterous manipulation tasks

`locomotion/velocity/g1_rough.py` registers the 37-action G1 task (`UnitreeG1RoughRL-v1`).
Its terrain curriculum comes from `G1RoughTerrains` and its robot comes from
`UnitreeG1Rough`, independently of the 29-joint G1 flat archive. Its catalog
exposes a single `default` deployment (`env.yaml` + `agents/ppo.yaml`) running
`physics: newton` with `solver_type: dexuni` and Newton heightfield contacts.

`manipulation/dexterous/` registers five USD-native `EmbodiedEnv` tasks
(`ShadowPenFixedRL-v1`, `ShadowPenContinuousRL-v1`, `LeapCubeFaceTurnRL-v1`,
`LeapCubeReorientRL-v1`, `LeapCubeRotateZAxisRL-v1`) backed by
`DexterousManipulationEnv`. Runtime scene loading uses `SimulationManager.add_usd()`
and public `Robot`/`Articulation`/`RigidObject` facades with zero MJCF dependency;
each task catalog exposes a single `default` deployment (`env.yaml` +
`agents/ppo.yaml`) running `physics: newton` with `solver_type: dexuni` (GPU
MJWarp CUDA).
8 changes: 8 additions & 0 deletions agent_context/topics/rl-learning/training.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ Variable-horizon APG transition budgets use the actual update count to check
an explicit restored LR horizon.
Legacy standard/APG-v1 files restore available fields and report missing metadata
or RNG. `checkpoint.py` owns shared validation and atomic persistence.
Optional environment training state uses the same v2 file and is collected per
rank. Environments implement `get_training_state`, side-effect-free
`validate_training_state`, and `load_training_state`; validation runs before
learning state changes, and restoration precedes the fresh episode reset.
Missing state in older files produces a warning for environments with these
hooks. G1 Rough uses them for terrain assignments and its task generator,
requiring the original terrain map and per-rank environment count. Its first
reset preserves the restored assignments; physical episodes still restart.
Configuration snapshots are frozen from loaded data on the first save; later
saves update the run manifest without rewriting those snapshots. Final saves
reuse the current update's periodic or best checkpoint. Subsequent saves and final
Expand Down
272 changes: 272 additions & 0 deletions docs/source/api_reference/public_api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5990,6 +5990,278 @@ native listener binds all IPv4 interfaces and lives until World teardown.
:no-index:


embodichain.data.assets.rl_assets
---------------------------------

Robot, reset-state and frozen-terrain archives use the standard asset cache.

.. currentmodule:: embodichain.data.assets.rl_assets

.. autosummary::

UnitreeG1Rough
G1RoughTerrains
ShadowHandPen
LeapHandCubeFaceTurn
LeapHandCube

.. autoclass:: UnitreeG1Rough
:members:

.. autoclass:: G1RoughTerrains
:members:

.. autoclass:: ShadowHandPen
:members:

.. autoclass:: LeapHandCubeFaceTurn
:members:

.. autoclass:: LeapHandCube
:members:


embodichain_tasks.locomotion.velocity.g1_rough
----------------------------------------------

The G1 Rough environment registers the 37-action rough-terrain locomotion task with terrain height observations.

.. currentmodule:: embodichain_tasks.locomotion.velocity.g1_rough

.. autosummary::

UnitreeG1RoughEnv

.. autoclass:: UnitreeG1RoughEnv
:members:


embodichain_tasks.locomotion.velocity.rough
-------------------------------------------

Load rough-terrain velocity task configurations from a supplied scene bundle. The loader validates robot joint order, shared initial states and terrain inputs.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough

.. autosummary::

load_rough_env_cfg

.. autofunction:: load_rough_env_cfg


embodichain_tasks.locomotion.velocity.rough.config
--------------------------------------------------

Packaged task definitions retain policy joint order, observation dimensions and fixed physics/control periods.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough.config

.. autosummary::

RoughVelocityConfig
load_config

.. autoclass:: RoughVelocityConfig
:members:

.. autofunction:: load_config


embodichain_tasks.locomotion.velocity.rough.curriculum
------------------------------------------------------

Terrain assignment and selected-row curriculum updates use terminal positions and commands before resetting each episode.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough.curriculum

.. autosummary::

TerrainCurriculum

.. autoclass:: TerrainCurriculum
:members:


embodichain_tasks.locomotion.velocity.rough.environment
-------------------------------------------------------

Resolve the rough-terrain deployment against exported robot and terrain assets, with optional training randomization and contact generation selection.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough.environment

.. autosummary::

load_rough_env_cfg

.. autofunction:: load_rough_env_cfg


embodichain_tasks.locomotion.velocity.rough.mdp
-----------------------------------------------

Pure Torch state, action, observation, reward and termination functions share the rough-terrain task definitions. Device constants cache fixed values and reward joint selectors.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough.mdp

.. autosummary::

RoughVelocityState
RoughVelocityReward
RoughVelocityConstants
action_target
observation_terms
build_observation
compute_rewards
compute_termination
dc_motor_effort
update_contact_times

.. autoclass:: RoughVelocityState
:members:

.. autoclass:: RoughVelocityReward
:members:

.. autoclass:: RoughVelocityConstants
:members:

.. autofunction:: action_target

.. autofunction:: observation_terms

.. autofunction:: build_observation

.. autofunction:: compute_rewards

.. autofunction:: compute_termination

.. autofunction:: dc_motor_effort

.. autofunction:: update_contact_times


embodichain_tasks.locomotion.velocity.rough.runtime
---------------------------------------------------

The runtime adapts the shared task definitions to EmbodiChain managers, state and sensors. Explicit motors retain the configured outer physics cadence; implicit torque telemetry is sampled before the final physical step.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough.runtime

.. autosummary::

RoughVelocityEnv

.. autoclass:: RoughVelocityEnv
:members:


embodichain_tasks.locomotion.velocity.rough.terrain
---------------------------------------------------

The scanner evaluates 187 torso-relative terrain rays and replays a fixed-buffer CUDA graph when the geometry and buffers remain unchanged.

.. currentmodule:: embodichain_tasks.locomotion.velocity.rough.terrain

.. autosummary::

HeightScanner

.. autoclass:: HeightScanner
:members:


embodichain_tasks.manipulation.dexterous
----------------------------------------

USD-native Shadow Hand and LEAP Hand dexterous manipulation RL environments built on ``EmbodiedEnv``.

.. currentmodule:: embodichain_tasks.manipulation.dexterous

.. autosummary::

LeapCubeFaceTurnEnv
LeapCubeReorientEnv
LeapCubeRotateZAxisEnv
ShadowPenContinuousEnv
ShadowPenFixedEnv


embodichain_tasks.manipulation.dexterous.leap_cube_face_turn
------------------------------------------------------------

LEAP Hand top-layer articulated cube face-turning environment (``LeapCubeFaceTurnRL-v1``).

.. currentmodule:: embodichain_tasks.manipulation.dexterous.leap_cube_face_turn

.. autosummary::

LeapCubeFaceTurnEnv

.. autoclass:: LeapCubeFaceTurnEnv
:members:


embodichain_tasks.manipulation.dexterous.leap_cube_reorient
-----------------------------------------------------------

LEAP Hand in-hand cube 3D orientation tracking environment (``LeapCubeReorientRL-v1``).

.. currentmodule:: embodichain_tasks.manipulation.dexterous.leap_cube_reorient

.. autosummary::

LeapCubeReorientEnv

.. autoclass:: LeapCubeReorientEnv
:members:


embodichain_tasks.manipulation.dexterous.leap_cube_rotate_z
-----------------------------------------------------------

LEAP Hand continuous cube Z-axis rotation environment (``LeapCubeRotateZAxisRL-v1``).

.. currentmodule:: embodichain_tasks.manipulation.dexterous.leap_cube_rotate_z

.. autosummary::

LeapCubeRotateZAxisEnv

.. autoclass:: LeapCubeRotateZAxisEnv
:members:


embodichain_tasks.manipulation.dexterous.shadow_pen_continuous
--------------------------------------------------------------

Shadow Hand continuous multi-pose pen-spinning environment (``ShadowPenContinuousRL-v1``).

.. currentmodule:: embodichain_tasks.manipulation.dexterous.shadow_pen_continuous

.. autosummary::

ShadowPenContinuousEnv

.. autoclass:: ShadowPenContinuousEnv
:members:


embodichain_tasks.manipulation.dexterous.shadow_pen_fixed
---------------------------------------------------------

Shadow Hand fixed-grasp pen-spinning environment (``ShadowPenFixedRL-v1``).

.. currentmodule:: embodichain_tasks.manipulation.dexterous.shadow_pen_fixed

.. autosummary::

ShadowPenFixedEnv

.. autoclass:: ShadowPenFixedEnv
:members:


Parallel-jaw hand configuration and composition
----------------------------------------------

Expand Down
48 changes: 48 additions & 0 deletions docs/source/overview/rl/dexterous.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Pen and cube manipulation

Five USD-native dexterous manipulation tasks inherit from `EmbodiedEnv` (`DexterousManipulationEnv`) and integrate with the task catalog, `run-task`, `train-rl`, and `eval-policy`. Each task loads its physical scene directly from packaged USD (`.usdc`) via `SimulationManager.add_usd()` with zero runtime MJCF dependency.

| Task directory | Registered environment | Actions | Actor / critic observations | Asset |
|---|---|---:|---:|---|
| `shadow_pen_fixed` | `ShadowPenFixedRL-v1` | 18 | 87 / 87 | `ShadowHandPen` |
| `shadow_pen_continuous` | `ShadowPenContinuousRL-v1` | 18 | 87 / 87 | `ShadowHandPen` |
| `leap_cube_face_turn` | `LeapCubeFaceTurnRL-v1` | 16 | 65 / 65 | `LeapHandCubeFaceTurn` |
| `leap_cube_reorient` | `LeapCubeReorientRL-v1` | 16 | 57 / 128 | `LeapHandCube` |
| `leap_cube_rotate_z` | `LeapCubeRotateZAxisRL-v1` | 16 | 32 / 105 | `LeapHandCube` |

## Configuration and CLI Usage

Each task directory under `extensions/embodichain_tasks/configs/tasks/manipulation/dexterous/<task>/` provides a single `default` catalog deployment (`env.yaml` and `agents/ppo.yaml`) configured with `physics: newton` and `solver_type: dexuni` (`use_mujoco_cpu: false`, `use_mujoco_contacts: true`, GPU MuJoCo Warp on CUDA with full scene joint ownership).

```bash
embodichain data download --name ShadowHandPen
embodichain show-task embodichain_tasks:shadow_pen_fixed
embodichain run-task --gym_config extensions/embodichain_tasks/configs/tasks/manipulation/dexterous/shadow_pen_fixed/env.yaml
embodichain train-rl --config extensions/embodichain_tasks/configs/tasks/manipulation/dexterous/shadow_pen_fixed/agents/ppo.yaml
```

Continue training from a saved checkpoint with `embodichain train-rl --config CONFIG --resume latest --run RUN_DIRECTORY`, setting `trainer.iterations` to the cumulative update target. Evaluate a saved checkpoint via `embodichain eval-policy RUN_DIRECTORY --checkpoint latest --num-envs 64 --episodes 64 --renderer no-render`.

## Task Semantics and Evaluation

Fixed and continuous pen-spinning share the `ShadowHandPen` USD scene; continuous pen-spinning samples across 22 initial grasp poses. Cube face-turning (`LeapCubeFaceTurnRL-v1`) models the cube as a single-hinge articulation (`face_turn`) and applies passive detent torque before each physics substep. Cube reorientation (`LeapCubeReorientRL-v1`) and Z-axis rotation (`LeapCubeRotateZAxisRL-v1`) construct asymmetric actor and privileged critic observation groups from public `Robot` and `RigidObject` state, site transforms parsed from USD, and pre-integration PD actuator forces.

Evaluation reports `eval/success_rate` for `ShadowPenFixedRL-v1`, `ShadowPenContinuousRL-v1`, `LeapCubeFaceTurnRL-v1`, and `LeapCubeReorientRL-v1`: an episode counts as successful if at least one task success event occurs before termination or truncation (`info["success"]`), while per-step pulses remain in `info["metrics"]["success"]`. `LeapCubeRotateZAxisRL-v1` optimizes unbounded angular velocity without a binary success threshold; evaluate `rotation`, `angular_velocity_z`, `retained`, episode length, and return instead.

## USD Physical Fidelity and Solver Settings

All five environments import fixed tendons, joint limits, actuator `ctrlrange`, PD `target_ke`/`target_kd`, site local poses, and collision filter pairs directly from `.usdc`.

`LeapCubeFaceTurnRL-v1` reserves `njmax: 2048` constraint rows per world (`512` for the remaining tasks). Each `env.yaml` preserves authored solver settings (`20/50` Newton/line-search iterations on Shadow Hand tasks; `5/8` on LEAP Hand tasks) and sets `arena_space: 0.0` with `separate_worlds: true` so execution-model geometry stays spatially uniform across parallel worlds.

## DexUni Training and Evaluation Baseline

Sustained single-GPU PPO training, `--resume latest` continuation (`+10` updates), and `64`-episode `eval-policy` results under `env.yaml` (`physics: newton`, `solver_type: dexuni`, `4096` parallel environments, single NVIDIA RTX PRO 5000 Blackwell 72 GB GPU):

| Environment | Updates (`train + resume`) | Transitions | Steady SPS | Train `episode_reward_avg_100` (`start -> end`) | `eval-policy` (`64` episodes, `--checkpoint latest`) |
|---|---:|---:|---:|---|---|
| `ShadowPenFixedRL-v1` | `150 + 10` | `26,214,400` | `54,106` | `-50.49 -> +47.94` | `avg_reward=+74.26`, `avg_length=500.0`, `retained=100%`, `error=0.143 rad`, `rotation=2.40 rad` |
| `ShadowPenContinuousRL-v1` | `150 + 10` | `52,428,800` | `57,265` | `-39.46 -> +14.47` | `avg_reward=+9.64`, `avg_length=485.8`, `retained=96.9%`, `error=0.229 rad`, `rotation=0.82 rad` |
| `LeapCubeFaceTurnRL-v1` | `150 + 10` | `26,214,400` | `55,562` | `-0.47 -> +216.71` | `avg_reward=+255.53`, `avg_length=500.0`, `success_rate=100%`, `success_count=12.52`, `rotation=20.56 rad` |
| `LeapCubeReorientRL-v1` | `300 + 10` | `50,790,400` | `48,218` | `-4.58 -> +172.64` | `avg_reward=+180.98`, `avg_length=997.6`, `retained=98.4%`, `success_rate=3.1%`, `position_error=0.020 m` |
| `LeapCubeRotateZAxisRL-v1` | `300 + 10` | `50,790,400` | `94,216` | `-5.56 -> +27.40` | `avg_reward=+29.35`, `avg_length=497.5`, `retained=98.4%`, `rotation=29.43 rad`, `angular_velocity_z=1.39 rad/s` |
1 change: 1 addition & 0 deletions docs/source/overview/rl/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ See also
train_script.md
multi_gpu.md
locomotion.md
dexterous.md

See Also
--------
Expand Down
Loading
Loading