Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .agents/skills/add-task-env/references/rl.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ decision is which environment boundary the prompt requires.
| Differentiable dynamics or APG | Differentiable lightweight env | `trainer.learning_env` |

Do not use a simulator Gym config for APG. Differentiable algorithms require a
`DifferentiableVecEnv` and are rejected on the simulator path.
`DifferentiableEnvProtocol` and are rejected on the simulator path.

## Simulator Gym RL

Expand Down Expand Up @@ -71,7 +71,7 @@ extensions/embodichain_tasks/embodichain_tasks/<category_path>/<task_name>.py
```

Register the factory with `@register_learning_env` and implement the
`LearningVecEnv` structural contract:
`LearningEnvProtocol` structural contract:

- `num_envs`, `device`, `single_observation_space`, and
`single_action_space`;
Expand All @@ -81,7 +81,7 @@ Register the factory with `@register_learning_env` and implement the
- `close()`.

For APG or another differentiable algorithm, implement
`DifferentiableVecEnv.detach_state()` as the truncated-backpropagation
`DifferentiableEnvProtocol.detach_state()` as the truncated-backpropagation
boundary. It must detach dynamic state without resetting or resampling the
task.

Expand Down
22 changes: 16 additions & 6 deletions agent_context/MAP.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ topics:
title: Environment Framework
aliases: [env framework, environment framework, task registration, list task, expert trajectory, 环境框架, 任务环境,
专家轨迹]
keywords: [BaseEnv, EmbodiedEnv, EpisodeContext, EnvCfg, ControlTiming, resolve_control_timing, register_env, list-task, run-task, run-example, target_control_frequency,
keywords: [UnitreeG1MotionRL-v1, motion_tracking, BaseEnv, EmbodiedEnv, EpisodeContext, EnvCfg, ControlTiming, resolve_control_timing, register_env, list-task, run-task, run-example, target_control_frequency,
sim_steps_per_control, step_dt, environment.component, embodiment.component, file-owned physics backend,
physics_config, ControllerAction, EnvProfiler, startup_summary, dexsim_startup_info,
ExpertTrajectoryCfg, joint_command_mode, position_velocity, qpos qvel, expert action schema, ObservationHistorySnapshot, ObservationFrame,
Expand Down Expand Up @@ -112,7 +112,7 @@ topics:
title: IK Solver System
aliases: [ik solver, inverse kinematics, IK 求解器, 逆运动学]
keywords: [BaseSolver, SolverCfg, SRSSolver, OPWSolver, URSolver, PinkSolver, PytorchSolver, PinocchioSolver,
DifferentialSolver, NeuralIKSolver, FEPSolver, FEPSolverCfg, FEP, SSIKSolver, SSIKSolverCfg, SSIK, QposSeedSampler, QposSeedSelSampler, NullSpacePostureTask, compute kinematics, Warp
DifferentialSolver, NeuralIKSolver, FEPSolver, FEPSolverCfg, FEP, SSIKSolver, SSIKSolverCfg, SSIK, TreeKinematics, LinkKinematics, matrix_product, matrix_vector_product, QposSeedSampler, QposSeedSelSampler, NullSpacePostureTask, compute kinematics, Warp
kinematics]
paths: [topics/ik-solvers/ik-solvers.md]
source_of_truth:
Expand Down Expand Up @@ -380,7 +380,7 @@ topics:
title: RL Learning Pipeline
aliases: [rl learning, rl config, reinforcement learning, 强化学习, 训练配置]
keywords: [train-rl, Trainer, trainer.gym_config, learning_env, RolloutKind, DifferentiableTrainer,
PPO, GRPO, APG, SyncCollector, ScheduledDifferentiableVecEnv, DifferentiableRolloutSpec,
PPO, GRPO, APG, APGActorCritic, apg_actor_critic, SyncCollector, ScheduledDifferentiableEnvProtocol, DifferentiableRolloutSpec,
complete rollout, full horizon, gradient accumulation, action adjoint, observation normalization]
paths: [topics/rl-learning/rl-learning.md]
source_of_truth:
Expand Down Expand Up @@ -602,7 +602,7 @@ topics:
title: Data Pipeline
aliases: [real2sim motion, real2sim trajectory, real2sim, 真实轨迹原型, online data, online dataset, async recorder, dataset persistence, simple BC demo, ODS documentation, ODS 文档, 数据管线, 在线数据, 数据集持久化, 数据保存,
深度视频, 数据集校验, 数据集转换, 数据集划分]
keywords: [OnlineDataEngine, OnlineDataset, sample_batch, sample_batch_with_context, lock_index, LeRobotRecorder, AsyncLeRobotRecorder,
keywords: [MotionLibrary, MotionStartSampler, OnlineDataEngine, OnlineDataset, sample_batch, sample_batch_with_context, lock_index, LeRobotRecorder, AsyncLeRobotRecorder,
DepthSidecarManager, depth_meta.json, fragment_id, finalize, expert action schema, qpos_slice, qvel_slice,
validate_dataset, create_split_manifest, recover_recording, episode_uuid, language_persistent,
SharedLanguageRegistry, resolve_language, configure_ods, ODSReuseProfileCfg,
Expand All @@ -617,6 +617,8 @@ topics:
- embodichain/data_pipeline/configuration.py
- embodichain/data_pipeline/real2sim/
- embodichain/data_pipeline/runtime.py
- embodichain/data_pipeline/datasets/motion.py
- embodichain/data_pipeline/datasets/motion_sampling.py
- embodichain/data_pipeline/datasets/online_data.py
- embodichain/data_pipeline/datasets/sequence.py
- embodichain/data_pipeline/datasets/sources.py
Expand Down Expand Up @@ -699,14 +701,22 @@ topics:
- id: robot-world-model
title: Robot World Model
aliases: [robot world model, RWM, 机器人世界模型]
keywords: [RobotWorldModel, RWMPredictor, TransitionReplay, build_rwm_trainer,
keywords: [RobotWorldModel, KinematicWorldModel, MotionTrackingPolicy, MotionModelEnv, OnlineModelBasedSchedule, RWMPredictor, TransitionReplay, build_rwm_trainer,
replay_supervised_batch, learning/rwm]
paths: [topics/robot-world-model/robot-world-model.md]
source_of_truth:
- extensions/embodichain_models/embodichain_models/dynamics/rwm/
- examples/learning/rwm/
- examples/learning/motion_tracking/
- embodichain/learning/dynamics/schedule.py
- extensions/embodichain_models/embodichain_models/policies/motion_tracking.py
- extensions/embodichain_tasks/embodichain_tasks/locomotion/tracking/motion_model.py
watch_paths: [extensions/embodichain_models/embodichain_models/dynamics/rwm/,
examples/learning/rwm/, tests/models/dynamics/rwm/,
examples/learning/rwm/, examples/learning/motion_tracking/, tests/models/dynamics/rwm/,
embodichain/learning/dynamics/, tests/learning/test_online_schedule.py,
extensions/embodichain_models/embodichain_models/policies/motion_tracking.py,
extensions/embodichain_tasks/embodichain_tasks/locomotion/tracking/motion_model.py,
tests/gym/envs/tasks/test_motion_model.py,
docs/source/features/neural_dynamics/rwm.md, docs/source/api_reference/rwm.rst]
related_topics: [supervised-learning, learning-layers, neural-dynamics]
status: active
14 changes: 14 additions & 0 deletions agent_context/topics/data-pipeline/data-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,20 @@ reader. It measures initial loading rather than per-batch disk reads and does
not replace recording or offline-inspection tools. See the
[loading design](../../../docs/architecture/dataset-loading-design.md) for source boundaries.

## Reference motions

`datasets/motion.py` owns `MotionLibrary`: resident root/joint reference states,
explicit clip boundaries and current/future-frame queries. Portable archives
record joint order and quaternion convention. CSV preparation calls
`compute/trajectory/floating.py` for uniform-time resampling and velocity
estimation. Task environments own reference progress and reset selection;
world-model transition collection remains with the learning workflow.

`datasets/motion_sampling.py` owns reference-start curriculum statistics. It mixes
uniform bins, capped low-return softmax and historical low-return weights. Repeated
bins in one completed-episode batch are averaged before EMA updates. The sampler
checkpoints motion layout and statistics; the task owns its named sampling RNG.

## Online worker contract

Shared image buffers are allocated by `init_rollout_buffer_from_config()` in
Expand Down
10 changes: 10 additions & 0 deletions agent_context/topics/env-framework/env-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,16 @@ rewards and contact termination rules remain in the task package; robot defaults
and the distinct asset/contract joint orders are owned by
[robot-system](../robot-system/robot-system.md).

Reference-motion tracking is owned by
`extensions/embodichain_tasks/embodichain_tasks/locomotion/tracking/`.
`g1_motion.py` owns simulator state, reference-frame progress, selected-row resets
and pre-action/terminal snapshots; unusable nonfinite successors hard-terminate
and are excluded from WM supervision. `_task.py` owns tensor objectives shared
with model-driven rollouts. The task's `ReferenceJointPositionAction` records
policy outputs separately from the reference-combined setpoint and executed
joint target. Read the task configuration's README for data preparation,
training and per-clip evaluation commands.

## Launcher argument ownership

`embodichain run-task` is the sole task execution command, dispatched to
Expand Down
2 changes: 2 additions & 0 deletions agent_context/topics/ik-solvers/ik-solvers.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
| Chain/reduced-model construction | `embodichain/lab/sim/utility/solver_utils.py` |
| Seed generation and retrieval | `embodichain/lab/sim/motion/solvers/qpos_seed_sampler.py`, `qpos_seed_sel_sampler.py` in that directory |
| Pink posture integration | `embodichain/lab/sim/motion/solvers/null_space_posture_task.py` |
| Floating-root full-tree FK and link-origin velocities | `embodichain/compute/kinematics/tree.py` → `TreeKinematics`, `LinkKinematics`; URDF geometry and explicit tensor joint order |
| Batched matrix products shared with models/tasks | `embodichain/compute/kinematics/matrix.py` → `matrix_product`, `matrix_vector_product`; compiled small matrices and Torch fallback |
| Analytical kernels | `embodichain/compute/kinematics/_warp/` |
| Bounded B=1 waypoint branch search with solver callbacks | `embodichain/compute/kinematics/ik_path.py`; stateful adapter in `lab/sim/motion/ik.py` |

Expand Down
4 changes: 3 additions & 1 deletion agent_context/topics/learning-layers/learning-layers.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ checkpoint/device protocol. It loads on demand so FK/IK imports do not load Warp
Callers resolve robot caps and own geometric validation. Gradients are first-order
and local to fixed filtering, sampling topology and active constraints.
`learning/models/` owns general MLPs, feature normalization
and configuration-driven MLP construction. These shared components do not import
and configuration-driven MLP construction. `EmpiricalNormalizer.scale` exposes
the configured additive or quadratic epsilon scale; clipping is optional.
These shared components do not import
RL, TensorDict or simulation. Concrete NeRD models, codecs and the shared-SL
recipe live in the bundled `embodichain_models.dynamics.nerd` extension;
teacher collection, historical training and research belong to
Expand Down
42 changes: 35 additions & 7 deletions agent_context/topics/rl-learning/rl-learning.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,19 @@ Select this path with `trainer.learning_env`, either as a registered name or
as a mapping with `name` and `cfg`.

`build_learning_env()` resolves factories registered with
`@register_learning_env`. A learning environment implements the
`LearningVecEnv` protocol. Differentiable algorithms additionally require
`DifferentiableVecEnv.detach_state()` as the truncated-backpropagation
`@register_learning_env`. Online workflows may supply `resources` to
`build_learning_policy_runtime`: the objects are passed by reference to the selected
environment factory, and keys cannot also appear in its scalar configuration.
Custom registered policies may implement `from_config(config, *, obs_space,
action_space, device)` to own their configuration and space validation. A learning environment implements the
`LearningEnvProtocol` protocol. The former `LearningVecEnv`,
`DifferentiableVecEnv` and `ScheduledDifferentiableVecEnv` names remain exact
aliases for downstream import compatibility. Differentiable algorithms additionally require
`DifferentiableEnvProtocol.detach_state()` as the truncated-backpropagation
boundary.

Variable-horizon APG environments may also implement
`ScheduledDifferentiableVecEnv.prepare_differentiable_rollout()`. Its
`ScheduledDifferentiableEnvProtocol.prepare_differentiable_rollout()`. Its
`DifferentiableRolloutSpec` selects the next reset's complete horizon,
per-environment objective scale, and scalar rollout metadata.

Expand Down Expand Up @@ -137,10 +143,10 @@ RolloutKind.DIFFERENTIABLE
→ DifferentiableTrainer
→ DifferentiableCollector
→ graph-connected DifferentiableRollout segments
→ APG update
→ APG or APGActorCritic update
```

PPO and GRPO use `STANDARD`. APG uses `DIFFERENTIABLE`.
PPO and GRPO use `STANDARD`. APG and APGActorCritic use `DIFFERENTIABLE`.
`get_trainer_class()` centralizes this selection for lightweight learning
environments.

Expand All @@ -156,12 +162,24 @@ environment-provided horizon, mask post-terminal rewards, and accumulate full
trajectory gradients before one optimizer step. Read the training detail for
the complete-rollout safety and normalization contracts.

`apg_actor_critic` adds terminal-value gradients, detached lambda-return critic
regression and a separate critic optimizer. Actor/critic parameter groups must
be disjoint. Time truncations require the full pre-reset observation batch in
`info["terminal_observation"]`; hard termination suppresses bootstrap. Optional
soft continuation retains its input derivative. Task-data validity masks are
detached. See the field contract in `DifferentiableEnvProtocol` and
`DifferentiableTransition`. Normalizer updates belong between optimizer updates.

`DifferentiableTrainer.update_once()` provides the collection/update boundary
for online model learning, recording counters and training metrics. The outer
workflow owns evaluation and persistence when using this single-update API.

## Component Ownership

| Component | Owner |
|-----------|-------|
| Algorithm base, rollout kind, optimizer scheduling | `algo/base.py`, `utils/optimizer.py` |
| PPO, GRPO, APG implementations | `algo/ppo.py`, `algo/grpo.py`, `algo/apg.py` |
| Algorithm implementations | `algo/ppo.py`, `algo/grpo.py`, `algo/apg.py`, `algo/apg_actor_critic.py` |
| Standard rollout storage and views | `buffer/` |
| Standard and differentiable collection | `collector/` |
| Policy interface, actor-critic, actor-only, policy builder | `learning/rl/policies/` |
Expand Down Expand Up @@ -227,6 +245,16 @@ import from `embodichain.learning.models`. See [learning layers](../learning-lay
for differentiable IK ownership. Root exports load on demand; importing a
lightweight APG component must not eagerly load the simulator-backed Trainer.

APG Actor-Critic can call `policy.update_critic_normalization()` after actor
backward and before constructing detached lambda targets. Set
`update_critic_normalization=True` for that contract; the default leaves critic
statistics unchanged. Actor collection and backward keep critic parameters and
statistics fixed while preserving derivatives through critic inputs. Statistics
include weighted current states and valid bootstrap successors absent from the
next current-state row; complete-rollout post-terminal suffixes are excluded.
Critic minibatches contain only positive-weight samples. Empty valid sets leave
critic parameters and optimizer state unchanged, including Adam momentum.

## Invariants

- The selected environment must expose batched observation/action spaces and
Expand Down
4 changes: 2 additions & 2 deletions agent_context/topics/rl-learning/training.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,12 +131,12 @@ example is an experimental gradient reference, not a general simulator task.

### Add a Lightweight Environment

1. Implement `LearningVecEnv`, or `DifferentiableVecEnv` for APG.
1. Implement `LearningEnvProtocol`, or `DifferentiableEnvProtocol` for APG.
2. Register the factory with `@register_learning_env`.
3. Ensure finished rows auto-reset while returning terminal reward/done with
the next initial observation.
4. For variable complete APG rollouts, implement
`ScheduledDifferentiableVecEnv` and return the full non-truncated horizon.
`ScheduledDifferentiableEnvProtocol` and return the full non-truncated horizon.
5. Expose `observation_normalize_mask` when semantic dimensions must remain
raw during normalization.
6. Add an official config under
Expand Down
Loading
Loading