Skip to content

Add online world-model learning and APG motion tracking - #924

Merged
yuecideng merged 16 commits into
mainfrom
codex/supermimic-maintainer-candidate
Nov 11, 2026
Merged

yuecideng merged 16 commits into
mainfrom
codex/supermimic-maintainer-candidate

Conversation

@acrlw

@acrlw acrlw commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR adds online world-model learning and APG Actor-Critic with a G1 motion-tracking task. A physical environment collects state transitions, supervised learning updates the world model, and the policy learns by backpropagating through short model rollouts. Physical collection, the model task environment, and independent evaluation share observation, reward, and termination calculations.

Refs #848

Changes

  • Add APGActorCritic for short-rollout policy updates and value regression with lambda-return targets. Policy updates freeze world-model and critic parameters while retaining state and action gradients; normalization statistics remain fixed during rollouts.
  • Add OnlineModelBasedSchedule for collection, model warmup, and model/policy update ordering and ratios; add replay history and task-progress fields, and multi-step dynamics training through the shared SupervisedLearner.
  • Validate the G1 composition through trainer / policy / algorithm: MotionTrackingModelRL, motion_tracking, and apg_actor_critic, with a fixed kinematic world model and replay schema. The online workflow selects collection and evaluation environments separately and injects the shared WM, motion library, and replay start provider. Checkpoint recovery preserves optimizers, scheduling, and sampling state.
  • Add motion loading, resampling, reference-state queries, full-body kinematics, and G1 physical and model task environments.
  • Provide examples for data preparation, pretrained-model download, training, checkpoint recovery, policy training with a frozen world model, independent physical evaluation, and multi-step prediction validation. Runs record configuration, runtime environment, and training metrics.

Usage and dependencies

Entry points in examples/learning/motion_tracking/ run through embodichain run-example for data preparation, model download, training, evaluation, and playback. Data preparation uses a pinned upstream revision and preserves attribution to Ubisoft, the Unitree retargeting project, and the download mirror. Pretrained weights are available in embodichain_model. The example README provides complete commands.

The physical task uses Newton/MJWarp and depends on the shared simulation interfaces in #836. FastRT saved-trajectory playback uses RobotMarkers from #833.

Review order

This branch contains the learning and G1 task changes on main. The physical task uses the simulation APIs in #836, and saved-trajectory playback uses RobotMarkers from #833. Run integration CI after both prerequisites merge into main.

Area Entry points Main checks
Training setup examples/learning/motion_tracking/README.md, g1_motion/agents/model_based.yaml Data flow, configuration and update ratios
Policy learning learning/rl/algo/apg_actor_critic.py, differentiable collector and trainer Pathwise gradients, bootstrap and normalization boundaries
Model learning learning/dynamics/schedule.py, embodichain_models/dynamics/rwm/ Replay history, supervised targets and frozen-model rollouts
Task and geometry embodichain_tasks/locomotion/tracking/, motion policy, compute/kinematics/ Shared task calculations, observations and state conventions
Execution examples/learning/motion_tracking/ Training recovery, physical evaluation and playback

Validation with DexSim 0.6.0rc1

The branch uses the published wheel pinned by #1124. An earlier GitHub test run found a missing explicit visualization configuration in replay.py. The headless exporter now declares its live visualization backend as disabled. The failing example-coverage check passes locally. Previous CI failure

Torch 2.7 full-graph compilation

The Python 3.11 CI lane uses Torch 2.7.0+cu128. A default-device context made Dynamo fail on Tensor.unflatten during heading extraction. The same condition also affected compiled model transitions and supervised rollout loss in local reproduction. These paths now use equivalent reshape operations. The existing regressions explicitly exercise a device context and check values and gradients. Failing CI run

  • 47 focused tests passed with Python 3.11, Torch 2.7.0+cu128, and the published DexSim 0.6.0rc1 wheel.
  • 45 focused tests passed with Python 3.12 and Torch 2.11.
  • The exact-version before/after check reproduced the original full-graph failure. The updated heading output and gradients matched the original eager path for unbatched and batched inputs, including an empty batch.

Diff against the current target commit · GitHub checks

Earlier review and validation

Commit fe744372c addresses the architecture review: canonical *EnvProtocol names with exact *VecEnv compatibility aliases, OnlineModelBasedSchedule in learning/dynamics/schedule.py, early G1 recipe validation, and documented ODS/schedule/replay ownership. DifferentiableEnv and world-model family names retain their existing names.

  • 131 focused runtime tests pass, including APG/Actor-Critic, complete scheduled rollouts, checkpoint recovery, protocol aliases, and differentiable simulation contracts. The Newton planar checks reuse the two-line vector-gravity correction in b5961bdb9.
  • Ten recipe-assembly checks and seven CLI/checkpoint/evaluation-data checks pass. Unsupported recipe selections fail before environment allocation.
  • 53 API/context tests, API coverage (3628/3628), Black 26.10.1, and context checks pass. Sphinx 7.4.7 dummy build completes with 704 site-wide warnings. Existing alias coverage is extended; this follow-up adds no test files or cases.

Commit 58a02e62b preserves explicit options in a separate evaluation configuration. Collection settings fill omitted fields, except clip selection; omitted or null clips evaluates all clips in the selected archive. When both stages resolve to the same configuration file, evaluation retains workflow and CLI overrides. The README documents these precedence rules.

  • Seven CPU scenarios exercise the actual CLI, checkpoint serialization and evaluation archive loading before simulator construction. Five regression scenarios fail before the correction and pass afterward; same-file CLI overrides and path aliases retain their behavior.
  • A further 28 configuration-path, motion-data and model-environment tests pass. Black 26.10.1 and API coverage (3622/3622) pass. This correction changes two example files and adds no test files.

Commit 34a05bc42 normalizes quaternion endpoints before SLERP and aligns the Python 3.11 installation recipe with the declared Torch profile. Earlier commit 2e7c41ef6 corrects next-frame reference control, independent heading alignment, restricted external WM loading, and the G1 solver configuration override. The branch is based on main@c1f90ce7b.

  • A fresh Python 3.11 installation with Reduce RL rollout overhead and align batched physics updates #836 integrated passes dependency checks for all 276 packages. Torch 2.7.1+cu128, TorchVision 0.22.1+cu128 and TorchCodec 0.5 import successfully; 16 motion/model CPU tests and four installed CLI entries pass. The recipe selects CUDA wheels for Torch/TorchVision and the PyPI decoder for TorchCodec.
  • Scaled and sign-flipped quaternion inputs preserve the analytic rotation and angular velocity in float32 and float64. Both extended CSV cases fail with the original implementation and pass with the correction; the test count is unchanged.
  • At 2e7c41ef6, 164 focused CPU tests passed across kinematics, motion data, task calculations, APG, trainer configuration, scheduling, replay and dynamics models; six native/GPU cases were deselected.
  • Two repeated test setups are consolidated. Existing tests now cover the heading gradient and restricted WM checkpoint round-trip.
  • Black 26.10.1 and public API coverage (3622/3622) pass. Affected context was reviewed; the existing task and gradient ownership descriptions remain applicable.
  • Before branch separation, two G1 native state/reset configurations passed with these runtime corrections and the prerequisite branches installed.

The figures below use the previous heading calculation. The correction changes body-pose reward and evaluation metrics; quality measurements with the corrected calculation are pending.

Historical training and integration validation

G1/LAFAN1 training covers 40 motion clips with 4,096 parallel environments and 50 Hz control. Each seed completes 60,000 vector collection steps, 240,000 WM updates, 24,000 actor updates, and 480,000 critic updates.

  • Independent evaluation of three seeds on the 40 training clips completes all 120/120 full-length rollouts. Normalized returns are averaged equally over clips; the mean and sample standard deviation across seeds are 0.94149 ± 0.00231.
  • Candidate wheels install in an isolated Python 3.11 / Torch 2.10+cu130 environment with no dependency conflicts. A 128-environment run resumes to 220 collection steps with 226 / 88 / 1760 actual WM / actor / critic optimizer updates and no invalid transitions. A legacy checkpoint also resumes to 223 collection steps and continues updating.
  • The published model completes all 40 full-length clips through the installed candidate, with a macro normalized return of 0.94044. WM prediction checks pass at 1/8/16/32 steps. Data and model acquisition have passed anonymous, empty-cache validation.

Installed FastRT playback passes: 4 seconds at 1080p/50 fps, displaying the simulated robot, green reference robot, red/green tracking points, and clip timeline, with zero physics steps during playback.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Enhancement (non-breaking change which improves an existing functionality)

Screenshots

Historical training and evaluation figures

G1 training and full-clip evaluation across three seeds

Left: online WM training loss, showing raw values and an EMA (α=0.08) against vector collection iterations. Right: independent evaluation on the 40 training clips; each dot represents one clip and each diamond is the equally weighted clip mean. Each seed uses 4,096 environments and 60,000 collection iterations on an RTX PRO 5000 72 GB.

Fixed-WM multi-step prediction errors

The frozen published model predicts 1/8/16/32 steps from 128 complete windows of newly collected physical trajectories using measured actions. The figure compares the WM with holding the current state. Control runs at 50 Hz; model parameters and normalization statistics remain fixed. At 32 steps, mean root position error is 2.92 cm and joint position RMSE is 0.0291 rad.

Continuous multi-motion policy evaluation

g1-continuous-180s.mp4

Continuous evaluation preview

The seed-0 policy tracks 16 reference segments from 11 source clips, completing 15 transitions over 180 seconds with zero resets. Mean world-space root XY error is 9.36 cm. After root heading and horizontal-position alignment, the mean per-frame tracked-body position RMS error is 3.59 cm. The evaluation was recorded on 2026-10-31 and rendered from saved states on 2026-11-02 at 1920×1080 and 50 fps, with a 300×350-pixel World XY panel.

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
  • Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py), if applicable.
  • I verified appropriate behavioral coverage, reviewed the necessity of added or extended tests, and explained when no new tests were needed.
  • Dependencies have been updated, if applicable.

@acrlw acrlw added task Concrete tasks, deployments, objectives and recipes, including configured Task Program tasks. rl Reinforcement learning algorithms, collectors, trainers and training contracts. model Neural model architectures, inference, weights and model contracts. feature New user-facing capabilities. labels Nov 2, 2026

@han-xudong han-xudong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting together the motion-tracking workflow and the supporting simulation and visualization APIs. I reviewed the implementation and checked the declared DexSim 0.5.1 dependency at the artifact level.

There are correctness issues to address before merging, particularly heading/next-frame alignment, episode provenance, marker texture ownership and sensor isolation, external checkpoint loading, and native-test setup. The inline comments include concrete reproductions and suggested corrections. I am requesting changes for those issues; the two P3 notes are non-blocking.

The PR-linked 0.5.1 artifact corresponds to source e3ecde7f0; the current tested 0.5.1 CI artifact corresponds to a8cf41e2b0. They are different builds despite sharing the 0.5.1 release version. The package index also serves a 0.5.1 artifact from 28970b7d0 that lacks interfaces required by this PR; that is an artifact-selection mismatch, not a PR implementation finding. Please identify the supported artifact by build identity rather than the release version alone.

One observed failure is specifically caused by using the newer 0.5.1 build: the original contact reader works on the PR-linked e3ecde7f0, but a8cf41e2b0 changes fetch() semantics and removes fetch_async(). This is a compatibility/migration item if the newer artifact is supported, not a defect against the PR's linked SDK. In contrast, replicated-articulation removal fails on both compared builds and in an SDK-only scene; that binding issue needs an SDK fix and is not explained by switching between these builds. These dependency issues are separate from the 13 PR findings.

Validation totals below use PR head 80297a877ae0e4404f9e9f8e231d13e5b58426f4 with tested CI build a8cf41e2b0; they are not a claim that the entire suite ran on both SDK builds. The original PR and local candidate each pass 1,624 selected CPU tests. The focused 25 reproductions give 20 failures/5 passes on the original PR and 25 passes on the candidate. The candidate passes 31/33 additional native/GPU/slow cases, including the corrected G1 tests. These candidate edits are not in the reviewed PR head. The remaining failures are opaque-ghost sensor visibility and SDK articulation removal. Python/data/example checks use simulation-boundary doubles where noted. The ghost repair is not implemented or validated; the failure-path SIGSEGV remains unresolved, and full-scale/long-run training and multi-seed convergence were not qualified locally.

else {}
),
),
render_mode="scene" if scene_lighting else "overlay",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep opaque robot ghosts out of default sensor images

An opaque RobotMarkers instance with the default unlit=False takes this scene branch. It therefore enters the ordinary ray-traced scene, which the camera's visual-overlay opt-in does not control. With the tested 0.5.1 CI build a8cf41e2b0, the existing textured-ghost test changes the default camera from 0 to 3,990 red pixels before set_visual_overlay_enabled(True) is called. This violates the promised default sensor exclusion and can contaminate observations or recordings. Please resolve lighting and camera visibility as independent contracts, coordinating an SDK visibility channel/mask if needed. Forcing every marker into the existing overlay pass would remove the promised scene lighting and is not a complete fix. A concrete cross-SDK plan is to add a visualization visibility channel/mask independent of PBR routing, mark opaque ghosts with it, and have default cameras exclude it while opted-in cameras include it. Verify default exclusion, opted-in visibility, preserved lighting, and opacity transitions. This capability is not currently exposed; this is an unimplemented, unvalidated remediation plan, not a one-line fix. The pixel reproduction ran on the tested 0.5.1 a8 build. Source comparison shows the render-lowering function, native overlay policy, and shader are unchanged from the PR-linked 0.5.1 e3 build; the cited routing mechanism is not a change between those builds. A live pixel comparison on e3 was not performed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed; this remains open in #833, which owns RobotMarkers and native visualization. Opaque scene-lit ghosts currently enter the ordinary scene, while the camera opt-in controls the overlay pass. The correction will separate camera visibility from material lighting. Acceptance requires default-camera exclusion, opted-in visibility, preserved scene lighting and opacity transitions.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed that this remains unresolved. I am keeping this existing thread open on #924 while the implementation is tracked in #833.

The acceptance criteria remain: exclusion from default sensor images, visibility after explicit opt-in, preserved scene lighting, and correct behavior across opacity transitions. Please provide native image-based validation once that correction is included in the prerequisite and synchronized into the complete workflow.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sensor-isolation correction is implemented in #833, with native image validation repeated at fb23a4994 using the declared DexSim 0.6.0rc1 dependency.

Opaque scene-lit markers retain their material/scene-lighting path and receive a dedicated render category. Default mono and stereo cameras exclude that category; include_visual_overlays=True includes it. The camera mask also preserves marker exclusion when ground visibility changes.

The native texture/material/opacity image regression passes for Hybrid mono/stereo and FastRT mono. It checks default sensor exclusion, opted-in visibility, opaque override and asset-material restoration, zero/translucent opacity, and removal. The post-merge selection passed 3 Hybrid and 2 FastRT tests, including the ground-visibility interaction.

#833 and #836 are still open. #924 still needs synchronization after those prerequisites merge, followed by the required integrated state-injection/reset and saved-trajectory-replay checks. The implementation and prerequisite validation are complete; that final integration condition remains pending.

Comment thread examples/sim/reference_trajectory.py Resolved
@acrlw
acrlw force-pushed the codex/supermimic-maintainer-candidate branch from 773bfd0 to 2e7c41e Compare November 3, 2026 10:11
@acrlw

acrlw commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Twelve of the thirteen inline findings are implemented: four in this PR (2e7c41ef6), six in #833 (4fe72d87e) and two in #836 (9ed54ed42). Opaque scene-lit ghost visibility remains open in #833. The current #924 branch contains the WM/APG and G1 changes; its 164 focused CPU tests pass. The four local findings have original-versus-corrected reproductions, and historical quality figures are labeled with their original heading calculation.

The installation instructions explicitly select the supplied runtime wheel, and the maintainer validation record identifies the tested build. Contact-reader migration for a newer runtime remains pending in the shared simulation integration owned by #836. Replicated-articulation removal remains a runtime lifecycle issue requiring its own fix and validation.

Both #836 and #833 have been rebased onto main c1f90ce7b; #836 passes 1,142 focused CPU tests and #833 passes 448. Their combined changes also merge without conflicts. Both are prerequisites for the complete #924 workflow: #836 supplies physical collection interfaces, and #833 supplies saved-trajectory visualization. The planned merge order is #836, #833, then #924, all targeting main. #833 and #836 can be reviewed independently. After each prerequisite merges, the remaining branches will be synchronized and their integration checks rerun.

@han-xudong

Copy link
Copy Markdown
Collaborator

Thanks for addressing the feedback and adding focused regression coverage. I verified the corrections supporting the twelve resolved findings; those threads can remain resolved.

Please address the two new inline findings: quaternion normalization before SLERP and the installation recipe’s Torch-profile conflict. The existing ghost sensor-isolation thread remains open.

All eighteen marker tests pass with DexSim 0.5.1 build a8cf41e2b0. Their CI setup errors came from using 0.5.1rc1. Please identify the supported SDK artifact and align CI accordingly. If a8 is selected, the contact reader needs the corresponding API adaptation. Articulation removal remains a separate SDK issue.

I am keeping “changes requested” pending these fixes and validation of the synchronized prerequisites. Please retain the historical-quality labels until corrected-heading evaluation is available.

@acrlw

acrlw commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Regarding the follow-up review:

The two new #924 findings are corrected. The existing ghost sensor-isolation finding remains with #833: default camera exclusion, explicit opt-in, scene lighting and opacity transitions still require native image validation. The twelve previously resolved findings retain their status.

The runtime validation record identifies the runtime wheel and its source revision. CI must use that same artifact when the #836 prerequisite is integrated; the existing 0.5.1rc1 setup does not exercise that runtime. Runtime selection and contact API migration remain with #836. The corrected-heading quality evaluation is pending, and the historical figures retain their original labels.

@han-xudong han-xudong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the code changes in #924 at 34a05bc. The two latest findings are fixed and verified, with no remaining blocking code findings in the reviewed scope. Thanks for addressing the feedback.

Please merge only after #836 and #833 are completed and integrated, and integration CI passes. The existing ghost sensor-isolation thread remains open for dependency tracking.

@acrlw
acrlw marked this pull request as ready for review November 4, 2026 06:32
@greptile-apps

greptile-apps Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The PR adds behavior across learning, task planning, simulation, and data-collection workflows.

The PR still needs the previously reported selected-row planning fix before merging.

Fix All in CodexFindings

  1. P1 Selected-row motion fails ▶
Fix with agent prompt
### Issue 1
embodichain/lab/sim/motion/motion_generator.py:815-821
`fixed_grid` planning ignores the generator's selected rows when reading speed limits. For a robot with four environments and `env_ids=(1,)`, the path has one row but `authored_joint_velocity_limits()` returns four. `joint_velocity_validity()` then raises instead of returning a plan. Selecting all rows in a different order can instead check the wrong limits.

Apply the same row selection to these limits. Also pass it to the new sequential `compute_ik()` and Cartesian-start `compute_fk()` calls, which currently default to all environments.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR adds online world-model learning and APG motion tracking for G1.

  • Measured robot motion now trains a world model that helps a policy learn.
  • G1 motion tracking now uses shared physical and imagined task calculations.
  • Task Engine turns an image and instruction into a checked task plan.
  • GenSim workflows can select model services through explicit profiles.
  • Geometry tools now measure contact support and propose object placements.
  • Real2Sim demonstrations can now become task-specific executable skills.
  • Expansion Engine collects and records batches of task episodes.
  • Scene and simulation workflows gain reusable state, contact, and geometry tools.

The earlier evaluation-settings issue is fixed: separate evaluation files retain explicit options, while collection settings supply omitted defaults.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Physical[Physical collection] --> Replay[Measured replay]
  Replay --> Model[Supervised world-model updates]
  Replay --> Starts[Measured rollout starts]
  Model --> Rollout[Frozen-model rollouts]
  Starts --> Rollout
  Rollout --> APG[Actor and critic updates]
  APG --> Physical
  APG --> Evaluation[Separate physical evaluation]
Loading

Reviews (6) · Last reviewed commit: "fix(tracking): preserve fullgraph compil..." · Reviewed by Greptile

@yuecideng yuecideng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architecture review: naming, extension boundaries, and ODS ownership

Architecture follow-up for PR #924 at 34a05bc42aef31ded7efdb71360e1b26e15db3b9.

The overall layering is sound: differentiable environment contracts, RL updates, supervised model fitting, and online update scheduling have distinct responsibilities. Please make the scoped adjustments below in this PR. Broader abstractions should follow a second concrete use case. This comment records architectural requests, not new correctness findings or a revalidation of previously reported defects.

Requested in this PR

1. Use environment names that treat batching as the default contract.

EmbodiedEnv already supports concurrent environments. Keep batching explicit in the documented shapes and num_envs contract, without encoding it again as Vec in these protocol names. Use Protocol to distinguish structural interfaces from implementation base classes:

Current name Requested name
LearningVecEnv LearningEnvProtocol
DifferentiableVecEnv DifferentiableEnvProtocol
ScheduledDifferentiableVecEnv ScheduledDifferentiableEnvProtocol
EmbodiedEnv Keep unchanged
Simulation base class DifferentiableEnv Keep unchanged

Keep DifferentiableEnv as the unified, backend-neutral implementation entry point. Do not rename it to NewtonKinematicEnv. Backend-specific behavior belongs behind configuration/adapters/internal implementations. Document and validate the capabilities actually supported today; a unified name does not imply that every solver already supports differentiable dynamics.

Preserve compatibility aliases for existing public *VecEnv imports, and update internal references, public exports, type annotations, API documentation, examples, and project context together. Verify that existing downstream imports remain usable, including NMG's shared rollout interfaces.

2. Make the new scheduler's name and location match its scope.

  • Move embodichain/learning/dynamics/online.py to embodichain/learning/dynamics/schedule.py.
  • Rename OnlineLearningSchedule to OnlineModelBasedSchedule.
  • Rename OnlineLearningCfg to OnlineModelBasedScheduleCfg.
  • Keep the implementation under learning/dynamics/; do not promote it to learning/online.py or introduce a general workflow package in this PR.

The component schedules physical collection, model-only warmup, model updates, and policy updates. It owns update credit and progress counters, while callbacks own data, optimizers, and completed backward work. Update imports, exports, tests, and documentation consistently.

3. State the supported extension boundary and validate recipe assumptions.

Keep MotionTraining explicitly documented as the G1/SuperMimic composition entry point. The reusable pieces are the environment protocols/registries, DifferentiableTrainer.update_once(), SupervisedLearner, and the callback-based schedule.

The current entry constructs KinematicWorldModel, selects a particular objective/replay schema, derives task-specific policy dimensions, and expects an actor-critic algorithm with a separate critic optimizer. Document these requirements and reject unsupported selections with a clear error at the assembly boundary. The presence of trainer / policy / algorithm blocks should not imply that changing a name in YAML is sufficient to run arbitrary algorithms or model families through this example.

4. Document the ODS boundary without merging storage implementations.

Component Ownership
ODS OnlineDataEngine and data sources Data production, shared storage, sample selection, validity, and producer lifecycle
ODS ODSFlowController Refill, bounded waits, delivery limits, and data-supply pacing
OnlineModelBasedSchedule Ordering and ratio of collection/model/policy updates, warmup, and learning progress
TransitionReplay Device-resident measured transitions, episode-local windows/history, and imagination starts

These are complementary responsibilities. Keep the device-resident replay for the current synchronous training path. Specify ownership, episode boundaries, history padding, action semantics, control period, and validity masks. If ODS is later connected, its supply pacing must not independently duplicate the learning workflow's collection trigger or update accounting. ODS should not own actor/critic optimization or autograd lifecycle.

Follow-up work, not required in this PR

  • Model/provider factories: When a second model is integrated, extract construction of the model, objective, replay adapter, and imagination-start provider from MotionTraining. Avoid adding task/model conditionals throughout the existing example.
  • Algorithm-owned optimization: When supporting multiple algorithms in the online workflow, remove its fixed assumption of three optimizers. Put optimizer/scheduler ownership in the corresponding learner or an explicit component interface.
  • Stateful and alternative models: Define hidden-state initialization, history burn-in, reset/detach, sampling, and checkpoint semantics when implementing an actual RNN, latent-state, or ensemble provider. These are not currently plug-and-play YAML substitutions.
  • ODS integration: Add an explicit adapter through the existing BatchSource / SequenceBatchSource / SupervisedBatch contracts for an ODS-fed model-learning use case. Align sequence semantics before deciding what sampling code to share; do not force GPU replay into CPU shared-memory/locking machinery.
  • General workflow infrastructure: Consider learning/workflows/ only after multiple use cases demonstrate shared composition, evaluation, checkpoint, and recovery needs.
  • Unified environment capabilities: Extend backend/provider support behind DifferentiableEnv incrementally, with capability and gradient tests. A universal backend implementation is not a prerequisite for this PR.
  • NMG migration: Any migration of NMG's own training loop to the shared trainer should be separate and preserve its complete K-dependent rollouts, objective scaling, effective-interaction counting, semantic normalization, and recovery behavior. The previously identified Franka preset/TCP upgrade issue is separate from this PR's APG architecture changes.

Scope and validation

World-model naming is explicitly unchanged: keep KinematicWorldModel and the existing model-family names in this PR.

For this cleanup, use focused import/alias, configuration, scheduler, and existing APG/trainer regression tests, plus affected API/context checks. The requested naming and boundary clarification does not itself require a new full-scale convergence run. This comment does not claim new runtime validation of the current head or supersede separate correctness and acceptance reviews.

@acrlw

acrlw commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@yuecideng The four changes requested in your architecture review are implemented in fe744372c.

1. Environment protocols and compatibility

The canonical names are now LearningEnvProtocol, DifferentiableEnvProtocol, and ScheduledDifferentiableEnvProtocol. The former *VecEnv names remain exact aliases from both embodichain.learning.rl and embodichain.learning.rl.env. Internal annotations, exports, examples, API documentation, and project context use the new names. The protocol documentation specifies num_envs, leading batch dimensions, single-row spaces, and per-environment reward and termination shapes.

EmbodiedEnv and DifferentiableEnv retain their names. The latter documents its current Newton kinematics capability and validates NewtonPhysicsCfg(requires_grad=True) before constructing the simulation. Existing tests cover that capability boundary and the Warp/PyTorch gradient bridge.

The protocol test checks alias identity at both public import paths. Shared rollout and trainer regressions pass for complete scheduled horizons, objective scaling, gradient accumulation, and checkpoint recovery.

2. Model-based schedule

The implementation is now embodichain.learning.dynamics.schedule, exporting OnlineModelBasedSchedule and OnlineModelBasedScheduleCfg. Imports, exports, documentation, and the existing schedule test have been updated.

The schedule owns collection/model/policy ordering, model-only warmup, fractional update credit, and progress counters. Callbacks own their tensors, optimizers, and completed backward passes. Checkpoint fields and update calculations are unchanged; the existing prefill, warmup, callback-ordering, and resumed fractional-credit test passes.

3. G1 composition and recipe validation

MotionTraining is explicitly documented as the G1/SuperMimic composition entry. Before constructing the physical environment, it validates MotionTrackingModelRL, the motion_tracking policy, apg_actor_critic, and the G1 collection/evaluation task IDs. Unsupported selections produce an error naming the configuration field, supported value, and supplied value.

The README documents the fixed KinematicWorldModel, fitting objective, replay fields, task-derived policy dimensions, and separate WM/actor/critic optimizers.

4. ODS and replay responsibilities

The README now defines ownership for OnlineDataEngine and data sources, ODSFlowController, OnlineModelBasedSchedule, and TransitionReplay. It also specifies detached storage ownership, episode boundaries, history padding and completeness, normalized setpoint versus policy actions, the 0.02 s control period, and validity masks.

The synchronous path uses device-resident replay. Future ODS integration will adapt BatchSource / SequenceBatchSource to SupervisedBatch. The schedule will own collection triggers and update accounting; learners and trainer callbacks will own optimizer steps and autograd lifecycle. Model/provider factories and alternative-model state handling will be designed around the second model integration.

Validation

  • 131 focused runtime tests pass, covering protocols, APG/Actor-Critic, collectors, scheduled rollout/recovery, runtime assembly, point mass, Newton planar reach, and differentiable simulation contracts.
  • 10 recipe-assembly checks pass: four supplied recipes reach assembly, and six unsupported selections or forms fail before environment allocation. Seven existing CLI/checkpoint/evaluation-data checks also pass.
  • 53 API-documentation and project-context tests pass; API coverage is 3628/3628. Black 26.10.1 and context checks pass.
  • Validation reuses the existing test suite and extends the protocol test with alias-identity assertions.

The Newton planar-reach checks exposed a scalar-gravity incompatibility with the installed Newton API. Commit b5961bdb9 reuses the existing two-line vector-gravity correction; all 11 affected checks pass afterward.

Retain upstream trajectory exports and quaternion regressions alongside floating-root kinematics and motion tracking.
Comment on lines +815 to +821
limits = authored_joint_velocity_limits(
self.robot, options.control_part
).to(raw_result.positions)
success = torch.as_tensor(
raw_result.success, device=raw_result.positions.device, dtype=torch.bool
)
valid = joint_velocity_validity(raw_result.positions, raw_result.dt, limits)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Selected-row motion fails

fixed_grid planning ignores the generator's selected rows when reading speed limits. For a robot with four environments and env_ids=(1,), the path has one row but authored_joint_velocity_limits() returns four. joint_velocity_validity() then raises instead of returning a plan. Selecting all rows in a different order can instead check the wrong limits.

Apply the same row selection to these limits. Also pass it to the new sequential compute_ik() and Cartesian-start compute_fk() calls, which currently default to all environments.

Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/lab/sim/motion/motion_generator.py
Line: 815-821

Comment:
**Selected-row motion fails**

`fixed_grid` planning ignores the generator's selected rows when reading speed limits. For a robot with four environments and `env_ids=(1,)`, the path has one row but `authored_joint_velocity_limits()` returns four. `joint_velocity_validity()` then raises instead of returning a plan. Selecting all rows in a different order can instead check the wrong limits.

Apply the same row selection to these limits. Also pass it to the new sequential `compute_ik()` and Cartesian-start `compute_fk()` calls, which currently default to all environments.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9bbf45201. Fixed-grid velocity checks now select limits in env_ids order, and sequential IK and Cartesian-start FK receive the same environment selection. Continuous IK recovery also selects the corresponding joint bounds and velocity limits.

The regression cases cover a single selected row in a four-environment robot, all rows reordered as (3, 1, 0, 2), distinct per-row speed limits and base poses, and selected-row continuous recovery. These cases fail with the original implementation and pass after the correction. The targeted motion/configuration/APG suite passes all 189 CPU tests.

@yuecideng yuecideng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review at 9bbf452 (base main@460939c83b6fc62f113b62af4798a89c42b3bec2).

The architecture cleanup is in place: canonical *EnvProtocol names retain exact compatibility aliases, DifferentiableEnv keeps its unified name, OnlineModelBasedSchedule lives in learning/dynamics/schedule.py, recipe assumptions fail early, and ODS/replay/learning ownership is documented. The selected-row planning fixes also look consistent on static review.

Three P2 issues remain in the inline comments: zero-weight critic minibatches still execute Adam updates; SDK installation instructions conflict with the pinned dependency; and resume compatibility does not identify changed motion contents.

Conditions for an initial infrastructure/experimental-workflow merge:

  1. Address these three P2 issues and pass focused regressions for the critic boundary and motion-content compatibility, plus verification of the corrected installation instructions.
  2. Merge prerequisites #836 and #833 into main, synchronize this branch, and pass required integration CI on the resulting commit using the declared DexSim 0.6.0rc1 dependency. Confirm physical state injection/reset and saved-trajectory replay against the integrated code; a conflict-free merge alone is insufficient.
  3. Preserve the distinction between initial infrastructure delivery and formal learning acceptance. Corrected-heading quality evaluation is pending; historical 0.94/full-clip figures must remain labeled historical and cannot establish current-version acceptance. Track corrected evaluation and the agreed training-budget evidence as follow-up acceptance work if the initial merge is scoped accordingly.

Model/provider factories, stateful-model interfaces, a general workflow package, and NMG trainer migration can remain follow-up work.

Validation from this review: 57 learning/model tests, 96 numerical tests, and 8 configuration/protocol/schedule tests passed; API coverage was 4489/4489 and the project-context check passed. The critic behavior and motion-identity gap were reproduced on CPU. Some native-dependent tests could not collect with the reviewer's locally installed older SDK, so this review does not claim current-head native/GPU integration or full-training qualification.

This COMMENT review records the outstanding work and merge conditions.

Comment on lines +411 to +412
if bool(torch.isfinite(loss) & torch.isfinite(norm)):
self.critic_optimizer.step()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Skip critic minibatches with zero total valid weight

Complete rollouts can contain a zero-weight suffix after early termination or invalid task data. Those rows are still shuffled into minibatches. A batch containing only zero weights produces zero loss and gradients, passes this finite check, and calls Adam.step(); existing momentum then changes the critic despite there being no valid samples.

Reproduced at this head with the existing LinearEnv, termination at step one, horizon three, and critic_minibatches=3: weights are [1, 0, 0]. With the valid row first, the two zero-gradient batches change the parameter by approximately -0.000670058 and -0.000517957, and critic_updates reports 3. Existing termination tests use one critic minibatch and miss this case. Please filter positive-weight rows before partitioning or skip batches whose total weight is zero, with a focused early-termination/multiple-minibatch regression.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in a0058614f.

Fixed critic batching to select positive-weight samples before shuffling and partitioning. An empty valid set performs no critic optimizer step, so existing Adam momentum cannot change the parameters. Positive soft-survival weights retain their values, and critic_updates counts actual steps. Regressions compare the critic parameters and Adam state for a single valid transition versus the same transition followed by terminated/invalid suffixes, and verify that an all-invalid update preserves an already-initialized Adam state.

The Python 3.11 dependency profile uses PyTorch 2.7, TorchVision 0.22 and
TorchCodec 0.5. The first install command selects CUDA 12.8 wheels for Torch and
TorchVision; the second installs TorchCodec's CPU decoder from PyPI. The SDK
requirement from #836 is `dexsim_engine>=0.5.1,<0.6`; use the review build selected above.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Align SDK installation instructions with the pinned dependency

This paragraph directs readers to select a review wheel in the range dexsim_engine>=0.5.1,<0.6, but the current pyproject.toml pins dexsim_engine==0.6.0rc1. The pinned release candidate is excluded by the documented range. A user following the stated range and supplying a 0.5.x wheel to the explicit wheel-install command above encounters a dependency conflict.

Please update the requirement and wheel-selection instructions to the published 0.6.0rc1 artifact used by current validation, and check the documented installation against that profile. This is separate from the earlier Torch-profile issue.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in a0058614f.

Updated the install recipe to use the published dexsim_engine==0.6.0rc1 wheel from the SDK index, matching pyproject.toml, and removed the review-wheel and <0.6 instructions. I ran the revised commands in a new Python 3.11.15 environment; they resolved Torch 2.7.1+cu128, TorchVision 0.22.1+cu128 and TorchCodec 0.5. uv pip check, runtime imports and the installed embodichain run-example learning/motion_tracking/train.py --help entry all passed.

Comment on lines +320 to +324
"motion": {
"names": self.env.motion.names,
"lengths": self.env.motion.lengths.cpu().tolist(),
"fps": self.env.motion.fps,
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Include motion contents in checkpoint compatibility identity

If motion.npz is regenerated or replaced at the same motion_path with identical clip names, lengths, FPS, and joint order but different qpos/qvel values, this identity remains unchanged. restore() then accepts the old policy/critic, normalization statistics, and adaptive sampling history for a different reference dataset. MotionLibrary and sampler validation do not add a content check downstream.

A CPU probe at this head using otherwise identical libraries with joint trajectories differing by 0.5 rad produces equal training motion identities, and the sampler accepts the old state. Please persist a stable digest of the selected motion contents and compare it before restoring any learning state. A same-layout/different-content rejection case can cover this without simulation. URDF compatibility already has a separate model extra-state check.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in a0058614f.

Added a versioned content identity to MotionLibrary, computed once from the selected float32 motion data, clip layout, joint order and sampling frequency before device upload. Both the training checkpoint and sampler state retain it, and restore checks it before loading learning state. Regressions accept unchanged data, reject qpos/qvel changes at the same path with unchanged clip metadata, ignore changes to unselected clips, and verify rejection leaves state untouched. Checkpoints without the content identity are rejected explicitly; this compatibility change is documented.

@yuecideng
yuecideng merged commit 02d0fda into main Nov 11, 2026
8 of 12 checks passed
@yuecideng
yuecideng deleted the codex/supermimic-maintainer-candidate branch November 11, 2026 11:46
yuecideng added a commit that referenced this pull request Nov 11, 2026
Merge main through #924 while preserving named and direct provider configuration and capability-aware articulation reset. Validate NeRD with the shared collector/trainer, including pre-reset terminal gradients and a real APG Actor-Critic update. Repair existing PR CI failures in test fixtures, visualization declarations and runtime-version comparison.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New user-facing capabilities. model Neural model architectures, inference, weights and model contracts. rl Reinforcement learning algorithms, collectors, trainers and training contracts. task Concrete tasks, deployments, objectives and recipes, including configured Task Program tasks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants