Skip to content

Reduce RL rollout overhead and align batched physics updates - #836

Merged
yuecideng merged 19 commits into
mainfrom
codex/rl-rollout-performance
Nov 11, 2026
Merged

yuecideng merged 19 commits into
mainfrom
codex/rl-rollout-performance

Conversation

@acrlw

@acrlw acrlw commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Reduce allocation and property-update overhead in batched RL environments while preserving episode provenance, selected environment rows, and physical joint limits. This PR targets the DexSim 0.6.0rc1 baseline on main and provides shared simulation APIs for #837 and the G1 environment in #924.

Changes

  • Allocate action histories and metadata rows on demand. Preserve reset-time seeds for dataset, rollout, and trajectory-only recording. Existing configurations retain episode context in info; training can explicitly disable those snapshots.
  • Reuse device-side joint selectors and batch motor, mass, inertia, and COM updates. Reduce packed contact data using the current solver-report validity and substep timing.
  • Allow raw PD targets with clamp_target_qpos=False while retaining physical joint limits, including named robot control groups and Newton mimic followers.
  • Add static PlaneCfg, explicit joint-limit gains, solver selection, and the cubql mesh BVH option. Ground collision and geometry changes take effect before preparation on both backends.
  • Read and update static-surface friction on Default and Newton, preserve unselected rows, and retain changes across topology rebuilds.
  • Validate asset-cache cleanup against the actual download and extraction roots. Preserve the existing DEFAULT_DATA_MODULES package export.

Validation

Python 3.12.13, Torch 2.11.0+cu128, the published DexSim 0.6.0rc1 wheel, Newton 1.7.0, and RTX 5090 D v2:

  • 923 focused CPU tests passed before the final corrections; 254 affected recording, cache, adapter, episode-context, and planar-reach regressions passed after them.
  • Regression cases reproduced four failures before the provenance/export fixes and all seven selected cases passed afterward.
  • Nine native tests passed against the installed package: static friction reads/writes and rebuilds, disabled ground collision and replacement geometry on Default/Newton, per-arena contact forces, inertia updates, and selected resets.
  • Built and installed the package into an isolated target; the installed CLI and bundled task/robot imports passed.
  • Black 26.10.1, public API documentation coverage (4441/4441), and the agent-context check passed. Existing context still describes the intended APIs and lifecycle; these corrections restore that behavior.

The preceding CI test job stopped before pytest because the test-plan artifact download returned ECONNRESET. Current results are available under GitHub checks. Native validation covers short simulation and reset sequences; long training performance below retains its original runtime conditions.

Dependencies

Targets main. #837 uses this branch as its base; #924 uses its G1 simulation APIs.

Type of change

  • Bug fix
  • Performance improvement

Checklist

  • Ran black . with Black 26.10.1.
  • Reviewed affected documentation and agent context.
  • Public API documentation coverage passed.
  • Verified focused behavioral coverage, including native backend regressions.
  • Built and installed the package using the supported runtime dependencies.
  • Current remote CI and maintainer review.
Historical G1 rough-terrain benchmark and runtime versions

Integration benchmarks use the G1 Rough environment from #837, one RTX PRO 5000 Blackwell 72 GB GPU, driver 580.126.20, and 4096 environments. Each run performs 100 PPO updates; timing excludes compilation, initialization, and the first 10 updates. Results are the mean of two runs per path.

Rollout SPS = transitions / collection time. Training SPS = transitions / (collection + PPO update time). Memory values are process-tree peaks across initialization and training.

Path Rollout SPS Training SPS Rollout relative to Isaac Peak RAM (MiB) Peak GPU memory (MiB)
Isaac Lab 153978 138982 — 3848 4470
EmbodiChain / MJWarp 152490 138074 -0.97% 5602 4378
EmbodiChain / DexUni 145738 132576 -5.35% 5811 4634

DexUni Training SPS is 4.61% below Isaac. All three paths use the 37-joint G1, 187 height rays, world dt=0.005 s, 2 substeps, solver dt=0.0025 s, and control dt=0.02 s. Newton contacts update every solver substep. Shared settings are implicitfast / pyramidal, 100 solver iterations, 50 line-search iterations, and 24 steps/update.

Component Benchmark version
Python / Torch / torchvision 3.11.15 / 2.10.0+cu128 / 0.25.0
Newton 1.8.0.dev0, 66bf4840
MuJoCo / MJWarp / Warp 3.15.0 / 3.15.0 / 1.17.0
RSL-RL / TensorDict 5.4.1 / 0.14.2
NumPy / SciPy / Gymnasium / USD 2.4.6 / 1.17.1 / 1.3.0 / 26.8
Isaac Lab 3.0.0-EA, ae37b028; frozen terrain adapter and shared training entry point
EmbodiChain Common fixes 4e08405c + tasks 9f32fa08

Initial policies and PPO configurations are identical. Randomized mass and inertia samples differ across adapters because they consume random numbers in different orders.

@acrlw acrlw added bug Incorrect existing behavior or a fix restoring the intended behavior. enhancement Improvements to existing capabilities; use feature for new capabilities. physics Physics backends, dynamics, contacts, fluids, deformables and physics solvers. rl Reinforcement learning algorithms, collectors, trainers and training contracts. labels Oct 22, 2026
@acrlw
acrlw marked this pull request as ready for review October 22, 2026 07:42
@greptile-apps

greptile-apps Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes reach RL recording and shared simulation paths used by batched environments.

The previous early-pass bug in PositionRangeMonitor.observe() remains a merge blocker.

Fix All in CodexFindings

  1. P1 Joint hold passes early ▶
Fix with agent prompt
### Issue 1
embodichain/compute/trajectory/stability.py:113-117
`PositionRangeMonitor.observe()` counts the first good sample after a failed target check as a completed qualifying interval. If the joint enters tolerance by moving less than `position_drift`, `_counts` increases to 1 even though the previous sample was outside the target. The joint-stability policy can then pass one control step before the full `cfg.duration` has elapsed inside the target.

Track whether the previous sample qualified, and start a new window at zero when the joint re-enters the target.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR reduces rollout allocations and adds batched physics controls.

  • Recorded episodes keep their context without allocating unused rows.
  • Training can leave episode context out of step and reset info.
  • Action snapshots reuse joint selectors across rollout steps.
  • Joint writes can keep raw PD targets while limits still bound positions.
  • Selected physics writes update links in batches and keep friction live.
  • Physics configs now expose reference settings for joints and surfaces.
  • Newton scenes can register batched DC joint motors.
  • Contact sensors can sum normal impulses without expanding contact rows.
  • Asset-cache cleanup checks the actual download and extraction roots.

Reviews (10) · Last reviewed commit: "fix(rl): preserve recording metadata and..." · Reviewed by Greptile

@yuecideng yuecideng added performance Performance, memory, throughput and synchronization-cost improvements. assets Robot and scene assets, meshes, materials, asset catalogs and downloads. dependencies Dependency versions, upgrades and compatibility. dexsim DexSim SDK interfaces, version adaptation and integration. docs Documentation, API references and example explanations. gym Environment lifecycle, managers, configuration composition and registration. sensor Sensor models, attachment, lifecycle, extension interfaces and observation contracts. labels Oct 25, 2026
@acrlw
acrlw requested a review from yuecideng October 26, 2026 01:16
@acrlw acrlw changed the title Reduce RL rollout overhead and fix gravity and asset-cache handling Reduce RL rollout overhead and align batched physics updates Nov 2, 2026
@acrlw
acrlw force-pushed the codex/rl-rollout-performance branch from b9448d4 to 944715e Compare November 2, 2026 02:52
@acrlw
acrlw marked this pull request as draft November 2, 2026 03:01
@acrlw
acrlw marked this pull request as ready for review November 2, 2026 05:21
@acrlw
acrlw force-pushed the codex/rl-rollout-performance branch from 3649faf to 9ed54ed Compare November 3, 2026 10:50
Keep solver-report contact sampling and carry the batched controls, selected resets, and mesh query settings onto the current main branch.
Comment on lines +113 to +117
reset = ~good | ~torch.isfinite(self._low) | ((high - low) > self._drift)
self._low = torch.where(reset, values, low)
self._high = torch.where(reset, values, high)
self._counts = torch.where(reset, 0, self._counts + int(elapsed_steps > 0))
self._satisfied = good & (self._counts >= self._required)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Joint hold passes early

PositionRangeMonitor.observe() counts the first good sample after a failed target check as a completed qualifying interval. If the joint enters tolerance by moving less than position_drift, _counts increases to 1 even though the previous sample was outside the target. The joint-stability policy can then pass one control step before the full cfg.duration has elapsed inside the target.

Track whether the previous sample qualified, and start a new window at zero when the joint re-enters the target.

Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/compute/trajectory/stability.py
Line: 113-117

Comment:
**Joint hold passes early**

`PositionRangeMonitor.observe()` counts the first good sample after a failed target check as a completed qualifying interval. If the joint enters tolerance by moving less than `position_drift`, `_counts` increases to 1 even though the previous sample was outside the target. The joint-stability policy can then pass one control step before the full `cfg.duration` has elapsed inside the target.

Track whether the previous sample qualified, and start a new window at zero when the joint re-enters the target.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 21afa5e3b. PositionRangeMonitor now retains each row's previous qualifying state. The first sample after re-entering the target starts a new window at zero completed intervals and resets its range bounds. Initial pre-sample anchors and duplicate-step behavior are preserved.

The new batched regression covers entry, exit and re-entry independently per row. The existing policy-duration test now also covers entering tolerance with movement below position_drift: entering at step 10 with a 25-interval hold completes at step 35, instead of the previous step 34. Both regressions fail on the previous implementation. All 50 position-window and simulation-policy tests pass, including recorded-window replay and initial-anchor checks. Black and the 4441-export API documentation check also pass.

@yuecideng yuecideng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 75e4466e6a4f56999ff2e842d6ee2147942aa46b.

Two P2 findings are attached inline: preserve reset-time provenance for lazily materialized metadata rows, and implement selective episode reset for the registered DC motor.

Actuator migration will be handled in a separate follow-up PR. Keep the current fused DC-motor kernel and GraphRuntimeControl implementation in this PR; adopting Newton's Actuator is not a prerequisite for this change. The reset fix can stay within the current design.

Validation used focused CPU checks and targeted reproductions. Native World/GPU execution, CUDA Graph capture, and sustained training performance were not rerun during this review.

Comment on lines +1563 to +1567
if not getattr(self, "_demo_metadata_initialized", True) and (
self.dataset_manager is not None
or getattr(self, "rollout_buffer", None) is not None
or getattr(self, "_traj_buffer", None) is not None
):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve reset-time provenance for unmaterialized metadata rows

When no dataset manager, rollout buffer, or trajectory buffer is attached, this guard skips capturing reset-time metadata. For two environment rows, run reset(seed=12), then reset(seed=35, options={"reset_ids": [0]}), and finally get_demo_episode_metadata(1). Row 1 has not reset, but the lazy fallback reads the current cfg.seed and reports 35 instead of 12. A CPU comparison using the production reset/getter lifecycle reproduced base 12 versus head 35 for the untouched row.

The existing seed-preservation cases attach rollout or trajectory storage, so they exercise the protected _EpisodeMetadataRows path. Please retain lightweight reset-time seed/provenance per row independently of recorder attachment, while keeping full metadata materialization lazy, and cover an untouched row after a partial reseed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 2c2baa4ba.

Fixed by retaining each row's reset-time seed and shared provenance snapshot independently of recorder attachment. Full episode metadata and row UUIDs remain lazy. The regression resets both rows with seed 12, then resets only row 0 with seed 35; row 1 retains seed 12 and its original configuration hash when first read. Both getter orders, recording enabled/disabled, repeated partial reset and returned-data ownership are covered.

previous_velocity[row, column] = velocity


class _BatchedJointDCMotorControl(GraphRuntimeControl):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Implement selective episode reset for the registered DC motor

This subclass inherits RuntimeControl.prepare_episode_reset(), whose default implementation raises NotImplementedError. In a finalized Newton/DexUni scene with this motor installed, the existing public sim.spawn_result.reset_arenas([0]) path preflights every registered control and fails with _BatchedJointDCMotorControl does not support episode reset. A CPU probe through the actual SDK reset binding reproduced the failure.

Please implement the hook so it returns a commit callback that restores the selected arenas' motor targets and clears their effort/velocity buffers, preserving unselected rows and captured buffer addresses. Add focused reset coverage. This issue is limited to Scene.reset_arenas(); ordinary manager resets use batch dynamics clearing, and #837 explicitly restores the aliased motor buffers, so those paths are not claimed to raise this exception. The current fused motor implementation can be retained.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 2c2baa4ba.

Added prepare_episode_reset() using the resolved articulation handles to map Scene arena indices to motor rows. Preflight does not mutate state; its callback restores selected targets and clears their effort/velocity history in place, preserving other rows and buffer addresses. CPU coverage includes reordered arenas, empty selections and repeated resets. A two-arena DexUni test also passes through Scene.reset_arenas() and subsequent CUDA Graph stepping, with the graph remaining captured. The fused motor/GraphRuntimeControl implementation is retained; Actuator migration remains separate in #1131.

@yuecideng

Copy link
Copy Markdown
Contributor

Once the two P2 findings are addressed, this PR can be merged:

  1. Preserve reset-time seeds for unmaterialized metadata rows after partial reseeding.
  2. Implement the motor's selective episode-reset protocol for Scene.reset_arenas().

Keep the current motor implementation in this PR. The reusable Actuator integration will be handled separately in #1131 and does not block this PR.

@yuecideng
yuecideng merged commit 69c28a9 into main Nov 11, 2026
10 of 12 checks passed
@yuecideng
yuecideng deleted the codex/rl-rollout-performance branch November 11, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

assets Robot and scene assets, meshes, materials, asset catalogs and downloads. bug Incorrect existing behavior or a fix restoring the intended behavior. dependencies Dependency versions, upgrades and compatibility. dexsim DexSim SDK interfaces, version adaptation and integration. docs Documentation, API references and example explanations. enhancement Improvements to existing capabilities; use feature for new capabilities. gym Environment lifecycle, managers, configuration composition and registration. performance Performance, memory, throughput and synchronization-cost improvements. physics Physics backends, dynamics, contacts, fluids, deformables and physics solvers. rl Reinforcement learning algorithms, collectors, trainers and training contracts. sensor Sensor models, attachment, lifecycle, extension interfaces and observation contracts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants