Repository navigation
Reduce RL rollout overhead and align batched physics updates - #836
Conversation
[Medium impact] The changes reach RL recording and shared simulation paths used by batched environments. The previous early-pass bug in
|
b9448d4 to
944715e
Compare
3649faf to
9ed54ed
Compare
Keep solver-report contact sampling and carry the batched controls, selected resets, and mesh query settings onto the current main branch.
| reset = ~good | ~torch.isfinite(self._low) | ((high - low) > self._drift) | ||
| self._low = torch.where(reset, values, low) | ||
| self._high = torch.where(reset, values, high) | ||
| self._counts = torch.where(reset, 0, self._counts + int(elapsed_steps > 0)) | ||
| self._satisfied = good & (self._counts >= self._required) |
There was a problem hiding this comment.
PositionRangeMonitor.observe() counts the first good sample after a failed target check as a completed qualifying interval. If the joint enters tolerance by moving less than position_drift, _counts increases to 1 even though the previous sample was outside the target. The joint-stability policy can then pass one control step before the full cfg.duration has elapsed inside the target.
Track whether the previous sample qualified, and start a new window at zero when the joint re-enters the target.
Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/compute/trajectory/stability.py
Line: 113-117
Comment:
**Joint hold passes early**
`PositionRangeMonitor.observe()` counts the first good sample after a failed target check as a completed qualifying interval. If the joint enters tolerance by moving less than `position_drift`, `_counts` increases to 1 even though the previous sample was outside the target. The joint-stability policy can then pass one control step before the full `cfg.duration` has elapsed inside the target.
Track whether the previous sample qualified, and start a new window at zero when the joint re-enters the target.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 21afa5e3b. PositionRangeMonitor now retains each row's previous qualifying state. The first sample after re-entering the target starts a new window at zero completed intervals and resets its range bounds. Initial pre-sample anchors and duplicate-step behavior are preserved.
The new batched regression covers entry, exit and re-entry independently per row. The existing policy-duration test now also covers entering tolerance with movement below position_drift: entering at step 10 with a 25-interval hold completes at step 35, instead of the previous step 34. Both regressions fail on the previous implementation. All 50 position-window and simulation-policy tests pass, including recorded-window replay and initial-anchor checks. Black and the 4441-export API documentation check also pass.
yuecideng
left a comment
There was a problem hiding this comment.
Reviewed commit 75e4466e6a4f56999ff2e842d6ee2147942aa46b.
Two P2 findings are attached inline: preserve reset-time provenance for lazily materialized metadata rows, and implement selective episode reset for the registered DC motor.
Actuator migration will be handled in a separate follow-up PR. Keep the current fused DC-motor kernel and GraphRuntimeControl implementation in this PR; adopting Newton's Actuator is not a prerequisite for this change. The reset fix can stay within the current design.
Validation used focused CPU checks and targeted reproductions. Native World/GPU execution, CUDA Graph capture, and sustained training performance were not rerun during this review.
| if not getattr(self, "_demo_metadata_initialized", True) and ( | ||
| self.dataset_manager is not None | ||
| or getattr(self, "rollout_buffer", None) is not None | ||
| or getattr(self, "_traj_buffer", None) is not None | ||
| ): |
There was a problem hiding this comment.
[P2] Preserve reset-time provenance for unmaterialized metadata rows
When no dataset manager, rollout buffer, or trajectory buffer is attached, this guard skips capturing reset-time metadata. For two environment rows, run reset(seed=12), then reset(seed=35, options={"reset_ids": [0]}), and finally get_demo_episode_metadata(1). Row 1 has not reset, but the lazy fallback reads the current cfg.seed and reports 35 instead of 12. A CPU comparison using the production reset/getter lifecycle reproduced base 12 versus head 35 for the untouched row.
The existing seed-preservation cases attach rollout or trajectory storage, so they exercise the protected _EpisodeMetadataRows path. Please retain lightweight reset-time seed/provenance per row independently of recorder attachment, while keeping full metadata materialization lazy, and cover an untouched row after a partial reseed.
There was a problem hiding this comment.
Implemented in 2c2baa4ba.
Fixed by retaining each row's reset-time seed and shared provenance snapshot independently of recorder attachment. Full episode metadata and row UUIDs remain lazy. The regression resets both rows with seed 12, then resets only row 0 with seed 35; row 1 retains seed 12 and its original configuration hash when first read. Both getter orders, recording enabled/disabled, repeated partial reset and returned-data ownership are covered.
| previous_velocity[row, column] = velocity | ||
|
|
||
|
|
||
| class _BatchedJointDCMotorControl(GraphRuntimeControl): |
There was a problem hiding this comment.
[P2] Implement selective episode reset for the registered DC motor
This subclass inherits RuntimeControl.prepare_episode_reset(), whose default implementation raises NotImplementedError. In a finalized Newton/DexUni scene with this motor installed, the existing public sim.spawn_result.reset_arenas([0]) path preflights every registered control and fails with _BatchedJointDCMotorControl does not support episode reset. A CPU probe through the actual SDK reset binding reproduced the failure.
Please implement the hook so it returns a commit callback that restores the selected arenas' motor targets and clears their effort/velocity buffers, preserving unselected rows and captured buffer addresses. Add focused reset coverage. This issue is limited to Scene.reset_arenas(); ordinary manager resets use batch dynamics clearing, and #837 explicitly restores the aliased motor buffers, so those paths are not claimed to raise this exception. The current fused motor implementation can be retained.
There was a problem hiding this comment.
Implemented in 2c2baa4ba.
Added prepare_episode_reset() using the resolved articulation handles to map Scene arena indices to motor rows. Preflight does not mutate state; its callback restores selected targets and clears their effort/velocity history in place, preserving other rows and buffer addresses. CPU coverage includes reordered arenas, empty selections and repeated resets. A two-arena DexUni test also passes through Scene.reset_arenas() and subsequent CUDA Graph stepping, with the graph remaining captured. The fused motor/GraphRuntimeControl implementation is retained; Actuator migration remains separate in #1131.
|
Once the two P2 findings are addressed, this PR can be merged:
Keep the current motor implementation in this PR. The reusable |
Description
Reduce allocation and property-update overhead in batched RL environments while preserving episode provenance, selected environment rows, and physical joint limits. This PR targets the DexSim 0.6.0rc1 baseline on
mainand provides shared simulation APIs for #837 and the G1 environment in #924.Changes
info; training can explicitly disable those snapshots.clamp_target_qpos=Falsewhile retaining physical joint limits, including named robot control groups and Newton mimic followers.PlaneCfg, explicit joint-limit gains, solver selection, and thecubqlmesh BVH option. Ground collision and geometry changes take effect before preparation on both backends.DEFAULT_DATA_MODULESpackage export.Validation
Python 3.12.13, Torch 2.11.0+cu128, the published DexSim 0.6.0rc1 wheel, Newton 1.7.0, and RTX 5090 D v2:
The preceding CI test job stopped before pytest because the test-plan artifact download returned
ECONNRESET. Current results are available under GitHub checks. Native validation covers short simulation and reset sequences; long training performance below retains its original runtime conditions.Dependencies
Targets
main. #837 uses this branch as its base; #924 uses its G1 simulation APIs.Type of change
Checklist
black .with Black 26.10.1.Historical G1 rough-terrain benchmark and runtime versions
Integration benchmarks use the G1 Rough environment from #837, one RTX PRO 5000 Blackwell 72 GB GPU, driver 580.126.20, and 4096 environments. Each run performs 100 PPO updates; timing excludes compilation, initialization, and the first 10 updates. Results are the mean of two runs per path.
Rollout SPS = transitions / collection time. Training SPS = transitions / (collection + PPO update time). Memory values are process-tree peaks across initialization and training.
DexUni Training SPS is 4.61% below Isaac. All three paths use the 37-joint G1, 187 height rays, world dt=0.005 s, 2 substeps, solver dt=0.0025 s, and control dt=0.02 s. Newton contacts update every solver substep. Shared settings are implicitfast / pyramidal, 100 solver iterations, 50 line-search iterations, and 24 steps/update.
66bf4840ae37b028; frozen terrain adapter and shared training entry point4e08405c+ tasks9f32fa08Initial policies and PPO configurations are identical. Randomized mass and inertia samples differ across adapters because they consume random numbers in different orders.