Skip to content

feat(gym): align expert and policy substep EEF control - #985

Merged
yuecideng merged 5 commits into
DexForce:mainfrom
yuecideng:codex/rlinf-configured-stack
Nov 5, 2026
Merged

yuecideng merged 5 commits into
DexForce:mainfrom
yuecideng:codex/rlinf-configured-stack

Conversation

@yuecideng

@yuecideng yuecideng commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Let Task Program experts and policies execute through one configured EEF controller. RLinf's existing Stack Cube checkpoints use unbounded relative requests and a target held across physics substeps; the existing bounded, single-step contract has different semantics.

  • Add eef_pose.relative_substep_dls@1 (RelativeEefDlsAction). Reuse the existing solver-order binding and arena/TCP Jacobian adaptation, while keeping each action contract's bounds and timing separate.
  • Keep physics/sensor orchestration in EmbodiedEnv, substep counting/refresh/masking in ActionManager, and joint-command generation in action terms. BaseEnv and SimulationManager have no PR diff. The environment connects ActionManager.get_substep_callback(num_substeps) to the existing batched update(after_substep=...) path. Each callback owns one interval, skips refresh after the final physics step and preserves recording/visualization publication.
  • Add expert_trajectory.use_action_manager, default false. Position-only experts retain their exact FK target and existing storage schema. Completed rows keep measured joint holds after conversion and refresh, including partly closed binary grippers.
  • Add binary-gripper open_on_zero, default true. Stack Cube selects false to preserve its zero-closes boundary.
  • Reset instantiated observation/reward functors between episodes; the old class check ran after instantiation and omitted their reset hooks.

The PR is based on main 16c429bc48ba0bc09e2256be26d8994fcb3b5e0b. Compatibility was also checked in a temporary merge with main 184fa6031fc8e377441c7e936962064b1466621a. No new dependencies or solver/physics backend changes. Consumer: yuecideng/RLinf#8. Related #836 and #959 address broader optimization and stability work.

Review corrections and simplification

Review item Final implementation
Greptile: Jacobian frame and TCP Rotate both Jacobian blocks into the arena and shift the linear block to the configured TCP.
Greptile: joint order Map selected joints into solver order for FK and map increments back before applying commands.
Greptile: completed expert rows Apply inactive-row masking after expert conversion and on every substep refresh.
Greptile: recording cadence Use batched simulation callbacks so intermediate frames remain published and only the final frame is deferred.
Public action facade Export the new action in managers.__all__.

Removed the single-use expert-conversion and validation helpers, eliminated repeated expert-target FK, and replaced a duplicated DLS test implementation with a closed-form expectation. The new controller class decreased from 195 to 175 lines. Shared private frame/order helpers follow the existing action binding rather than adding a controller framework. The manager exposes one interval callback in place of a separate refresh flag and immediate-refresh method; the environment contains no substep counter or action-refresh policy. The two stateful manager-reset fixes remain because they are required by episode ownership.

Frame adaptation preserves the solver Jacobian's strides for substep DLS; the bounded controller retains its original contiguous layout. This avoids changing CUDA reduction order while correcting the geometry.

Type of change

  • Bug fix
  • New feature
  • Documentation update

Validation

Python 3.11.14, Torch 2.7.1+cu128, NumPy 2, DexSim 0.5.1rc1:

  • black==26.10.1: black ., passed.
  • Latest scheduling/lifecycle CPU selection: 160 passed, including manager callback counts, fresh control intervals, completed-row holds, direct-action fallback, experts/replay, configured RL and context/API-doc checks. Earlier geometry/functor coverage passed 303 tests, including 303 against a temporary merge with main 184fa60; those unchanged numerical paths were not rerun for the callback move.
  • Simulation publication selection in tests/sim/test_sim_manager.py: 13 passed, including real SimulationManager.update/render scheduling against a fake world, physics-rate recording, final-frame camera sharing and substep callbacks.
  • Targeted before/after regressions: old head e9cfe070 has 6 assertion failures / 4 passes; the fixed head passes all 10. Failures measure coordinates, joint order, completed-row commands and missing frames rather than missing mock methods.
  • RLinf CPU environment selection: 53 passed. Public action star-import and action-contract resolution agree.
  • python docs/scripts/check_api_docs.py: 3733/3733 exports documented. Context checks passed. Sphinx dummy build completed with the same 839 existing warnings as the main baseline; normalized warning sets are identical.

Physical RTX PRO 5000 72 GB, GPU 5:

  • Fixed-action replay, 8 environments × 450 control steps: initial and per-step states, cube positions, rewards, termination, truncation, stability and both raw RGB hashes are exactly equal to the preceding implementation. Exit 0; zero simulators remain after cleanup. This is replay equivalence, not a learned-policy success rate.
  • Actual Task Program collection, formal request seed 50000, 600 steps: actions, states, physical measurements and both raw RGB streams match until expert completion. The first inactive control step is 592; post-completion state/image differences begin at recorded frame 593 as the corrected joint hold takes effect. Both implementations stabilize at step 563, complete the program and pass 600-step acceptance; both fail the unchanged 450-step threshold. This deliberate completed-row correction supersedes the earlier full-600-frame equivalence claim.

Earlier independent RLinf PPO integration collected two 450-step rollouts, performed one update and saved DCP plus full weights: 198 Adam states at step 1, 477 frozen tensors unchanged and 675 finite exported tensors. That training integration was not rerun for this review; the new validation above checks the affected control/lifecycle contracts. No new learning-quality claim, full repository/backend/multi-node qualification or formal training run is included. Remote CI results remain separate from local validation. AI assistance was used for review, implementation and validation.

Screenshots

No UI change. Raw camera and recording cadence are covered above.

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context and updated the affected contracts.
  • Public API changes are reflected in the API docs.
  • I verified focused behavioral coverage and reviewed the necessity of extended tests.
  • Dependencies have been reviewed; no SDK dependency change is needed.

Signed-off-by: yuecideng <dengyueci@qq.com>
@yuecideng yuecideng added enhancement Improvements to existing capabilities; use feature for new capabilities. gym Environment lifecycle, managers, configuration composition and registration. rl Reinforcement learning algorithms, collectors, trainers and training contracts. labels Nov 4, 2026
Signed-off-by: yuecideng <dengyueci@qq.com>
@yuecideng
yuecideng marked this pull request as ready for review November 5, 2026 11:23
Signed-off-by: yuecideng <dengyueci@qq.com>
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issue remains.

Summary

Adds RelativeEefDlsAction and an opt-in route that lets experts use the configured policy controller.

  • Relative EEF requests can now refresh joint targets between physics steps.
  • Position-only experts can run through the policy's configured action terms.
  • Stateful observation and reward functors reset with their episode rows.

All four earlier, unnumbered findings are addressed: arena/TCP coordinates, solver joint order, completed-row holds, and recording cadence. No new actionable issues were found in the changes since the previous review.

Diagram

sequenceDiagram
    participant E as EmbodiedEnv
    participant A as ActionManager
    participant S as SimulationManager
    E->>A: Process and apply initial command
    E->>A: Get callback for control interval
    E->>S: Run batched physics update
    loop Each physics substep
        S->>E: After-substep callback
        E->>E: Sample sensors
        E->>A: Run interval callback
        opt Another physics step remains
            A->>A: Refresh terms and restore inactive holds
            A->>A: Apply refreshed commands
        end
        S->>S: Publish intermediate frame
    end
Loading

Reviews (3) · Last reviewed commit: "refactor(gym): move substep scheduling i..." · Reviewed by Greptile

Signed-off-by: yuecideng <dengyueci@qq.com>
Signed-off-by: yuecideng <dengyueci@qq.com>
@yuecideng
yuecideng merged commit a943ce0 into DexForce:main Nov 5, 2026
10 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Improvements to existing capabilities; use feature for new capabilities. gym Environment lifecycle, managers, configuration composition and registration. rl Reinforcement learning algorithms, collectors, trainers and training contracts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant