Skip to content

Integrate cables with Gym observations, replay and visualization - #839

Merged
yuecideng merged 40 commits into
mainfrom
codex/cable-framework
Nov 12, 2026
Merged

yuecideng merged 40 commits into
mainfrom
codex/cable-framework

Conversation

@KeqiYe

@KeqiYe KeqiYe commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Integrate CableObject with Gym scene construction, the unified ObservationManager, trajectory recording and Viser through the DexUni runtime. Cable observations carry world XYZ, XYZW, COM linear velocity and angular velocity for every segment. Shared history retains T+1 state boundaries for T actions.

Per-environment cable exports now save meta.arena_origins_w in exported row order. Dynamic replay translates captured reset positions into the recorded world frame for initial validation, so a trajectory automatically saved from arena 1 can replay in a fresh single-environment scene. Raw observations retain their world coordinates. Files without origin metadata retain the existing matching-world-placement requirement. Replay still requires matching environment counts and full-reset initial states.

Viser gathers visible environment/component poses on their source device before CPU transfer and quaternion normalization. Segment capsules retain their original world placement, orientation and node order. Gym accepts both file-backed and inline cable visual meshes.

Stack

Review fixes

Finding Change and coverage
Per-environment replay placement Persist source origins and translate only the captured validation positions. CPU cases cover both source rows, changed target origins, immutable observations/actions and genuine initial-state mismatches. A native two-environment test commits arena 1 through public reset with auto-save enabled, creates a fresh single-env scene, and replays the saved artifact through its final action.
Visible pose export scaling Gather visible poses before conversion. The existing exporter test now covers NumPy, Torch CPU and Torch CUDA with noncontiguous visible rows, checks the conversion-boundary shape and verifies exported poses.

Validation

Latest main synchronization is committed at 60046dc577ff558d725f27fb3a5d2dac7a2120ad, integrating main at 7b9f55b836da7d9795e51259ae6ff2006f661872. Conflict resolution preserves both cable and fluid configuration, recording and replay paths.

  • 658 focused CPU tests passed across cable/fluid configuration, observations/history, persistence, recording provenance, replay, trajectory utilities, SimManager and Viser runtime/export; eight native/GPU cases were excluded from this CPU run.
  • 53 context and API-checker tests passed.
  • One native CUDA regression passed: two-environment cable auto-save followed by saved arena-1 action replay in a fresh single-environment scene.
  • Whole-tree Black 26.10.1, whitespace, syntax, context-map and API coverage checks pass (4883/4883 exports documented).
  • Owning context was reviewed against the merged implementation; both main's new demo-execution guidance and the existing cable scene/replay guidance are retained and remain accurate.
  • Earlier review-fix validation at 6a7a29d5d also exercised the CUDA visible-row exporter and confirmed that new regression cases fail before the fixes. Existing legacy replay, T+1 storage, missing-cable and genuine reset-mismatch checks remain covered.

Runtime checks use Python 3.12, Torch 2.11.0+cu130 and local DexSim build 88ac9206e0cb9c7894df6c7780f3e157806f0239 on an RTX 5090. Runtime dependency metadata follows main.

Export measurement

One visible environment, float32 input, seed 42, two warmups and seven repetitions. These CUDA timings include pose gathering, transfer to CPU and quaternion conversion, with Torch CUDA synchronization around timing. Native state fetching and browser publication are outside the measurement. Outputs match exactly.

Environments × segments Before, median After, median Pose bytes sent to CPU before → after
16 × 64 0.0505 ms 0.0507 ms 28,672 → 1,792
1024 × 64 1.652 ms 0.0405 ms 1,835,008 → 1,792
4096 × 256 30.39 ms 0.0573 ms 29,360,128 → 7,168

Hardware/software: Ryzen 9 9950X, RTX 5090, NumPy 2.2.6, Torch 2.11.0+cu130 and 16 Torch CPU threads. Native batch fetching remains owned by the object adapter.

Type of change

  • New feature
  • Bug fix
  • Enhancement
  • Documentation update

Example

python -m embodichain run-example sim/demo/cable_viser --num-envs 2 --port 8080 --duration 12

Checklist

  • Run black . with Black 26.10.1.
  • Review and update the owning project context.
  • Check API documentation coverage.
  • Verify the identified regressions through production entry points.
  • Measure equivalent pose-export workloads and preserve output values.

xiemenghong and others added 19 commits September 17, 2026 09:48
Migrate DexUni example settings to the current namespaced solver interface and preserve the existing cloth contact values. Update the Newton config regression test and document the new mapping contract.
Port the retained MJVBD V2 integration consumers to the current DexUni public interface, while keeping AutoSolver as the Newton default. Preserve namespaced solver options, tutorial calibration, reset/mimic handling, and current task configuration.
Allow geometry-only ContactQuery data for DexUni VBD paths and full impulse/friction data for pure MuJoCo. Add capability tests and GPU end-to-end coverage.
…e-adaptation

# Conflicts:
#	embodichain/lab/sim/cfg/simulation.py
#	tests/sim/test_cfg.py
@KeqiYe KeqiYe added enhancement Improvements to existing capabilities; use feature for new capabilities. visualization Browser viewers, markers, trajectory previews and interactive debugging displays. dataset gym Environment lifecycle, managers, configuration composition and registration. labels Oct 22, 2026
@yuecideng yuecideng added data-pipeline Data collection, ODS, sampling, recording, persistence, conversion and analysis. feature New user-facing capabilities. sim-object Simulation entity configuration, state, creation, destruction and lifecycle. and removed enhancement Improvements to existing capabilities; use feature for new capabilities. dataset labels Oct 25, 2026
yuecideng and others added 9 commits November 7, 2026 14:06
Synchronize main at 6881882 with the open environment's DexSim 0.6.0+c3e92584 build. Read solver contact reports through the new API, retaining per-row impulse validity alongside main's contact cache, observer lifecycle and fused loss counters. Forward raw MuJoCo contact parameters through sparse configuration and document the geometry-detection/report distinction. Require dexsim_engine>=0.6.0,<0.7.0.

Validation: 3474 focused CPU regressions passed; 17 CUDA sampling tests passed; 2 native DexUni coupling tests passed; 10 native contact/support/camera checks passed (the support fetch hook was updated and rerun separately). Black 26.10.1: 2215 files unchanged. API docs: 4286/4286. Context checks and tests passed. All 146 referenced DexSim import symbols resolved. Wheel built and verified against worktree bytes and installed DexSim version. Hybrid headless tests used SDL_VIDEODRIVER=dummy; shared conda packages were not reinstalled. The complete example scene matrix and interactive windows were not rerun.
Keep gravity compensation and opt-in mimic stabilization as the supported Newton extensions. Remove seven unused root fields, generic SDK option forwarding and articulation-wide SDF rewriting. Leave native creation/geometry choices at source or SDK defaults and update the owning context.

Reuse existing configuration, source overlay and mimic coverage; reject withdrawn fields at both public decode boundaries and roundtrip non-default values for retained fields. Remove obsolete capability-forwarding cases. Validation on conda open / DexSim 0.6: 724 CPU tests passed, 7 deselected; 2 native two-way coupling tests passed with and without CUDA Graphs. Black 26.10.1: 2215 files unchanged; API docs 4286/4286; context checks/tests and final wheel/source metadata validation passed.
Use the current DexUni configuration contract directly. Remove AutoSolver aliases, legacy flat-option translation and attribute injection, SDK-version probing, and stale solver guards. Preserve native default routing, grouped expert options and contact_mode=auto. Update configuration regression coverage and project context.

Validation: 728 focused CPU tests passed, 20 deselected; black 26.10.1 left 2215 files unchanged; API docs cover 4286/4286 exports; context validation and routing checks pass. The rebuilt wheel matches the changed source and accepts the installed DexSim 0.6.0+c3e92584 build. Native GPU selections were not rerun for this configuration cleanup.
Replace retired solver terminology throughout the tracked documentation and tests. Document the default DexUni family separately from scene ownership and contact_mode=auto, migrate the example's iteration options into vbd_options, and correct path-specific contact-report guidance. Cover the explicit automatic contact policy and native ownership defaults using existing configuration cases.

Validation: 738 focused CPU/docs regressions passed, 20 deselected; nine Python snippets parse, and updated Python/YAML configurations convert to native DexUni automatic defaults. Black 26.10.1, API documentation coverage (4286/4286), context validation and whitespace checks pass. A tracked-file scan finds no retired solver names. The Sphinx build is retried after this commit because the architecture generator requires committed source.
Keep reset-time contact observations readable until the next solved report, require eager native acquisition outside caller-owned CUDA capture, and normalize Newton impulses with the internal solver duration while retaining World-step history clocks.

Cover real articulation resets, native graph consumers, multi-substep support forces, variable World intervals and history clocks; update sensor guidance and align timeline reset test buffers with quantity validity.
@KeqiYe
KeqiYe force-pushed the codex/cable-framework branch from 485dfd5 to 6ca6566 Compare November 9, 2026 02:20
@KeqiYe
KeqiYe force-pushed the codex/cable-foundation branch from 612b68d to 0e304ee Compare November 9, 2026 02:20
@KeqiYe KeqiYe changed the title Integrate cables with Gym, visualization and trajectory data Integrate cables with Gym observations, replay and visualization Nov 9, 2026
@KeqiYe
KeqiYe marked this pull request as ready for review November 9, 2026 05:36
@greptile-apps

greptile-apps Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The PR changes cable behavior across Gym recording, replay, and browser viewing.

Fix inline cable visual loading before merging.

Fix All in CodexFindings

  1. P1 Inline cable meshes cannot load ▶
Fix with agent prompt
### Issue 1
embodichain/lab/gym/utils/gym_utils.py:972-973
`config_to_cfg()` rejects cable visuals defined with `vertices` and `triangles`. These meshes leave `MeshCfg.fpath` as `None`, but this branch passes it to `get_data_path()`, where `os.path.isabs(None)` raises `TypeError`. The environment cannot load.

Only resolve the path when `cable.visual_shape.fpath` is present, and add a loader test for an inline mesh.

```suggestion
        if isinstance(cable.visual_shape, MeshCfg) and cable.visual_shape.fpath is not None:
            cable.visual_shape.fpath = get_data_path(cable.visual_shape.fpath)
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds cable scene declarations, described observations, trajectory storage and reset-based dynamic replay. Browser export draws each segment as a capsule.

  • Gym environments can declare cables and observe every segment’s state.
  • Cable recordings keep a state for every segment at each action boundary.
  • Cable recordings replay actions from a matching full-reset scene.
  • Viser shows each cable segment as a capsule with its live pose.
  • Viser startup allows more time for its browser client to finish building.
  • Viser startup keeps an explicitly chosen command setting.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Config[Gym cable config] --> Scene[Cable scene]
  Scene --> Observation[Described segment state]
  Observation --> History[Shared history and trajectory storage]
  History --> Replay[Full reset and initial-state check]
  Replay --> Actions[Replay recorded actions]
  Scene --> Export[Capsule geometry and world poses]
  Export --> Browser[Viser browser]
Loading

Reviews (1) · Last reviewed commit: "feat(cable): integrate unified observati..." · Reviewed by Greptile

Comment thread embodichain/lab/gym/utils/gym_utils.py Resolved
@yuecideng

Copy link
Copy Markdown
Contributor

#839 录制回放与 Gym 修改方案(6ca65663)

沿用 #838 的 DexUni 整场 episode reset。建议将轨迹坐标契约与 Gym 配置修复作为本层合入门槛。

合入前必须修复

  1. 非零 arena 的自动存档应能单独回放。 默认自动保存将 env 1 写成单环境文件,但线缆 XYZ 包含源 arena 原点。在新格式中标记坐标系,写入环境相对位置,或持久化源环境原点并在校验时转换。仅平移受影响的位置量,保留四元数、速度以及实时 segment_state_w 的世界坐标语义。旧文件继续按既有世界坐标解释;缺少源原点且初态不匹配时给出明确错误。动态回放继续从完整 reset 开始,并保留对中途跳帧和广播的限制。

    验收:建立两个原点不同的环境,打开默认自动保存,实际提交 env 1 的 episode,再用普通单环境加载该文件并执行全部动作。检查 T 个动作对应 T+1 个状态边界,初态按相同坐标系匹配,平移不改变四元数和速度。覆盖同原点旧文件、真实初态不匹配、禁止中途跳帧三种情形。

  2. 支持 Gym 配置中的内联网格。 config_to_cfg 对没有 fpath 的合法 vertices/triangles 网格调用 get_data_path(None),会在 Spawn 之前抛出 TypeError。仅对有文件路径的 MeshCfg 执行路径解析,复用现有几何校验。

    验收:从实际配置入口加载带内联三角面的线缆,验证描述符保留顶点和索引;文件网格仍可解析;缺少三角形索引时报出几何配置错误。

后续优化

Viser 先在设备端筛选可见环境,再传输姿态。 流体导出已有这个顺序。按 1,024 环境、每条 128 段计算,每帧全部姿态约 3.5 MiB,只看一个环境所需约 3.5 KiB。建议在扩大并行可视化规模前处理。验收时选择单个非零环境及非连续多环境,检查送入 CPU 转换的段数只随所选行增长,输出位置只应用一次原点偏移。帧耗时收益需另行实测。

回放初态检查 · 流体可视化选行 · Isaac Lab 3.0.0 相对坐标录制

@KeqiYe
KeqiYe force-pushed the codex/cable-foundation branch from 0e304ee to 91ae175 Compare November 10, 2026 06:05
Base automatically changed from codex/cable-foundation to main November 11, 2026 07:10
KeqiYe and others added 2 commits November 11, 2026 15:21
Synchronize through fcca082 while retaining cable Gym observations, replay and Viser integration on the current simulation APIs. Preserve main's cable lifecycle fixes and published runtime configuration.

@yuecideng yuecideng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the 25-file delta from fcca082 to 5d8f201 with independent architecture, engine/HPC and API passes. Two P2 findings remain, covering per-environment cable replay and visible-environment export scaling. The inline mesh configuration regression is fixed. Focused tests passed, including the existing two-environment CUDA recording check on this commit. The saved-artifact reproduction uses production CPU recording/save/load APIs. Export timings measure the CPU conversion phase and exclude native fetching and actual CUDA transfer. Recommend addressing the findings before merge.

)
for key, expected in initial.items(include_nested=True, leaves_only=True):
observed = current[key].to(expected)
if observed.shape != expected.shape or not torch.allclose(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve source arena placement for per-environment cable exports

With record_trajectory=True and the default trajectory_auto_save=True, a batched scene saves each row via _save_trajectory_for_env() -> save_trajectory(env_ids=[env_id]). Each artifact declares num_envs=1 while retaining that row's cable world positions. For source arena 1, an otherwise matching single-environment replay resets at arena 0 and this comparison rejects it. Replaying the artifact in the original batch also fails the environment-count guard at lines 113-116. meta.env_ids is saved but never used to resolve placement. A production save/load CPU probe accepts row 0 at X [0.1, 0.2, 0.3], but rejects row 1 at X [5.1, 5.2, 5.3]; both files contain raw_observations. The CUDA recording test disables auto-save, so it misses this default output path. Persist source arena origins or store relative positions, then compare the initial states in a common frame while retaining the full-reset/solver-history restriction. Cover auto-saving a nonzero row and replaying the resulting file in a fresh single-env deployment.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a7a29d. Cable exports now retain arena_origins_w in selected-row order. Replay translates the captured reset positions into the source frame for comparison, while preserving raw world observations. Coverage includes source rows 0/1, changed target origins, legacy files and real mismatches. The native regression now uses public reset with auto-save enabled for arena 1, loads the actual file into a fresh single-env scene and verifies the final state after action replay.

raise RuntimeError(
"Call SimulationManager.prepare() before capturing cable state."
)
pose = data.segment_pose_w

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Select visible cable rows before converting poses to CPU

This returns the full CUDA (B,S,7) batch, and pose_to_position_wxyz() copies and normalizes all rows before the subsequent env_ids selection. Even the default env_ids=[0] therefore incurs work proportional to every simulated cable. A transfer-boundary probe for B=4096/S=256 receives 29,360,128 pose bytes while publishing 256 nodes needing 7,168 bytes. On a Ryzen 9 9950X / NumPy 2.2.6, the actual CPU conversion-plus-selection phase measured 49.21 ms median versus 0.015 ms for equivalent selected-first conversion, with identical outputs (two warmups, seven repetitions). These timings exclude native fetching and actual CUDA transfer. Other asset paths already follow this order in main, but this new branch extends that cost to cable scenes. Gather the selected environment/segment rows on device before conversion. Add a regression that checks the shape at the CPU boundary for a nonzero or noncontiguous visible selection.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a7a29d. Pose batches now gather visible environment/component rows on their source device before CPU conversion. The regression verifies the boundary shape and noncontiguous row order for NumPy, Torch CPU and CUDA. On the RTX 5090, the 4096x256 transfer-plus-conversion median decreased from 30.39 ms to 0.0573 ms with equal outputs; the 16x64 case remained approximately 0.05 ms. Measurements exclude native state fetching and browser publication. Full workload and timing details are recorded in the PR description.

yuecideng added a commit that referenced this pull request Nov 12, 2026
Integrate PR #839 at 6a7a29d, retain the YAM support already on main, and migrate cable grasping and routing to the canonical Atomic Skill API.
Integrate main at 7b9f55b while preserving cable and fluid configuration, recording and replay paths. Both replay suites and native cable recording pass.
@yuecideng
yuecideng merged commit d9f50a4 into main Nov 12, 2026
7 of 12 checks passed
@yuecideng
yuecideng deleted the codex/cable-framework branch November 12, 2026 12:16
@KeqiYe KeqiYe mentioned this pull request Nov 13, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data-pipeline Data collection, ODS, sampling, recording, persistence, conversion and analysis. feature New user-facing capabilities. gym Environment lifecycle, managers, configuration composition and registration. sim-object Simulation entity configuration, state, creation, destruction and lifecycle. visualization Browser viewers, markers, trajectory previews and interactive debugging displays.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants