Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
75d3c9a
feat(sim): default Newton rigid simulation to MJVBD V2
Sep 17, 2026
731a20d
fix(sim): tune MJVBD V2 manipulation examples and record limitations
Sep 18, 2026
51af322
fix(tutorial): align drawer damping across physics backends
Sep 20, 2026
bc5369b
docs(sim): clarify MJVBD V2 coupling and control limitations
Sep 20, 2026
5f19889
fix(sim): adapt examples to DexUni solver config
yuecideng Sep 29, 2026
1e9d5f9
Merge remote-tracking branch 'origin/main' into codex/dexuni-interfac…
yuecideng Sep 29, 2026
347ce76
fix(sim): unify Newton solver adaptation on DexUni
yuecideng Sep 29, 2026
aa9f0b6
feat(sim): expose DexUni contact sensor data
yuecideng Sep 29, 2026
1832e56
test(sim): cross-check DexUni contact impulses
yuecideng Sep 29, 2026
c763aad
test(sim): assert DexUni contact invariants
yuecideng Sep 29, 2026
32e08fc
Merge remote-tracking branch 'origin/main' into codex/dexuni-interfac…
yuecideng Oct 3, 2026
7049fcf
fix(sim): update DexUni migration validation
yuecideng Oct 3, 2026
a09bd7e
docs(sim): use DexUni terminology consistently
yuecideng Oct 3, 2026
ad9ba87
wip
yuecideng Oct 4, 2026
b589354
Merge latest main into DexUni interface adaptation
yuecideng Oct 10, 2026
9428020
feat(physics): support current DexUni two-way coupling contracts
yuecideng Oct 10, 2026
6e60b8e
fix(dexsim): adapt latest 0.5.1 runtime contracts
yuecideng Oct 20, 2026
794bbb9
feat(dexsim): consolidate two-way coupling adaptation from PR #720
yuecideng Oct 21, 2026
e8a6419
fix(dexsim): reconcile current particle material and task contracts
yuecideng Oct 21, 2026
b8922a0
Merge latest main into DexSim adaptation
yuecideng Oct 27, 2026
57d0262
fix(dexsim): follow current DexUni dispatch and device defaults
yuecideng Oct 27, 2026
1eef398
feat(sim): expose DexSim controls and deformable initialization
Oct 27, 2026
9a2d016
fix(sim): preserve continuous URDF joint limits for DexUni
yuecideng Oct 27, 2026
a48f07c
fix(examples): complete DexUni workflows and physical placement checks
yuecideng Oct 27, 2026
f9f128d
Merge main be57e1b97 into DexUni adaptation
yuecideng Oct 27, 2026
4d72509
fix(examples): preserve placement asset frames across renderers
yuecideng Oct 27, 2026
da67a3e
Merge main into codex/adapt-latest-dexsim
yuecideng Nov 2, 2026
37085da
Merge main and adapt DexSim 0.6 contact reports
yuecideng Nov 7, 2026
731427f
refactor(sim): narrow Newton articulation root options
yuecideng Nov 7, 2026
834f88d
refactor(sim): remove legacy AutoSolver compatibility
yuecideng Nov 7, 2026
a6c3693
docs(sim): standardize native DexUni automatic dispatch
yuecideng Nov 7, 2026
f249fbe
update dexsim version
yuecideng Nov 7, 2026
8f792d0
Merge branch 'main' into codex/adapt-latest-dexsim
yuecideng Nov 7, 2026
28f8b6a
fix(sensors): honor native contact sampling boundaries
yuecideng Nov 7, 2026
0e304ee
feat(cable): adapt foundation and episode defaults to DexSim 0.6
KeqiYe Nov 9, 2026
6ca6566
feat(cable): integrate unified observations replay and visualization
KeqiYe Nov 9, 2026
edf83b9
fix(gym): preserve inline cable mesh geometry in config loading
KeqiYe Nov 11, 2026
5d8f201
Merge main into codex/cable-framework
yuecideng Nov 12, 2026
6a7a29d
fix(cable): preserve replay origins and select visible poses before e…
yuecideng Nov 12, 2026
60046dc
Merge latest main into cable framework before landing PR 839
yuecideng Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion agent_context/MAP.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ topics:
aliases: [browser visualization, viser, sim visualization, marker visualization, 标志物可视化, 仿真可视化]
keywords: [VisualizationCfg, ViserServerCfg, SceneManifest, visualization, EntityGizmoManipulator, robot_ik_gizmo,
GizmoCfg, MarkerGroup, MarkerPrototypeCfg, RobotMarkers, VisualizationMarkers, robot markers, ghost robot, 机器人可视化, backpressure, capture_visualization, Newton deformable, particle set,
render topology, quaternion, xyzw]
render topology, quaternion, xyzw, cable visualization, cable capsule]
paths: [topics/sim-visualization/sim-visualization.md]
source_of_truth:
- embodichain/lab/visualization/protocol.py
Expand Down
1 change: 1 addition & 0 deletions agent_context/topics/env-framework/env-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ managers, demonstration collection and the opt-in Task Program bridge.
| Raw schema, optional history and exact historical queries | `managers/observation_manager.py`; Env dispatches capture/reset/close boundaries |
| Optional measured physical objectives | `embodichain/lab/gym/envs/objectives/`; Gym owns observation/reset hooks independently of expert acceptance |
| Controller-ready commands | `embodichain/lab/gym/envs/types.py`: `ControllerAction` |
| Cable scene declarations and raw recording | `gym_utils.py` → `EmbodiedEnv._setup_interactive_objects()`; `trajectory_state.py` → `observations.get_entity_state` |
| Demo segment execution / outcomes / collection qualification | `embodichain/lab/gym/envs/demo.py` |
| Configured Task Program registration | `embodichain/lab/gym/envs/task_program/registration.py` |
| Timing instrumentation | `embodichain/lab/gym/utils/profiler.py` |
Expand Down
18 changes: 17 additions & 1 deletion agent_context/topics/env-framework/execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,8 @@ not orchestrate multiple experiments or qualify kinematic playback.
cursors keep their accepted-versus-diagnostic semantics; there is no independent
history instance or write lifecycle. Standard state fields use observation functors; the host
does not perform a separate physics read in its write hook. Full robot state,
articulations and selected rigid objects retain their existing action schema.
articulations, selected rigid objects and cable segments share that acquisition
and retain their existing action schema.
`cfg.trajectory_uids` restricts non-robot replay fields. Raw channels, T+1
timing and lifetime queries are owned by the
[recording contract](env-framework.md#expert-trajectory-contract).
Expand Down Expand Up @@ -347,6 +348,21 @@ not orchestrate multiple experiments or qualify kinematic playback.
keeps its own cursor and layout. Internally allocated expert policy observations
include the terminal T+1 slot in the backing store; the public T-transition
view and existing dataset action/annotation alignment stay compatible.
- **Cable observations**: `states.cables[uid].segment_state_w` and the matching
raw observation field share the T+1 rollout storage. `get_entity_state` describes
world segment position, XYZW orientation, COM linear velocity and angular
velocity, preserving each cable's segment topology and row validity. The field
records observations, not solver history. `ReplayWrapper` accepts dynamic
replay only when every recorded initial state matches the full reset; it
rejects scrubbing, partial reset and broadcasting world-space cable states.
Cable artifacts store `meta.arena_origins_w` in exported row order. Replay
translates captured reset positions into the source world frame for comparison,
allowing a per-environment export to replay at a different arena origin.
Raw observations, orientations and velocities retain their original meaning.
Older files without origin metadata require matching world placement.
`trajectory_state.restore_trajectory_state()` rejects cable entries before
any write. Use action replay to validate a new episode, not a visible-state
restore as a history-preserving checkpoint.
- **Fixed action layout**: an ActionManager-backed trajectory uses the batched
policy action space; a controller/expert trajectory uses the private
`ExpertActionSpec` layout. One trajectory cannot mix these schemas.
Expand Down
8 changes: 6 additions & 2 deletions agent_context/topics/manager-functor/manager-functor.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,12 @@ matching their source functions/parameters, including callable strings and
omitted default arguments. Env only supplies the declarations. Recording-only
fields never enlarge policy observations or policy rollout allocation.

`observations.get_entity_state` derives native UID, quantity, arena/joint frame,
representation, shape, dtype and joint order from its source. Immutable
`observations.get_entity_state` derives native UID, quantity, reference frame,
representation, shape, dtype and joint order from its source. Cable
`segment_state_w` uses world coordinates and a segment axis; its descriptor
includes COM linear velocity and angular velocity alongside position and XYZW
orientation. Recording declares these fields through `trajectory_observation_cfgs`,
so cable observations share the ordinary acquisition and history lifecycle. Immutable
schema descriptors are reused by semantic value; no simulation handles enter
the bounded cache. Pose acquisition uses the resolved UID without constructing
another `SceneEntityCfg` during each step. Explicit boundaries
Expand Down
13 changes: 8 additions & 5 deletions agent_context/topics/sim-visualization/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ are nested under `visualization.viser_server`; there is no top-level
| `point_cloud_max_points` | `100000` | Deterministic per-cloud downsampling limit |
| `sensor_image_fps` | `2.0` | Maximum RGB image capture rate; `None` captures once per eligible simulation step |
| `soft_body_fps` | `5.0` | Maximum soft-body and cloth vertex capture rate |
| `allow_commands` | `False` | Allow trusted Viser clients to drag configured Gizmos; common `--viser` launchers set this to `True` |
| `allow_commands` | `False` | Allow trusted Viser clients to drag configured Gizmos; common `--viser` launchers enable commands when the deployment does not specify a policy |
| `viser_server` | `ViserServerCfg()` | Server bind configuration |

### `ViserServerCfg`
Expand Down Expand Up @@ -76,7 +76,10 @@ manifest when manager APIs marked the topology dirty.

`merge_args_with_gym_config()` makes `--viser` set `headless=True`, changes the
backend to `"viser"`, and applies the CLI sampling/server overrides. A
file-based config may also contain a `visualization` section. The old nested
file-based config may also contain a `visualization` section. An explicit
`visualization.allow_commands=False` remains read-only when `--viser` enables
the viewer, so automated deployments can reserve robot drive targets for their
expert. The old nested
key `visualization.server` is accepted and normalized to
`visualization.viser_server`.

Expand All @@ -87,9 +90,9 @@ limiting.

`visualization_cfg_from_args()` only builds visualization config.
`SimulationManagerCfg` makes the simulation headless whenever the resulting
backend is `"viser"`. The common `--viser` option also enables Gizmo commands;
there is no separate `--viser-gizmo` option. Programmatic configurations may
retain `allow_commands=False` for read-only deployments.
backend is `"viser"`. The common `--viser` option defaults to enabling Gizmo
commands when no file policy is present; there is no separate `--viser-gizmo`
option. Programmatic configurations retain their configured command policy.

### Asset preview

Expand Down
17 changes: 15 additions & 2 deletions agent_context/topics/sim-visualization/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Read this when the request needs these details. [Topic overview](sim-visualizati

| API | Behavior |
|---|---|
| `start_visualization()` | No-op for backend `"none"`; otherwise starts or returns the active runtime and forces an initial capture |
| `start_visualization()` | No-op for backend `"none"`; prepares declared assets (including cable-only scenes), otherwise leaves an empty Spawn builder open; starts or returns the active runtime and forces an initial capture |
| `notify_visualization_topology_changed()` | Increments the manager-local topology revision |
| `refresh_visualization()` | Rebuilds and asynchronously publishes a manifest immediately |
| `capture_visualization(force=False)` | Refreshes dirty topology, then captures a due frame |
Expand Down Expand Up @@ -54,7 +54,7 @@ the browser marker update until the next explicit host capture or simulation
update.

Manager add methods mark topology dirty for rigid objects, rigid-object groups,
volume/surface deformables, robots, articulations, and `Camera` sensors. Supported
volume/surface deformables, cables, robots, articulations, and `Camera` sensors. Supported
`remove_asset()` branches do the same. The next simulation update refreshes the
browser automatically. Use `refresh_visualization()` when the refresh must
happen before another physics step. Code that changes mesh topology outside
Expand All @@ -80,13 +80,19 @@ fields already named `wxyz` remain protocol-native.

Arena offsets are added to rigid, robot, articulation, and camera poses.
Deformable vertices are stored relative to the corresponding arena node.
Cable `segment_pose_w` already contains world positions, including Arena
offsets; export must not add those offsets again.
Pose export gathers visible environment/component rows on the source device
before CPU conversion and quaternion normalization. Native state fetching remains
asset-owned; host conversion scales with the exported node count.

### Exported content

| Simulation content | Browser representation |
|---|---|
| `RigidObject` | Complete combined render mesh; multi-segment faces receive matching vertex offsets |
| `RigidObjectGroup` | One node and pose per constituent object |
| `CableObject` | One capsule node per segment, shared rest geometry, and live world poses |
| `Robot` | One mesh node per non-empty link |
| `Articulation` | One mesh node per non-empty link |
| `VolumeDeformableObject` | Live Newton render-surface vertices and triangles |
Expand All @@ -101,6 +107,13 @@ instead travel as `MeshMarkerOverlay` entries. Visual
materials and textures are not mirrored; meshes use category colors from
`SceneExporter._COLORS`.

Cables use capsule observer geometry generated from configured endpoint
distances and radius, centered along each segment's local +Z axis. Export does
not depend on native rendering resources and does not reproduce tube skins or
custom visual meshes. Prepared cable data is sampled once per cable per frame;
segment poses update at `scene_fps` through the static-mesh batch path, without
rebuilding geometry or using the deformable-vertex channel.

## Cameras

Only sensors whose `sensor_type` is exactly `"Camera"` are added to the
Expand Down
3 changes: 2 additions & 1 deletion agent_context/topics/sim-visualization/sim-visualization.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ this invariant.
| Symptom | Cause / action |
|---|---|
| `Visualization env_ids ... outside simulation range` | Selected IDs do not exist in the configured arenas. Validate them against `SimulationManager.num_envs`. |
| Startup timeout or address-in-use error | The Viser worker did not become ready or the configured port is occupied. Select another port and inspect `visualization_health.worker_error`. |
| Startup timeout, possibly just after Viser starts listening | The worker startup budget includes Viser frontend dependency installation/build and initial manifest publication. `VisualizationRuntime.start()` waits up to 120 seconds by default and returns as soon as ready; an unusually slow build may need a longer explicit `start(timeout=...)`. Inspect startup logs before retrying. |
| Address-in-use error | The configured port is occupied. Select another port and inspect `visualization_health.worker_error`. |
| Asset added after startup is missing | Step once, call `refresh_visualization()`, or mark topology dirty if the change bypassed manager APIs. |
| Native marker creation reports missing render-actor support | Use a DexSim build exposing Scene render-only mesh lifecycle and generic RenderBody/MaterialInst overlay properties, or run with `--viser`. Overlay routing supports Hybrid, FastRT and OfflineRT. |
| Newton Viser shows only the grid after declaring a robot | Check that Viser startup did not finalize an empty Spawn scene before asset declaration and that link poses remain finite after the first update. |
Expand Down
11 changes: 11 additions & 0 deletions embodichain/lab/gym/envs/embodied_env.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
VisualObjectCfg,
RigidObjectGroupCfg,
ArticulationCfg,
CableObjectCfg,
LightCfg,
)
from embodichain.lab.sim.objects import Robot
Expand Down Expand Up @@ -307,6 +308,8 @@ class EnvLightCfg:

articulation: List[ArticulationCfg] = []

cable_object: list[CableObjectCfg] = []
"""Fixed-topology cables declared before simulation preparation."""
fluid_object: list[FluidObjectCfg] = []
"""Fixed-count fluids owned by the physical environment component."""

Expand Down Expand Up @@ -3790,6 +3793,8 @@ def validate_resolved(cfg: RigidObjectCfg) -> None:
for cfg in self.cfg.rigid_object_group:
self.sim.add_rigid_object_group(cfg=cfg)

for cfg in self.cfg.cable_object:
self.sim.add_cable_object(cfg=cfg)
for cfg in self.cfg.fluid_object:
self.sim.add_fluid_object(cfg)

Expand Down Expand Up @@ -4297,6 +4302,12 @@ def save_trajectory(
self.get_demo_episode_metadata(int(env_id)) for env_id in env_ids
],
}
if "cables" in states.keys():
# Cable observations retain world coordinates. Keep each exported
# row's origin so action replay can validate another arena placement.
meta["arena_origins_w"] = (
self.sim.arena_offsets[env_ids_t].detach().cpu().tolist()
)
meta.update(self._trajectory_action_metadata())
final_observations = _export_final_observations(
getattr(self, "_final_acceptance_rules", ()),
Expand Down
38 changes: 31 additions & 7 deletions embodichain/lab/gym/envs/managers/observations.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,16 @@ def _state_descriptor(
names: tuple[str, ...],
) -> ObservationDescriptor:
# Only immutable schema values are cached; no environment or asset handle.
if quantity == "segment_state_w":
return ObservationDescriptor(
uid,
quantity,
"world",
"xyz_xyzw_com_lin_vel_ang_vel",
("m", "unitless", "m/s", "rad/s"),
shape,
dtype,
)
joint = quantity in ("qpos", "qvel")
pose = quantity in ("pose", "segment_start_pose")
return ObservationDescriptor(
Expand Down Expand Up @@ -132,15 +142,19 @@ def get_entity_state(
env: EmbodiedEnv,
obs: EnvObs,
entity_cfg: SceneEntityCfg,
quantity: Literal["pose", "qpos", "qvel", "lin_vel", "ang_vel"] = "pose",
quantity: Literal[
"pose", "qpos", "qvel", "lin_vel", "ang_vel", "segment_state_w"
] = "pose",
) -> ObservationSample:
"""Acquire a described original state channel for any observation consumer.

A capture shares the acquisition across aliases, recording and policy pose
functors. Missing entities remain invalid, even though policy views retain
the historical zero-fill behavior. Joint channels contain the complete
asset-defined order, including mimic joints; selectors are rejected rather
than silently describing a subset as the complete state.
than silently describing a subset as the complete state. Cable
``segment_state_w`` contains each segment's world position, XYZW quaternion,
COM linear velocity and angular velocity; it does not include solver history.

Args:
env: Environment owning the native scene.
Expand All @@ -151,7 +165,14 @@ def get_entity_state(
Returns:
Raw values and explicit row validity, before policy transformations.
"""
if quantity not in ("pose", "qpos", "qvel", "lin_vel", "ang_vel"):
if quantity not in (
"pose",
"qpos",
"qvel",
"lin_vel",
"ang_vel",
"segment_state_w",
):
raise ValueError(f"Unsupported raw state quantity: {quantity!r}.")
if entity_cfg.joint_names is not None or entity_cfg.joint_ids != slice(None):
raise ValueError("Raw entity state requires the complete joint order.")
Expand All @@ -169,11 +190,14 @@ def produce() -> ObservationSample:
joint = quantity in ("qpos", "qvel")
if obj is None:
width = 0 if joint else 7 if quantity == "pose" else 3
values = torch.zeros((env.num_envs, width), device=env.device)
shape = (0, 13) if quantity == "segment_state_w" else (width,)
values = torch.zeros((env.num_envs, *shape), device=env.device)
valid = torch.zeros(env.num_envs, dtype=torch.bool, device=env.device)
errors = (f"Entity {uid!r} is unavailable.",) * env.num_envs
else:
if quantity == "pose":
if quantity == "segment_state_w":
values = obj.data.segment_state_w
elif quantity == "pose":
values = obj.get_local_pose(to_matrix=False)
elif joint:
values = obj.get_qpos() if quantity == "qpos" else obj.get_qvel()
Expand All @@ -184,9 +208,9 @@ def produce() -> ObservationSample:
else:
values = getattr(obj.body_data, quantity)
valid = torch.isfinite(values).reshape(env.num_envs, -1).all(dim=-1)
if quantity == "pose":
if quantity in ("pose", "segment_state_w"):
valid &= (
(torch.linalg.vector_norm(values[..., 3:], dim=-1) > 0)
(torch.linalg.vector_norm(values[..., 3:7], dim=-1) > 0)
.reshape(env.num_envs, -1)
.all(dim=-1)
)
Expand Down
Loading
Loading