Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions agent_context/MAP.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -181,9 +181,10 @@ topics:
status: active
- id: sim-visualization
title: Simulation Visualization
aliases: [browser visualization, viser, sim visualization, 仿真可视化]
aliases: [browser visualization, viser, sim visualization, marker visualization, 标志物可视化, 仿真可视化]
keywords: [VisualizationCfg, ViserServerCfg, SceneManifest, visualization, EntityGizmoManipulator, robot_ik_gizmo,
GizmoCfg, backpressure, capture_visualization, Newton deformable, particle set, render topology, quaternion, xyzw]
GizmoCfg, MarkerGroup, MarkerPrototypeCfg, RobotMarkers, VisualizationMarkers, robot markers, ghost robot, 机器人可视化, backpressure, capture_visualization, Newton deformable, particle set,
render topology, quaternion, xyzw]
paths: [topics/sim-visualization/sim-visualization.md]
source_of_truth:
- embodichain/lab/visualization/protocol.py
Expand Down
2 changes: 2 additions & 0 deletions agent_context/topics/sensor-system/sensor-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,5 +134,7 @@ quota are integration contracts. Generic image tiling belongs to
| Backend contact behavior | `tests/sim/sensors/test_contact_default_e2e.py`, `test_contact_newton_e2e.py` in that directory |

For missing camera output, inspect enabled fields and `ViewFlags` together.
Camera and stereo images exclude visual markers by default. Set
`include_visual_overlays=True` for an explicit preview camera.
For missing contacts, inspect backend capabilities, the query filter and quota
before changing buffer allocation. Pure resolver tests do not require a renderer.
49 changes: 44 additions & 5 deletions agent_context/topics/sim-visualization/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,44 @@ Read this when the request needs these details. [Topic overview](sim-visualizati
| `destroy()` | Stops visualization before queuing deferred simulation cleanup |

During explicit `SimulationManager.update()`, each physics step increments the
visualization step/time counters and attempts a rate-limited capture. A topology
revision mismatch publishes a fresh manifest before its first matching frame.
visualization step/time counters and refreshes marker attachments after the
optional `after_substep` observer. `render_frame()` shares state publication
across camera observations, recording and rate-limited Viser capture. With
`render_final_step=False`, the final capture waits for that frame; state writes
between the last substep and the frame are included when render state is
published, including refreshed marker poses. A topology revision mismatch
publishes a fresh manifest before its first matching frame.
`BaseEnv.reset()` also requests a forced capture after resetting scene state.
Drawing markers and capturing visualization do not advance physics; interactive
loops call `SimulationManager.update(step=1)` to process Gizmos and step the world.

Marker groups default to `scope="env"` and inherit `SimulationManager.num_envs`.
`scope="world"` creates one global batch. The new group config has no
`arena_index`. Updates use `(E,M,3)` positions/scales, `(E,M,4)` xyzw/RGBA,
and `(E,M)` type/visibility arrays; `env_ids` selects and orders the E dimension.
One selected environment accepts the `(M,...)` shorthand. Counts may differ
between environments after selected updates; `count` is the total and `counts`
contains one value per environment. Selected clear/hide/update preserves others.

`attach(parent_uid, link_name=..., env_ids=...)` resolves prepared registered
rigid roots or articulation/robot links through public pose APIs. Stored marker
poses become parent-relative offsets. The manager refreshes these transforms
on host steps and explicit render-state synchronization; snapshots stay detached
and do not read physics. `detach(keep_world_pose=True)` preserves the displayed
pose by default. World-scope attachment is rejected rather than guessing a
source environment. Parent resolution must not retain borrowed native nodes.

Marker groups publish a complete detached snapshot after `update()`,
`set_visibility()`, `clear()`, or `remove()`. Native groups update DexSim debug
meshes immediately. Browser groups are included as mesh overlays and may force
a frame only when the currently published manifest matches the manager's
topology revision. Marker publication never calls `prepare()` or synchronizes
physics. An unpublished host substep defers browser marker capture to
`render_frame()`, after the observer and all attachment refreshes. Paused edits
still publish immediately when no frame is pending. A dirty topology defers
the browser marker update until the next explicit host capture or simulation
update.

Manager add methods mark topology dirty for rigid objects, rigid-object groups,
volume/surface deformables, robots, articulations, and `Camera` sensors. Supported
`remove_asset()` branches do the same. The next simulation update refreshes the
Expand Down Expand Up @@ -61,10 +93,11 @@ Deformable vertices are stored relative to the corresponding arena node.
| `SurfaceDeformableObject` | Live Newton render-surface vertices and triangles |
| `Camera` | Frustum plus optional low-frequency RGB preview |
| Default ground | 1000 m × 1000 m XY grid, 1 m cells, 10 m sections |
| `SceneOverlays` | Frames, targets, trajectories, and point clouds |
| `SceneOverlays` | Frames, targets, trajectories, point clouds, and marker meshes |

Lights, rigid constraints, markers/gizmos, contact sensors, and stereo cameras
are not exported as scene nodes by the current `SceneExporter`. Visual
are not exported as scene nodes by the current `SceneExporter`. Marker groups
instead travel as `MeshMarkerOverlay` entries. Visual
materials and textures are not mirrored; meshes use category colors from
`SceneExporter._COLORS`.

Expand Down Expand Up @@ -151,7 +184,13 @@ GUI callbacks enqueue events; the visualization worker applies them, preserving
the single-threaded Viser-handle invariant. Environment visibility affects
static batches, deformables, and camera frustums.

Overlays are backend-neutral protocol objects. Point clouds larger than
Overlays are backend-neutral protocol objects. Marker groups use triangle-mesh
overlays with per-instance RGBA, pose, scale, and visibility. Built-in
prototypes are box, sphere, cylinder, capsule, cone, arrow, and frame; custom
triangle meshes are also supported. This does not add GPU instancing, and the
existing point-cloud and polyline overlay paths remain independent.

Point clouds larger than
`point_cloud_max_points` are deterministically subsampled. The ordinary
`SimulationManager.capture_visualization()` helper currently supplies no
overlay argument; callers needing overlays must use the active
Expand Down
60 changes: 57 additions & 3 deletions agent_context/topics/sim-visualization/sim-visualization.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> Topic: Simulation visualization — `SimulationManager` lifecycle,
> backend-neutral scene snapshots, the Viser backend, cameras, deformables,
> interactive Gizmos, and launcher integration.
> interactive Gizmos, render-only marker groups, and launcher integration.

---

Expand All @@ -18,6 +18,7 @@
| `embodichain/lab/visualization/runtime.py` | Background worker, latest-frame queues, rate limiting, health, and telemetry |
| `embodichain/lab/visualization/backends/base.py` | Visualization backend contract |
| `embodichain/lab/visualization/backends/viser.py` | Viser server, scene handles, GUI controls, and browser publishing |
| `embodichain/lab/visualization/markers/` | Marker prototype/group configuration, geometry generation, validation, and lifecycle |
| `embodichain/lab/gym/utils/gym_utils.py` | Gym config parsing, common launcher arguments, and CLI overrides |
| `embodichain/lab/gym/envs/base_env.py` | Environment scene setup, post-setup start, and forced reset capture |
| `embodichain/lab/scripts/preview_asset.py` | Standalone rigid-object and articulation browser preview |
Expand Down Expand Up @@ -55,14 +56,49 @@ The simulation owns physics and assets. Scene export is read-only. When
`allow_commands=True`, Viser Gizmo callbacks enqueue pose commands that
`SimulationManager` later applies on the simulation thread.

`SimulationManager.add_marker_group()` owns named render-only marker groups.
`MarkerPrototypeCfg` supplies box, sphere, cylinder, capsule, cone, arrow,
frame, or caller-provided mesh geometry; `MarkerGroup.update()` publishes
batched pose, scale, prototype, color, and visibility arrays without stepping
physics. Groups default to all sub-environments; `env_ids` selects mutations,
while `scope="world"` creates one global batch. Attach/detach follows prepared
registered roots or links through public pose reads on the simulation thread.
During `SimulationManager.update()`, attachment refresh follows the optional
`after_substep` observer and precedes recording and visualization capture, so
captures include pose changes made by that observer.
Native markers use Scene-owned pure visual objects. Creation, cloning, batched
transforms and removal go through the existing scene facade. Failed updates
preserve previously published marker state. Marker operations do not prepare or
step physics; handles are invalidated when the owning scene closes. Browser
rendering carries detached mesh snapshots in `SceneOverlays`.

`VisualizationMarkers` adapts flat Isaac-style arrays (xyzw) with explicit
`environment_ids`. It defaults to world coordinates and environment 0.
`RobotMarkers` reads URDF/USD visuals and composes externally supplied link poses
with visual origins. URDF FK accepts all `joint_names`, including caller-resolved
mimic joints. Multiple poses can be displayed in one environment. Tasks own
motion loading, sampling and alignment.

`RobotMarkersAppearanceCfg` selects source appearance or a base-color override,
with adjustable opacity. Source textures and material parameters are preserved;
opaque visuals use scene lighting. Transparent shading may differ by renderer.
`unlit=True` selects a uniform color. Offscreen cameras exclude visuals until
explicitly enabled. File-backed and extracted texture resources remain available
until removal.

Batch updates read each selected environment's count directly. Viser combines
ordinary scene and marker environment IDs when creating visibility controls;
removing the last ordinary object preserves the environment's hidden state
while markers remain. The controls are rebuilt only when membership changes.

`SimulationManager` may start the Viser server during its own construction,
before a standalone caller has declared any assets. That empty visualization
startup must not call `prepare()` or finalize the Spawn scene. The caller's
first asset-bearing `prepare()` remains the initial topology commit; the next
capture observes the topology revision and republishes the manifest.

All DexSim and Torch reads happen in `SceneExporter` on the simulation thread.
Protocol objects make detached CPU copies before crossing into the runtime.
Protocol objects detach mutable poses and retain shared immutable marker geometry before crossing into the runtime.
Viser creation and every Viser handle mutation must happen on the single
visualization worker thread; `ViserBackend._assert_update_thread()` enforces
this invariant.
Expand All @@ -79,7 +115,7 @@ this invariant.
- Gizmo commands mutate targets only after simulation-thread validation.
- Simulation/DexSim reads stay on the simulation thread.
- Viser server and handle operations stay on one private worker thread.
- Cross-thread snapshots own detached contiguous CPU arrays.
- Cross-thread snapshots detach mutable state; marker geometry arrays may share validated read-only storage.
- A frame is accepted only when `run_id`, `scene_revision`, node IDs, and
dynamic vertex shapes match the published manifest.
- Topology changes require a new manifest; pose-only changes require only a
Expand All @@ -90,6 +126,9 @@ this invariant.
and offsets face indices into the concatenated vertex array. Links without
render meshes export empty geometry without querying mesh zero.
- Sampling rates are wall-clock limits, not simulation-time guarantees.
- Marker mutation never prepares or synchronizes an unprepared scene. When
topology is dirty, browser publication waits for the next explicit host
capture or `SimulationManager.update()`.
- Bind to loopback by default. Viser has no EmbodiChain authentication layer;
use SSH forwarding or an authenticated gateway for remote access.

Expand All @@ -100,6 +139,7 @@ this invariant.
| `Visualization env_ids ... outside simulation range` | Selected IDs do not exist in the configured arenas. Validate them against `SimulationManager.num_envs`. |
| Startup timeout or address-in-use error | The Viser worker did not become ready or the configured port is occupied. Select another port and inspect `visualization_health.worker_error`. |
| Asset added after startup is missing | Step once, call `refresh_visualization()`, or mark topology dirty if the change bypassed manager APIs. |
| Native marker creation reports missing render-actor support | Use a DexSim build exposing Scene render-only mesh lifecycle and generic RenderBody/MaterialInst overlay properties, or run with `--viser`. Overlay routing supports Hybrid, FastRT and OfflineRT. |
| Newton Viser shows only the grid after declaring a robot | Check that Viser startup did not finalize an empty Spawn scene before asset declaration and that link poses remain finite after the first update. |
| A Newton articulation link is missing geometry or emits `mesh_id 0 out of range` | Export all render-body mesh segments and treat a zero-mesh link as empty geometry. Do not use the single-mesh articulation helper for Newton render bodies. |
| Browser stops updating after an exporter/backend exception | `capture_visualization_safely()` latches the first error to protect simulation. Inspect health/logs, then stop and restart after fixing the cause. |
Expand All @@ -120,8 +160,22 @@ Relevant tests:
- `tests/visualization/test_scene_exporter.py`
- `tests/visualization/test_runtime.py`
- `tests/visualization/test_viser_backend.py`
- `tests/visualization/test_markers.py`
- `tests/visualization/test_native_markers.py`
- `tests/sim/test_sim_manager.py`
- `tests/sim/test_marker_rendering.py`
- `tests/gym/utils/test_gym_utils.py`
- `tests/lab/scripts/test_preview_asset.py`
- `tests/lab/scripts/test_preview_robot.py`
- `tests/lab/scripts/test_preview_joint_control.py`


Recorded task visualization uses `examples/sim/reference_trajectory.py`: named
world link poses have axes `(T,K,L,7)`, while task objects use `VisualObject`
and paths/targets use `VisualizationMarkers`. Offset layouts translate the
robot, object, target and path together. Policy execution stays task-owned.
`RobotMarkersCfg.source` accepts the same `UsdSourceCfg` selector as scene
assets. USD singleton selection and USDZ texture extraction are shared through
`sim/spawn/_usd_assets.py`; ghost-owned extraction directories survive until
removal. Scaled marker transforms use the Scene visual batch, while rigid
VisualObject poses use the World tensor batch.
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ Camera
:show-inheritance:
:exclude-members: __init__, copy, replace, to_dict, validate

Camera and stereo captures exclude native markers by default. Set
``include_visual_overlays=True`` on an explicit preview camera to include
both scene-lit markers and overlays.

Attachment resolution
---------------------

Expand Down
Loading
Loading