Repository navigation
feat(visualization): add robot markers and trajectory previews - #833
Conversation
[Medium impact] The changes reach robot visualization, camera previews, and recorded-trajectory playback. The replay example should keep accepting complete visual-link recordings before this merges.
|
Select marker prerequisites from #654 with source attribution. Fix batch count traversal and marker-only environment visibility. Add RobotVisual for externally supplied poses and recorded trajectory previews. Co-authored-by: yuecideng <dengyueci@qq.com>
e83a99f to
4fe72d8
Compare
Use the native camera category filter for scene markers and apply texture opacity through the native sampler. Use the scene pass for ray-traced offscreen marker previews.
yuecideng
left a comment
There was a problem hiding this comment.
Three actionable findings on the native robot marker and trajectory paths are attached inline.
| color = marker.color | ||
| texture_options = {} | ||
| if marker.texture is not None and not ( | ||
| scene_lighting and parameters.get("base_color_map") |
There was a problem hiding this comment.
[P2] Apply opaque override color to textured Hybrid markers
For RobotMarkersAppearanceCfg(mode="override", opacity=1), this branch retains the source base_color_map and only changes MaterialDesc.base_color later. EmbodiChain's Hybrid rendering path treats a base-color texture as replacing that material color factor (see randomization/visual.py), so a textured robot stays in its asset colors instead of taking the requested override. The rendering test covers opaque asset mode and translucent override mode, but not this combination. Please apply the tint through a texture or sampler mechanism that Hybrid actually uses, and cover the opaque textured case.
There was a problem hiding this comment.
Fixed in ca1dac7. Textured markers now apply the color factor through the texture sampler’s scale and bias, with a neutral material color to avoid double multiplication. The rendering regression covers opaque override mode and switching back to asset colors. Hybrid monocular/stereo and FastRT monocular checks passed.
| """Fit the whole clip once, then track XY at fixed height and distance.""" | ||
| anchors = np.zeros((len(poses), 3), dtype=np.float64) | ||
| if mode == "follow_xy": | ||
| anchors[:, :2] = poses[:, 0, 0, :2] |
There was a problem hiding this comment.
[P2] Resolve the root link by name for camera tracking
The NPZ schema accepts named links in any order, and the robot visuals are later reordered with names.index(name). Here poses[..., 0] is assumed to be the root; the velocity arrows make the same assumption below. If a recorded clip lists a moving end-effector first, follow_xy tracks that link and the arrows inherit its pose rather than the robot root. Please resolve the root from link_names, or explicitly require and validate root-first input.
There was a problem hiding this comment.
Fixed in ca1dac7, with the missing-root follow-up addressed in 9676548. Camera tracking, initial heading, and velocity arrows now resolve the root by name in the recorded link list. Fixed-view playback also accepts recordings containing only visual links. Reordered-link tests and native replay checks passed.
| if self._chain is None: | ||
| import pytorch_kinematics as pk | ||
|
|
||
| self._chain = pk.build_chain_from_urdf(self._urdf_path.read_bytes()) |
There was a problem hiding this comment.
[P3] Handle zero-joint URDFs in the joint-space API
A single-link URDF with visual geometry is accepted by RobotMarkers, but accessing joint_names or calling visualize_qpos() reaches pytorch_kinematics==0.10.0's build_chain_from_urdf(). In that version, root_link is assigned only while iterating joints, so a zero-joint URDF raises UnboundLocalError. Please handle the static-link case explicitly or reject it with a clear error at this boundary.
There was a problem hiding this comment.
Fixed in ca1dac7. A single-link URDF now gets an explicit static kinematic chain, so joint_names returns an empty tuple and visualize_qpos() accepts zero-width joint arrays. The regression passed with visual origins, root transforms, multiple reference instances, and partial environment updates.
han-xudong
left a comment
There was a problem hiding this comment.
Two P2 findings from the local review, both reproduced with CPU-only checks.
| if root_name not in names: | ||
| raise ValueError( | ||
| f"Recorded link_names must include the robot root {root_name!r}." | ||
| ) |
There was a problem hiding this comment.
Complete recordings fail to replay
_root_link_index() now rejects recordings that omit a root link with no visual geometry, even with --camera-mode fixed. These recordings can contain every link required by RobotMarkers.link_names and previously replayed correctly. Allow a recorded-link fallback for fixed views, or provide a separate camera anchor, rather than stopping an otherwise complete replay.
Prompt To Fix With AI
This is a comment left during a code review.
Path: examples/sim/reference_trajectory.py
Line: 87-90
Comment:
**Complete recordings fail to replay**
`_root_link_index()` now rejects recordings that omit a root link with no visual geometry, even with `--camera-mode fixed`. These recordings can contain every link required by `RobotMarkers.link_names` and previously replayed correctly. Allow a recorded-link fallback for fixed views, or provide a separate camera anchor, rather than stopping an otherwise complete replay.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 9676548. Fixed-view playback now accepts visual-only recordings without the root pose. If command or velocity fields are present, the example warns and omits the root-dependent arrows while continuing playback. follow_xy reports the missing root and suggests --camera-mode fixed. Regression tests and native replays with and without velocity fields passed.
There was a problem hiding this comment.
Greptile has paused reviews on this repository — it used its 750 free open-source review credits for this billing period. Reviews resume automatically on November 19. To continue before then, an organization admin can keep reviews running past the free credits — those bill as normal usage.
Description
Add
RobotMarkersto display robot poses and trajectory layers without creating physical articulations or changing simulated robot state. URDF/USD visuals retain their geometry, origins, textures and material parameters. Callers can preserve source appearance, override base color or adjust opacity.Changes
The branch uses the published
dexsim_engine==0.6.0rc1pinned by #1124. It includes the marker foundations needed from #654 with the original author attribution. ExistingVisualObjectsupport supplies task objects.DexSim 0.6.0rc1 compatibility and validation
Opaque markers use scene lighting and carry a dedicated render category. Default monocular and stereo sensors exclude that category. Set
include_visual_overlays=Trueon a preview sensor to include markers. Marker groups own their extracted textures until removal, and snapshot memory accounting includes normals and material UV coordinates.The Hybrid overlay uses the texture sampler scale to apply opacity. FastRT and OfflineRT use the filtered scene pass for offscreen overlays. The published wheel supplies the required APIs.
Diff against the current target commit · GitHub checks
Earlier review corrections
Commit
4fe72d87ecovers staged-texture ownership, failed-replacement cleanup, independently optional command and velocity arrows, window-open refusal, trajectory layout offsets, and shared-array accounting. These visualization findings were reported during #924 review.Earlier integration validation
Before #1124, the manual GitHub Actions run passed 9,403 fast tests and had one native-marker failure on a wheel without the required Scene and overlay interfaces. Local tests at
8af687b6used a matching development runtime.The local simulation run used the downloaded CI plan's
not slow and requires_sim and not gpuselection. Two asset tests passed after correcting a local dependency mismatch. The four selected skips are existing Pink/Newton coverage exclusions; no skip markers were added. GPU and slow cuRobo modules are outside this simulation selection.Earlier regression for unchanged behavior: 708 passed, 3 skipped. Both prerequisite fixes have failing reproductions and passing regressions. A CPU-only benchmark at 1024 environments × 1 marker reduced the full-update median from 32.4580 to 12.5741 ms across seven repetitions; frame rendering is excluded.
Visual examples
The comparison figures use historical captures from the integrated visualization runtime. They retain the observed appearance differences: green-arrow noise and translucent shading are still under separate rendering maintenance. Click either figure for full resolution.
Recorded Panda replay. Two trajectories are offset by 1.1 m with a fixed camera height. This recording used a matching local visualization runtime.
panda-offset.mp4
Additional recorded-policy examples: Go2 velocity tracking · G1 motion tracking · Panda overlapping trajectories. These historical replays use fixed camera height and do not re-evaluate policy success rates.
Type of change
Checklist
black .command to format the code base.python docs/scripts/check_api_docs.py), if applicable.