Skip to content

feat(visualization): add robot markers and trajectory previews - #833

Merged
acrlw merged 12 commits into
mainfrom
codex/reference-ghost-standalone
Nov 11, 2026
Merged

acrlw merged 12 commits into
mainfrom
codex/reference-ghost-standalone

Conversation

@acrlw

@acrlw acrlw commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add RobotMarkers to display robot poses and trajectory layers without creating physical articulations or changing simulated robot state. URDF/USD visuals retain their geometry, origins, textures and material parameters. Callers can preserve source appearance, override base color or adjust opacity.

Changes

  • Compose the existing marker system for multiple environments and robot layers, with link-pose updates, URDF joint-position input, visibility and cleanup.
  • Replay overlapping or spatially offset trajectories with task objects, paths and targets. Tracking cameras follow X/Y at a fixed height; fixed views are also supported.
  • Publish pending native transforms before each render, including saved-state playback without physics steps; use scene lighting for opaque robot markers.
  • Fix two marker prerequisites: reuse saved per-environment counts during batch updates, and preserve Viser visibility controls for marker-only environments.

The branch uses the published dexsim_engine==0.6.0rc1 pinned by #1124. It includes the marker foundations needed from #654 with the original author attribution. Existing VisualObject support supplies task objects.

DexSim 0.6.0rc1 compatibility and validation

Opaque markers use scene lighting and carry a dedicated render category. Default monocular and stereo sensors exclude that category. Set include_visual_overlays=True on a preview sensor to include markers. Marker groups own their extracted textures until removal, and snapshot memory accounting includes normals and material UV coordinates.

The Hybrid overlay uses the texture sampler scale to apply opacity. FastRT and OfflineRT use the filtered scene pass for offscreen overlays. The published wheel supplies the required APIs.

  • 660 focused CPU tests passed on Python 3.12 with the published wheel.
  • Six Hybrid and two FastRT native tests passed. Coverage includes opaque and translucent markers, texture and material changes, removal, and isolation in monocular and stereo sensors.
  • OfflineRT monocular and stereo checks passed with the metallic sample. In the same lighting setup, the nonmetallic lit sample did not produce the expected color in a native-only reproduction.
  • Black 26.10.1, API documentation coverage (4452/4452), and the agent context check passed.

Diff against the current target commit · GitHub checks

Earlier review corrections

Commit 4fe72d87e covers staged-texture ownership, failed-replacement cleanup, independently optional command and velocity arrows, window-open refusal, trajectory layout offsets, and shared-array accounting. These visualization findings were reported during #924 review.

Earlier integration validation

Before #1124, the manual GitHub Actions run passed 9,403 fast tests and had one native-marker failure on a wheel without the required Scene and overlay interfaces. Local tests at 8af687b6 used a matching development runtime.

Check Result
Simulation manager, visualization and example CLI regression 564 passed, 2 skipped with the matching runtime
Published-runtime compatibility 136 passed; 4 USD-material checks skipped because the required runtime interfaces are unavailable
Native appearance and lifecycle 5 passed, including the texture/opacity/material-switch test that failed on the published runtime
Simulation group with the matching local runtime 373 passed, 4 existing skips across all 377 selected tests; completed in batches after a VRAM guard stop
Overlay camera and ray-query isolation 9 historical overlay-state combinations passed on the overlay path
G1 example 4 environments × 4 layers; 544 visuals; 30 updates; zero physics steps
Panda trajectory example 2 offset trajectories; 30 frames at 15 FPS; fixed camera Z; zero physics steps
Formatting and public API documentation 1497 files passed; 2807/2807 API entries covered
Sphinx dummy build Passed with 836 existing warnings

The local simulation run used the downloaded CI plan's not slow and requires_sim and not gpu selection. Two asset tests passed after correcting a local dependency mismatch. The four selected skips are existing Pink/Newton coverage exclusions; no skip markers were added. GPU and slow cuRobo modules are outside this simulation selection.

Earlier regression for unchanged behavior: 708 passed, 3 skipped. Both prerequisite fixes have failing reproductions and passing regressions. A CPU-only benchmark at 1024 environments × 1 marker reduced the full-update median from 32.4580 to 12.5741 ms across seven repetitions; frame rendering is excluded.

Visual examples

The comparison figures use historical captures from the integrated visualization runtime. They retain the observed appearance differences: green-arrow noise and translucent shading are still under separate rendering maintenance. Click either figure for full resolution.

Marker geometry and orientation comparison

Robot appearance and reference-layer comparison

Recorded Panda replay. Two trajectories are offset by 1.1 m with a fixed camera height. This recording used a matching local visualization runtime.

panda-offset.mp4

Additional recorded-policy examples: Go2 velocity tracking · G1 motion tracking · Panda overlapping trajectories. These historical replays use fixed camera height and do not re-evaluate policy success rates.

Type of change

  • New feature
  • Bug fix
  • Documentation update

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
  • Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py), if applicable.
  • I have added tests that prove my fix is effective or that my feature works.
  • The branch uses the published runtime pinned by Adapt EmbodiChain to DexSim 0.6.0rc1 on Python 3.12 #1124.

@acrlw acrlw added docs Documentation, API references and example explanations. enhancement Improvements to existing capabilities; use feature for new capabilities. visualization Browser viewers, markers, trajectory previews and interactive debugging displays. robot Robot runtime, presets, control groups, calibration and robot support. labels Oct 22, 2026
@acrlw
acrlw marked this pull request as ready for review October 22, 2026 07:38
@greptile-apps

greptile-apps Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes reach robot visualization, camera previews, and recorded-trajectory playback.

The replay example should keep accepting complete visual-link recordings before this merges.

Fix All in CodexFindings

  1. P1 Complete recordings fail to replay ▶
Fix with agent prompt
### Issue 1
examples/sim/reference_trajectory.py:87-90
`_root_link_index()` now rejects recordings that omit a root link with no visual geometry, even with `--camera-mode fixed`. These recordings can contain every link required by `RobotMarkers.link_names` and previously replayed correctly. Allow a recorded-link fallback for fixed views, or provide a separate camera anchor, rather than stopping an otherwise complete replay.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds render-only robot markers and recorded trajectory previews.

  • Marker groups show reusable shapes and meshes across environments without changing physics.
  • Robot markers display robot visuals from supplied link poses or joint positions.
  • Recorded robot trajectories can be replayed as offset visual layers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[URDF or USD visuals] --> B[RobotMarkers]
    P[Link poses or joint positions] --> B
    B --> G[MarkerGroup]
    G --> N[Native scene visuals]
    G --> S[Detached mesh snapshots]
    S --> V[Viser]
    N --> C[Preview camera with markers enabled]
Loading

Reviews (6) · Last reviewed commit: "fix(visualization): address robot marker..." · Reviewed by Greptile

Comment thread examples/sim/robot_markers.py Resolved
Comment thread examples/sim/reference_trajectory.py Resolved
Comment thread examples/sim/reference_trajectory.py Resolved
@yuecideng yuecideng added feature New user-facing capabilities. bug Incorrect existing behavior or a fix restoring the intended behavior. rendering Image generation, lighting, material rendering, native renderers and image quality. and removed enhancement Improvements to existing capabilities; use feature for new capabilities. labels Oct 25, 2026
@acrlw
acrlw requested a review from yuecideng October 26, 2026 01:16
@acrlw
acrlw force-pushed the codex/reference-ghost-standalone branch from e83a99f to 4fe72d8 Compare November 3, 2026 10:50
Use the native camera category filter for scene markers and apply texture opacity through the native sampler. Use the scene pass for ray-traced offscreen marker previews.
@yuecideng
yuecideng requested a review from han-xudong November 9, 2026 04:50

@yuecideng yuecideng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three actionable findings on the native robot marker and trajectory paths are attached inline.

color = marker.color
texture_options = {}
if marker.texture is not None and not (
scene_lighting and parameters.get("base_color_map")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply opaque override color to textured Hybrid markers

For RobotMarkersAppearanceCfg(mode="override", opacity=1), this branch retains the source base_color_map and only changes MaterialDesc.base_color later. EmbodiChain's Hybrid rendering path treats a base-color texture as replacing that material color factor (see randomization/visual.py), so a textured robot stays in its asset colors instead of taking the requested override. The rendering test covers opaque asset mode and translucent override mode, but not this combination. Please apply the tint through a texture or sampler mechanism that Hybrid actually uses, and cover the opaque textured case.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ca1dac7. Textured markers now apply the color factor through the texture sampler’s scale and bias, with a neutral material color to avoid double multiplication. The rendering regression covers opaque override mode and switching back to asset colors. Hybrid monocular/stereo and FastRT monocular checks passed.

Comment thread examples/sim/reference_trajectory.py Outdated
"""Fit the whole clip once, then track XY at fixed height and distance."""
anchors = np.zeros((len(poses), 3), dtype=np.float64)
if mode == "follow_xy":
anchors[:, :2] = poses[:, 0, 0, :2]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Resolve the root link by name for camera tracking

The NPZ schema accepts named links in any order, and the robot visuals are later reordered with names.index(name). Here poses[..., 0] is assumed to be the root; the velocity arrows make the same assumption below. If a recorded clip lists a moving end-effector first, follow_xy tracks that link and the arrows inherit its pose rather than the robot root. Please resolve the root from link_names, or explicitly require and validate root-first input.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ca1dac7, with the missing-root follow-up addressed in 9676548. Camera tracking, initial heading, and velocity arrows now resolve the root by name in the recorded link list. Fixed-view playback also accepts recordings containing only visual links. Reordered-link tests and native replay checks passed.

if self._chain is None:
import pytorch_kinematics as pk

self._chain = pk.build_chain_from_urdf(self._urdf_path.read_bytes())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Handle zero-joint URDFs in the joint-space API

A single-link URDF with visual geometry is accepted by RobotMarkers, but accessing joint_names or calling visualize_qpos() reaches pytorch_kinematics==0.10.0's build_chain_from_urdf(). In that version, root_link is assigned only while iterating joints, so a zero-joint URDF raises UnboundLocalError. Please handle the static-link case explicitly or reject it with a clear error at this boundary.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ca1dac7. A single-link URDF now gets an explicit static kinematic chain, so joint_names returns an empty tuple and visualize_qpos() accepts zero-width joint arrays. The regression passed with visual origins, root transforms, multiple reference instances, and partial environment updates.

@han-xudong han-xudong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two P2 findings from the local review, both reproduced with CPU-only checks.

Comment thread examples/sim/reference_trajectory.py Outdated
Comment on lines +87 to +90
if root_name not in names:
raise ValueError(
f"Recorded link_names must include the robot root {root_name!r}."
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Complete recordings fail to replay

_root_link_index() now rejects recordings that omit a root link with no visual geometry, even with --camera-mode fixed. These recordings can contain every link required by RobotMarkers.link_names and previously replayed correctly. Allow a recorded-link fallback for fixed views, or provide a separate camera anchor, rather than stopping an otherwise complete replay.

Prompt To Fix With AI
This is a comment left during a code review.
Path: examples/sim/reference_trajectory.py
Line: 87-90

Comment:
**Complete recordings fail to replay**

`_root_link_index()` now rejects recordings that omit a root link with no visual geometry, even with `--camera-mode fixed`. These recordings can contain every link required by `RobotMarkers.link_names` and previously replayed correctly. Allow a recorded-link fallback for fixed views, or provide a separate camera anchor, rather than stopping an otherwise complete replay.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9676548. Fixed-view playback now accepts visual-only recordings without the root pose. If command or velocity fields are present, the example warns and omits the root-dependent arrows while continuing playback. follow_xy reports the missing root and suggests --camera-mode fixed. Regression tests and native replays with and without velocity fields passed.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile has paused reviews on this repository — it used its 750 free open-source review credits for this billing period. Reviews resume automatically on November 19. To continue before then, an organization admin can keep reviews running past the free credits — those bill as normal usage.

@acrlw
acrlw merged commit 2f53b07 into main Nov 11, 2026
8 checks passed
@acrlw
acrlw deleted the codex/reference-ghost-standalone branch November 11, 2026 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Incorrect existing behavior or a fix restoring the intended behavior. docs Documentation, API references and example explanations. feature New user-facing capabilities. rendering Image generation, lighting, material rendering, native renderers and image quality. robot Robot runtime, presets, control groups, calibration and robot support. visualization Browser viewers, markers, trajectory previews and interactive debugging displays.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants