Skip to content

Pin the reference trajectory and add a UR5 expansion replay tutorial - #1244

Merged
yuecideng merged 58 commits into
mainfrom
cj/expansion-workloads-visualization
Nov 19, 2026
Merged

yuecideng merged 58 commits into
mainfrom
cj/expansion-workloads-visualization

Conversation

@matafela

@matafela matafela commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Keep the original recording visible while comparing how different layouts and grasps change a UR5 pick-and-place motion. This PR builds on #1192, which has merged into main.

What changed in ghost replay

On the target branch, clicking a candidate made that candidate the solid model. The reference and the other candidates had separate group visibility controls.

  • The robot and moving objects from --reference now stay solid throughout playback. If the user omits --reference, the first input recording supplies the solid model.
  • Augmented trajectories start hidden. Each candidate button independently toggles its translucent robot and objects. Clicking the same button again hides just that overlay. Several buttons can stay highlighted together.
  • The space, affordance and trajectory filters determine which enabled ghosts are visible. The viewer remembers individual selections across filter changes. The group checkbox shows or hides all current matches. The reference continues playing even when the filters match zero candidates.
  • With the replay focused, Space plays or pauses. Pressing Space at the end restarts playback.

Bilingual WebRTC replay

embodichain expansion preview now opens the recorded replay over the native WebRTC viewport from #954. The language selector updates the replay controls, status messages, deviation curves and object heatmap in Simplified Chinese or English. Authenticated commands affect only recorded playback and the observer camera. Video and replay metadata travel separately.

The preview uses a single browser transport. The former JPEG snapshot routes, image readback and encoding, and transport flags are removed.

  • Ran the UR5 replay with one reference, 16 candidates and two visible overlays. Chromium decoded 1100 × 760 H.264. Language, filters, seeking and camera controls passed. Physics stayed at zero steps, and closing the replay released the native listener.
  • On the same RTX PRO 5000 and loopback connection, median presentation rose from 6.6 to 24.6 fps. Median seek-to-visible-frame latency was 98.7 ms for JPEG and 97.6 ms for WebRTC.
  • The replay page, stylesheet, scripts and offline distribution template are packaged in embodichain.lab.expansion._frontend. The installed wheel served the bilingual UR5 WebRTC replay and generated an offline distribution report. 99 focused and context tests passed. Black and API documentation checks passed.

Scene-first collision and layout screening

  • The expansion prefilter previously stopped at 512 physical shapes. The isolated Newton query now keeps decomposed colliders and budgets up to 1,048,576 eligible pairs per scene after structural exclusions. Batched environments run in chunks that fit the pair and contact buffers. A detected collision identifies both original physical shapes. A full buffer leaves the check unverified.
  • All operators that use a layout screen share the same physical collision geometry. Existing primitive and convex shapes keep their native geometry. Rigid-object and robot-link triangle meshes are converted in collider-local coordinates. Closed convex meshes use one hull, volumetric concavities use CPU V-HACD, and planar surfaces use thin convex prisms that preserve openings. The query caches parts by mesh contents and applies each collider's native pose. Physical IDs still identify contact permissions and failures. The report records part counts in scene-first-checks.json under mesh_convex_parts. Physical execution and measured qualification use the original colliders.
  • A bounded XY translation of a rigid leaf can use the contact surfaces under its authored pose even when its body overhangs the largest inscribed rectangle. The sampler verifies contact around the projected centroid throughout the admitted range and uses the full body bounds for clearance against neighbors. This handles separate shelf slots and stationary neighbors on the same named support.
  • The feasibility planner accepts the binary integer validity masks returned by analytic IK solvers such as OPW and checks each valid branch against the existing joint limits and forward-kinematics tolerances.

Eight-environment verification

The downstream CLI ran these profiles using the installed EmbodiChain wheel at 6e53943bd934f0c0555f800ce3514c8e536f837c and DexSim 0.6.0 in Python 3.12. The visualization branch includes main at b715f35d2.

Deployment Object pose Affordance Joint residual Standard five-operator preset
CobotMagic bottle relay 3 / 3 3 / 12 3 / 3 5 / 5
Piper X book pickup 3 / 3 3 / 11 3 / 5 5 / 5
Tianji dual-arm marker insertion 3 / 3 3 / 12 3 / 3 5 / 5

In the three scene-first columns, each cell gives accepted episodes / candidate attempts. Each task ran in one World with eight environments. All 27 accepted episodes passed the task, event, terminal-state and native path-contact checks. The CLI exported them to LeRobot and read them back. The saved-run integrity checks verified 1,672 artifacts. Each browser served the recorded frames, deviation curves and three initial object-pose samples with zero physics steps.

The standard preset column gives accepted / requested episodes. Each standard preset collected one episode per operator, including via points and null-space residuals, for a separate total of 15 accepted LeRobot readbacks. Direct reads of the Parquet files verified 56,087 frames across both presets, with contiguous indices, increasing timestamps and finite measured joint states and actions.

The bundled workspace pick-and-place integration test also recorded eight accepted episodes for each of scene_first, robot_first and bidirectional.

Run the downstream scene-first examples

Start in the DexEChain checkout containing the three task profiles. Prepare the task assets listed in its Sim2Real guide and use a package source that provides DexSim 0.6.0. The downstream simulation extra and runtime check share the exact upstream revision. If an older shell session exported a revision override, clear it with unset DEXECHAIN_SIM2REAL_EMBODICHAIN_REVISION.

conda activate py312
python -m pip install -c extensions/dexechain_sim2real/constraints-py312.txt \
  -e 'extensions/dexechain_sim2real[sim]'

tasks=extensions/dexechain_sim2real/src/dexechain_sim2real/configs/tasks
collect_scene_first() {
  dexechain-sim2real expand --example "$1-coverage-expansion" \
    --expansion-config "$tasks/$2/expansion.scene_first.yaml" \
    --output "outputs/scene_first/$1" --timeout-s 7200 --no-headless
}

collect_scene_first bottle-relay cobotmagic_bottle_relay
collect_scene_first piper piper_x_take_book
collect_scene_first tianji-bimanual tianji_bimanual_marker_in_holder

Each command opens the replay after exporting nine episodes. The first recording supplies the solid comparison. To supply a separately recorded original, use the tutorial below. The Piper profile samples translations up to ±2 mm along the bookshelf X and Y axes and checks a 2.8 s native two-finger hold.

Piper trajectory perturbation defaults

The downstream Piper presets use trajectory.joint_offset_scale: 0.006, twenty times the previous 0.0003. The native joint_residual, via_points and nullspace_residual operators share this parameter. It sets the sampling amplitude to 0.6% of each controlled joint’s travel. The active operator list still comes from each preset.

In the amplitude sweep at 6530d9cd0 with DexSim 0.6.0rc2, each setting used the same three book layouts, candidate seed 2 and nominal seed 1234. Each perturbed rollout had the same starting state and original right-arm approach commands as its unperturbed control. TCP poses were calculated from measured joint positions and the configured tool transform.

Joint offset scale Planned joint peak (°) Measured joint peak (°) Measured TCP translation peak (mm) Measured TCP rotation peak (°)
0.0003 (previous default) 0.062 0.062 0.506 0.108
0.003 0.622 0.620 5.061 1.084
0.006 (new default) 1.244 1.240 10.116 2.168
0.03 6.221 6.198 50.289 10.812

Each cell is the mean of three per-episode peaks during the 4.52 s pick_up.approach phase. The joint metric takes the largest absolute deviation among the six right-arm joints at each frame. All 12 perturbed recordings and three controls passed physical acceptance and LeRobot readback. The standard five-operator Piper preset also recorded one accepted episode per operator, including a null-space trajectory within the configured TCP translation and rotation tolerances.

Run the UR5 example

Run these commands from the repository root with EmbodiChain, DexSim and LeRobot installed in the py312 environment. Use a fresh output directory for collection. If recordings already exist, skip the three collection commands and point --reference at the job directory containing the original episode.

conda activate py312
demo=scripts/tutorials/expansion/ur5_pick_place
out=outputs/ur5-affordance-demo

python "$demo/collect.py" reference   --output "$out"
python "$demo/collect.py" scene_first --output "$out"
python "$demo/collect.py" robot_first --output "$out"

python -m embodichain expansion preview \
  --reference "$out/reference/collection/dataset/object_pose" \
  --input "$out/scene_first/collection" \
  --input "$out/robot_first/collection" \
  --port 0 --open

scene_first proposes tabletop layouts and checks that the arm can execute them. robot_first starts from reachable arm poses and derives compatible layouts. Each strategy collects four fixed-grasp object_pose episodes and four affordance episodes for the pick, yielding 16 augmentations alongside one original. The tutorial defaults to seed 7. Tutorial README · Expansion configuration

Type of change

  • Enhancement to expansion replay with a runnable tutorial.
  • Bug fix for collision and IK screening of decomposed robots and mesh-based scenes.

Validation

  • After merging main, 174 focused tests passed with GPU tests enabled, including task discovery from an installed downstream package, wheel resources, contact sensing, expansion screening and replay. API documentation coverage was 5,132/5,132. The task catalog uses the existing environment-component path resolver, so the documented configuration contract still applies.

  • 165 focused scene expansion, feasibility and simulation-adapter tests passed with GPU tests enabled. Another 18 acceptance and layout-collection tests and 42 project-context tests passed. The regression cases check an open passage in a volumetric mesh, physical-ID permissions across parts in scalar and batched checks, bounded shelf overhang, rejection over a gap and persisted layout validation. The 514-shape test exercises 65,792 eligible native pairs and selected-row ordering across chunks.

  • Sixteen probe points checked the two occupied Piper shelf slots, 10 to 80 mm above each book base. The previous bounding box covered all 16. The original triangle surface and the new convex parts both classified all 16 as free. A separate planar annulus retained its open center and material on both sides.

  • The installed wheel at 6e53943bd contains both the replay JavaScript files and the bundled web assets from main. The downstream wheel contains all three scene-first profiles, with byte-identical source and packaged YAML. The integrity checks and browser readbacks are reported above.

  • On an AMD Ryzen 9 9950X with one CPU thread and Newton 1.7.0, the median of 20 warmed scalar queries was 0.49 ms at 33 shapes on both revisions. The 514-shape case completed in 2.89 ms with about 28 MiB of additional process peak RSS, using a 64-contact buffer. The previous revision rejected that scene at construction.

  • 22 preview, preview-job and measured comparison tests passed, along with 42 project-context tests. Heatmap assertions cover unequal X and Y spans, sub-millimeter variation, constant positions, and counts in the outermost bins.

  • Verified all 2,048 displayed cell counts against the eight original object_pose episode files. Desktop hover and mobile taps read back counts at the edges. Tested object selection, ghost filtering, the affordance-only empty state and distinct tick labels at narrow ranges. Native replay reported zero physics steps.

  • Built the wheel from f3bcfb24e and verified the packaged viewer HTML, both JavaScript assets and the Python heatmap and loader modules against the source files.

Checklist

  • Ran black . with Black 26.10.1.
  • Updated the tutorial, the preview guide and the expansion-engine context. The context map check passed.
  • Ran python docs/scripts/check_api_docs.py successfully.
  • Reviewed test coverage and ran the affected tests and JavaScript syntax checks.
  • Verified the tutorial and viewer use existing project dependencies.

Follow-up #1251 measures and reduces grasp retries in the three eight-environment examples.

Order permission queries after the latest sensor sample on the caller's CUDA stream without resampling or advancing histories. Cover pending forbidden contacts and dropped evidence in standalone and history sampling paths.

Validation: 75 targeted tests passed; Black 26.10.1, public API documentation coverage and agent context checks passed.
Share Default CPU numeric observations within bounded command-gather scopes, reconstruct immutable standard targets, and transfer fresh internal command ownership while preserving public snapshots and validation.

Keep native multi-chunk grasp geometry and reductions resident on CUDA within each query, preserving samples, thresholds, stream ordering and independent results. Add focused regression coverage and a reproducible runtime comparison driver.

Validation: 201 focused tests including CUDA and 42 context tests passed; Black 26.10.1, API docs 4924/4924 and diff checks passed. Sources match the final benchmark manifest.

Forward/reverse sensor-free UR5 execution versus main 0119171 (seconds): small main 10.34-10.45, off 9.07-9.31, on 8.31-9.08; medium main 49.24-57.12, off 47.19-50.90, on 48.66-49.57; large main 251.77-260.14, off 189.57-196.42, on 192.07-200.29. All 492 program attempts accepted. Default CPU physics retains legacy CUDA grasp classification; these shared-host single-seed results are not physical qualification.
Restore the existing Default CUDA backend policy. Remove the newly introduced public selector and its ordinary-reporting test variant, retaining CPU contact permission coverage and documenting Direct GPU static-identity limits.

Validation: 53 configuration/contact tests including CUDA and 42 context tests passed. Black 26.10.1, API docs and diff checks passed.
Preserve legacy catalog identities when phase paths are inactive, while keeping explicit phase declarations in registration fingerprints. Align preview error coverage with aggregate input support and update state test doubles for Direct GPU queries, including publication guard coverage.

Validation: reproduced all 11 failed CI cases; 65 focused tests passed; broader Task Program and command/context suite 327 passed, 3 GPU cases skipped. Black 26.10.1, API docs 4963/4963, context map and diff checks passed. Existing context already describes inactive-default fingerprint compatibility; no context update required. Main synchronized through 438e111.
Move the existing pre-Scene collection after clearing facade-owned data and entity references. Cyclic wrappers newly orphaned by those clears must finalize while their Scene and World parents are still alive, instead of being collected during a later teardown.

Extend the existing deferred-destruction test with a cyclic view finalizer; both registry-ownership cases fail before the change. Keep backend synchronization, ownership checks, physical validation and GC count unchanged.

Validation: 288 manager CPU tests passed; the full 149-node failed CI GPU shard passed 143 with 6 pre-existing capability skips on pinned DexSim 0.6.0rc2. Black 26.10.1, API docs 4963/4963, context and diff checks passed. Lifecycle context already requires child-before-parent cleanup and remains accurate. The reported native abort itself was not deterministic locally; this fixes the demonstrated lifetime gap at its cleanup boundary.
…ings

Identify explicit references after aggregate expansion; round contact holds up to the control grid and verify measured elapsed time; resolve the default-plane actor only when enabled. Retire expired warming attempts without proposal/credit and allow active peers to drain and export, including expiry during initial validation.

Reject incompatible expert command modes, ActionManager declarations and controlled-joint selections before constructing a World. Preserve component-owned resource paths when relocating deployment snapshots by reusing the component resolver's resource normalization.

Add focused regressions and extend real workload integration with an expired group alongside a peer that commits and exports a receipt. Validation: 106 expansion unit tests, 4 native integration tests on pinned DexSim 0.6.0rc2, and 42 context tests passed. Regressions reproduce all six review findings against 722e385. Black 26.10.1, API docs 4963/4963 and diff checks passed. Update workload guidance for action-schema compatibility and admission expiry.
Normalize ULP-scale error in exact-grid hold durations before ceiling. Measure hold elapsed time with bounded fsum reductions of actual physics callback intervals instead of subtracting accumulated World clocks. Preserve distinct samples, upward rounding of real off-grid durations, both-finger contact and drift requirements.

Compare control_parts and active_joint_ids from resolved environment configurations rather than their shared implementation class. Replace artificial-class coverage with real decoded configurations and a full deployment-file regression sharing ordinary EmbodiedEnv.

Validation: 117 expansion unit tests, four native integration tests on pinned DexSim 0.6.0rc2 and 42 context tests passed. Regressions reproduce all three follow-up findings on 8c244cd, including 50 Hz/100 Hz boundary holds, 500k/5M-step clock accumulation, truly short windows and pre-World schema rejection. Black 26.10.1, API docs 5010/5010, context map and diff checks passed. Update context for measured hold timing and configuration-owned joint selections.
Keep landing-pose acceptance independent of the raised release target, covering zero and default clearance as well as configured lowerer ownership. Reconstruct the coordinated-transport configuration through its normal validator instead of replacing its normalized binding dictionary.

Validation: reproduced both failed CI cases; all 62 GenSim workflow tests and the native placement/release integration probe passed on pinned DexSim 0.6.0rc2. Black 26.10.1, API docs 5010/5010, context map and diff checks passed. No production behavior changed; existing GenSim context already describes clearance and landing validation correctly.
Bind the test validator to its measured environment and retain the real MeasuredTaskMonitor with deterministic joint evidence. This keeps hold-duration and contact assertions intact while exercising the new batched predicate boundary, instead of supplying an incomplete synthetic monitor.

Validation: 471 related expansion/bridge/configuration/GenSim tests passed, plus both native validation/selected-handover probes on pinned DexSim 0.6.0rc2. Black 26.10.1, API docs 5013/5013, context map and diff checks passed. Dataset cache was isolated to task-owned tmpfs because the host root filesystem had no non-reserved free space. Production behavior and documentation contracts are unchanged.
Include scene preparation declarations and parameters in shared-World compatibility before creating the host. Reject cartesian_lift combined with phase_paths, consistent with Place's explicit conflict handling.

Cover real deployment decoding, matching and mismatched preparation hooks, and public fixed-grid PickUp planning. Keep the dual-arm test robot aligned with selected-row FK queries. Update the affected project context.
Keep settling rows controller-active without trajectory or receipt credit. Retain configured grasp preload through coordinated placement. Use explicitly supported preceding-seed path kernels and cooperatively merge compatible requests, preserving active planning budgets and legacy solver fallbacks.

Validation: actual reset/grasp negative regressions reproduce on the published SDK; selected-row real path batching passes, motion checks 123 passed, workload integration 3 passed, SSIK checks 2 passed. Five-task repeated physical qualification follows separately.
Keep W times B physical capacity and original attempt/time/byte/step limits. Competing retries reserve only running slots; pending writes and confirmed commits retain the hard quota. Cancel peers through the original command handshake after confirmed persistence, retaining canonical outcomes and measured partial evidence.

Preserve phase metadata on safe stops and do not advance an already pending rejection hold twice. Native Task Program jobs enable idle-row reuse; custom sources opt in with an explicit cancellation port.

Validation: 41 focused lifecycle, configuration and real simulation checks pass on DexSim rc2. The actual known-bad scheduler copy fails the new unexecuted-hold regression. Independent engine/robotics/verification passes completed; API/context/Black gates pass. Full five-task after measurements follow on an installed VCS package.
A finished runtime may still have its last ordinary motion awaiting env.step. Keep the default fail-closed abort contract, but allow the joint controller owner to supply actual applied target positions when no runtime safe hold exists. Never reuse the pending motion or acknowledge before real physics consumption.

Retain the interrupted phase when terminal fields are empty, and keep the last actual scalar controller metadata and row identity in cancelled diagnostics without per-step serialization. Validation: 12 focused abort, safe-hold and actual reuse cases pass on rc2; full bottle trigger and repeated five-task qualification follow on the installed revision. Independent reviews completed.
@matafela matafela added enhancement Improvements to existing capabilities; use feature for new capabilities. motion gen Robot motion planning, MotionGenerator, trajectory generation, retiming and expansion. visualization Browser viewers, markers, trajectory previews and interactive debugging displays. labels Nov 19, 2026
@matafela
matafela requested a review from yuecideng November 19, 2026 03:09
Base automatically changed from codex/expansion-workloads to main November 19, 2026 09:46
@yuecideng
yuecideng merged commit 22946b0 into main Nov 19, 2026
5 of 8 checks passed
@yuecideng
yuecideng deleted the cj/expansion-workloads-visualization branch November 19, 2026 14:08
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The PR reaches replay users and multiple scene-screening workflows.

The PR appears safe to merge, with a non-blocking fix needed for focused buttons when Space is pressed.

Fix All in CodexFindings

  1. P2 Space activates the wrong button ▶
Fix with agent prompt
### Issue 1
embodichain/lab/expansion/_frontend/preview-controls.js:110-112
when a candidate or reset button has keyboard focus. The document-wide handler calls `preventDefault()` and clicks the play button instead, so Space cannot activate the focused button and unexpectedly starts or pauses playback. Exclude focused buttons from this shortcut so they keep their normal keyboard behavior.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR pins the original recording during replay, adds bilingual WebRTC controls and comparison charts, and broadens collision and layout screening for larger, mesh-based scenes.

  • The original recording stays solid while candidates appear as separate ghosts.
  • The replay viewer streams native video and compares recorded motion.
  • A UR5 tutorial collects and replays layout and grasp variations.
  • Scene checks can test decomposed meshes and larger batches of collision pairs.
  • Layout sampling can shift supported objects beyond the main support rectangle.
  • Feasibility planning accepts valid integer masks from analytic IK solvers.
  • Task listings find environment settings beside each task configuration.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Confirmed recordings] --> B[Fixed reference and candidate states]
  B --> C[Native observer]
  C -->|WebRTC video| D[Replay page]
  D -->|Queued replay controls| B
  A --> E[Deviation bands and starting positions]
  E -->|Separate chart data| D
  F[Physical scene geometry] --> G[Convex parts and bounded collision checks]
  G --> H[Layout and motion screening]
Loading

Reviews (1) · Last reviewed commit: "refactor(expansion): simplify replay and..." · Reviewed by Greptile

Comment on lines +110 to +112
if(event.target.isContentEditable || event.target.closest?.('select, textarea, input:not([type="range"]), summary'))return;
event.preventDefault();
if(!event.repeat)$('play').click();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Space activates the wrong button when a candidate or reset button has keyboard focus. The document-wide handler calls preventDefault() and clicks the play button instead, so Space cannot activate the focused button and unexpectedly starts or pauses playback. Exclude focused buttons from this shortcut so they keep their normal keyboard behavior.

Prompt To Fix With AI
This is a comment left during a code review.
Path: embodichain/lab/expansion/_frontend/preview-controls.js
Line: 110-112

Comment:
**Space activates the wrong button** when a candidate or reset button has keyboard focus. The document-wide handler calls `preventDefault()` and clicks the play button instead, so Space cannot activate the focused button and unexpectedly starts or pauses playback. Exclude focused buttons from this shortcut so they keep their normal keyboard behavior.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Improvements to existing capabilities; use feature for new capabilities. motion gen Robot motion planning, MotionGenerator, trajectory generation, retiming and expansion. visualization Browser viewers, markers, trajectory previews and interactive debugging displays.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants