Repository navigation
Description
Proposal
Closure scope (2026-11-15): Shared visual contracts and render-only/static/rigid/link Mesh/3DGS support are delivered by #819 and #934, with unified USD source/import ownership in #795. Deformable 3DGS is tracked separately in #1199. Browser 3DGS is deferred and excluded from this issue's closure criteria. The proposal below retains the original design context.
Separate entity behavior, visual representation, and visual-to-state binding so Mesh and 3D Gaussian Splatting (3DGS) assets can be composed consistently across render-only objects, rigid bodies, deformables, and articulation links. Also separate whole-scene import from binding selected entities to EmbodiChain interfaces.
The class/config names below describe the intended contracts and are proposals, not existing APIs or a finalized public schema.
Motivation
Whole-scene USD import currently wraps parsed MeshObjectDesc entries as RigidObject, including entries with physics is None (which default to body_type="static"). There is no dedicated render-only object facade. Mesh geometry and material assumptions also appear throughout object APIs and browser visualization.
Adding only a standalone Gaussian object class would not cover the required combinations:
| Use case | Entity interface | Physics | Visual-state driver |
|---|---|---|---|
| Render-only Mesh/3DGS asset, no collision | VisualObject |
None | Object transform |
| Mesh/3DGS rigid object | RigidObject |
Rigid properties and optional collision proxies | Body pose |
| Mesh/3DGS deformable object | DeformableObject |
Simulation mesh/particles | Explicit deformation mapping |
| Mesh/3DGS appearance on an articulation link | Owning Articulation / Robot |
Existing link/joint structure | Named link pose |
Visual appearance, physical discretization, collision geometry, and semantic geometry can have different representations and resolutions. Changing appearance should preserve the authored physical behavior. Gaussian opacity is not physical density.
Related work:
- [Proposal] Shared sim.scene services and procedural scene construction #760 owns shared scene services; native resources, registries and lifecycle remain with Spawn/SimulationManager.
- [Proposal] Align Scene Engine exports with the Dextrove schema #817 tracks Scene Engine export alignment; coordinate, identity and asset-reference contracts should stay consistent.
- [Proposal] Semantics-preserving asset generalization for Task Programs #810 tracks semantics-preserving asset substitution; appearance changes must not invalidate its physical/semantic geometry requirements.
Additional context
1. Compose visuals independently of entity type
Introduce a lightweight render-only VisualObject and a shared visual-component contract usable by existing physical objects and individual articulation links.
Each visual element should describe:
- A typed representation/resource, initially Mesh and Gaussian Splat.
- A binding to an object transform, rigid body, named link, or deformable state.
- Local transform/scale, visibility, representation-specific appearance parameters, and a stable element identifier.
Allow multiple visual elements, including mixed Mesh/3DGS, under one entity/link. Link visuals remain owned by their articulation rather than becoming separately simulated objects. Physical types continue to express behavior rather than the renderer or source file format.
Reuse the existing BatchEntity foundation where appropriate; avoid requiring a broad entity inheritance rewrite or a second runtime scene graph.
2. Separate scene import from runtime-interface binding
Provide an import result describing source prim paths, resolved entity identity, representation/binding information and diagnostics. Bind selected imported entities to VisualObject, RigidObject, Articulation or Robot without loading the source again or creating duplicate native entities. Retain add_usd() as a convenience path that performs supported automatic classification and binding.
Fixed background assets may remain imported without per-object runtime state buffers. Explicitly managed visual objects should support selected-environment pose/appearance operations and restoration of their authored initial state.
Classify from resolved physics/collision capabilities and ownership, not Mesh/3DGS format alone. In particular, physics is None or a missing RigidBodyAPI alone must not misclassify static collision geometry as render-only. Preserve logical asset boundaries rather than registering every render submesh as an independent object.
SimulationManager and DexSim Spawn remain authoritative for materialization, stable handles, replication, topology revisions, removal and teardown. Keep the existing declaration -> prepare() -> runtime-binding boundary, and preserve USD-authored transforms/materials unless explicitly overridden.
3. Deferred: deformation binding (#1199)
The original deformable stage is moved to #1199. It retains the explicit simulation-to-Gaussian mapping, center and covariance updates, view-dependent appearance orientation, preparation-time allocation, update/reset synchronization, resource isolation and backend/solver qualification requirements. Completing that stage is tracked independently of closing this issue's delivered scope.
4. Update geometry consumers, visualization and resource ownership
- Add typed visual-representation access instead of assuming every visual provides vertices and triangles. Keep mesh-specific APIs explicit.
- Give planning, collision checks and grasp/placement consumers explicit physical or semantic geometry/proxies so appearance replacement preserves their inputs.
- Deferred: Browser Gaussian resources, updates and protocol extensions are excluded from this issue's closure scope. Existing unsupported Viser composition paths remain explicitly rejected.
- Qualify RGB, depth and segmentation support per renderer/representation combination, including stable entity/link identity.
- Share immutable resources across environment instances where supported; keep poses, visibility and deformation state isolated per instance. Avoid copying complete static Gaussian assets each frame.
- Preserve legacy
shape/visual_shapeconfigurations through documented normalization and reject ambiguous overlapping ownership.
Suggested delivery sequence
- Shared contracts and render-only objects: typed visuals/bindings,
VisualObject, import/binding separation, manager lifecycle integration, and legacy Mesh compatibility. - Static, rigid and link 3DGS: native resource/handle adaptation, mixed representations, transform synchronization, and qualified visualization/camera paths.
- Deformable 3DGS: explicit deformation mapping, dynamic Gaussian state, selective reset and supported backend/solver qualification. Moved to [Proposal] Add deformation-aware 3DGS bindings for deformable objects #1199.
Acceptance criteria
Completed items below apply to the delivered render-only/static/rigid/link scope and its documented backend/resource combinations.
- Render-only imports have an appropriate optional facade and do not acquire unintended physics/collision behavior.
- Selected entities bind by source path without duplicate native creation; source hierarchy and logical ownership remain traceable.
- A rigid asset can switch Mesh/3DGS appearance through supported companion visuals while retaining collision, mass/inertia and control behavior.
- Articulation-link visuals follow named links with correct local offsets and asymmetric rotations; mixed Mesh/3DGS links work.
- Moved to [Proposal] Add deformation-aware 3DGS bindings for deformable objects #1199: Qualified deformable bindings update centers and covariance/orientation and restore correctly after reset.
- Multiple environments share eligible immutable resources while selected writes/resets leave other instances unchanged.
- Planning and semantic geometry queries remain valid when visual representations change.
- Renderer/sensor capability gaps are explicit and supported native rendering/capture paths are qualified. Browser 3DGS is deferred and excluded from closure scope.
- Removal, failed initialization and teardown release resources through existing owners without stale bindings.
- Existing Mesh configurations retain documented behavior, with focused production tests and affected API/context documentation updates.
Completion evidence and supported scope
Reviewed against main at b629c7158, using current code and the existing merged-PR validation records; this tracking update does not claim a new GPU qualification run.
- Add VisualObject support with explicit USD sources #819 delivers render-only
VisualObject, explicit USD source selection and existing manager/Spawn lifecycle integration. - Unify USD loading through import recipes and authored scene semantics #795 unifies USD import recipes and source/resource mappings. Imported facades bind in place during
prepare()rather than requesting duplicate native entities. - Add Gaussian visuals, USD imports and scene grouping #934 delivers typed mixed Mesh/Gaussian visuals for visual objects, rigid bodies and named articulation/robot links; independent semantic geometry; immutable resource sharing; selected instance state/reset; native RGB/depth capture identity; and rollback/removal/rebuild coverage. Its validation records include native multi-Arena rigid/link capture and reset tests.
- Production coverage includes visual configuration, binding, geometry and rollback contracts, native Gaussian lifecycle/capture tests, and render-only object integration.
Explicit Gaussian companions are the qualified multi-Arena path. Native USD Gaussian references remain limited to one Arena with Default physics and Hybrid rendering; replacing their source appearance via visuals is unsupported. Gaussian segmentation and other unsupported sensor outputs retain explicit capability limits. Deformable 3DGS remains open in #1199; browser 3DGS is deferred without opening another tracking issue in this closure.
Implementation entry points
Current-code reference: 37a4e532.
embodichain/lab/sim/sim_manager.py: USD import/classification, registries and lifecycle.embodichain/lab/sim/spawn/scene.py,spawn/descriptors.py,spawn/usd.py: declaration/binding and native translation. Rigid descriptors already distinguishrendersfromcollisions.embodichain/lab/sim/cfg/,shapes.py,objects/: entity configuration, shared visual interfaces and geometry access. Surface deformables already support an independentvisual_shape.embodichain/lab/visualization/protocol.py,scene_exporter.py,backends/: typed resources, updates and backend capabilities.- Gym background construction and GenSim
usd_scene.pybindings: expose visual entities while retaining existing scene ownership.
References
- Isaac Lab 2.3.1 InteractiveScene: miscellaneous assets use
extras/XFormPrimand are excluded from scene reset/update. - Isaac Lab 3.0 EA InteractiveScene: static extras retain spawned configuration without automatically creating a runtime view.
- Isaac Lab asset configuration:
spawn=Nonebinds an asset already present in the scene. - PhysGaussian, sections 3.4-3.5: local-affine Gaussian deformation and appearance-orientation treatment; a reference for binding semantics, not a commitment to adopt its MPM solver.
Checklist
- I have checked that there is no similar issue in the repo (required). Searches covered all issue states for 3DGS, Gaussian, render, visual, USD, and scene assets. Related [Proposal] Add Visualization Marker Abstraction (Axes, Primitives, etc.) #129, [Proposal] Shared sim.scene services and procedural scene construction #760, [Proposal] Semantics-preserving asset generalization for Task Programs #810 and [Proposal] Align Scene Engine exports with the Dextrove schema #817 have distinct scope.
Activity
yuecideng commentedon Nov 15, 2026
Closing the delivered shared visual, render-only, static, rigid and articulation-link scope after #819, #795 and #934. The nine in-scope acceptance items are updated. The remaining deformable 3DGS requirements are retained in follow-up sub-issue #1199. Browser 3DGS is deferred and excluded from this closure. The revised body records supported native/backend limits and existing qualification evidence.