Skip to content

[Proposal] Decouple visual representations and bindings from physics entities for Mesh and 3DGS #818

Closed
0 of 1 issue completed

Description

@yuecideng

Proposal

Closure scope (2026-11-15): Shared visual contracts and render-only/static/rigid/link Mesh/3DGS support are delivered by #819 and #934, with unified USD source/import ownership in #795. Deformable 3DGS is tracked separately in #1199. Browser 3DGS is deferred and excluded from this issue's closure criteria. The proposal below retains the original design context.

Separate entity behavior, visual representation, and visual-to-state binding so Mesh and 3D Gaussian Splatting (3DGS) assets can be composed consistently across render-only objects, rigid bodies, deformables, and articulation links. Also separate whole-scene import from binding selected entities to EmbodiChain interfaces.

The class/config names below describe the intended contracts and are proposals, not existing APIs or a finalized public schema.

Motivation

Whole-scene USD import currently wraps parsed MeshObjectDesc entries as RigidObject, including entries with physics is None (which default to body_type="static"). There is no dedicated render-only object facade. Mesh geometry and material assumptions also appear throughout object APIs and browser visualization.

Adding only a standalone Gaussian object class would not cover the required combinations:

Use case Entity interface Physics Visual-state driver
Render-only Mesh/3DGS asset, no collision VisualObject None Object transform
Mesh/3DGS rigid object RigidObject Rigid properties and optional collision proxies Body pose
Mesh/3DGS deformable object DeformableObject Simulation mesh/particles Explicit deformation mapping
Mesh/3DGS appearance on an articulation link Owning Articulation / Robot Existing link/joint structure Named link pose

Visual appearance, physical discretization, collision geometry, and semantic geometry can have different representations and resolutions. Changing appearance should preserve the authored physical behavior. Gaussian opacity is not physical density.

Related work:

Additional context

1. Compose visuals independently of entity type

Introduce a lightweight render-only VisualObject and a shared visual-component contract usable by existing physical objects and individual articulation links.

Each visual element should describe:

  • A typed representation/resource, initially Mesh and Gaussian Splat.
  • A binding to an object transform, rigid body, named link, or deformable state.
  • Local transform/scale, visibility, representation-specific appearance parameters, and a stable element identifier.

Allow multiple visual elements, including mixed Mesh/3DGS, under one entity/link. Link visuals remain owned by their articulation rather than becoming separately simulated objects. Physical types continue to express behavior rather than the renderer or source file format.

Reuse the existing BatchEntity foundation where appropriate; avoid requiring a broad entity inheritance rewrite or a second runtime scene graph.

2. Separate scene import from runtime-interface binding

Provide an import result describing source prim paths, resolved entity identity, representation/binding information and diagnostics. Bind selected imported entities to VisualObject, RigidObject, Articulation or Robot without loading the source again or creating duplicate native entities. Retain add_usd() as a convenience path that performs supported automatic classification and binding.

Fixed background assets may remain imported without per-object runtime state buffers. Explicitly managed visual objects should support selected-environment pose/appearance operations and restoration of their authored initial state.

Classify from resolved physics/collision capabilities and ownership, not Mesh/3DGS format alone. In particular, physics is None or a missing RigidBodyAPI alone must not misclassify static collision geometry as render-only. Preserve logical asset boundaries rather than registering every render submesh as an independent object.

SimulationManager and DexSim Spawn remain authoritative for materialization, stable handles, replication, topology revisions, removal and teardown. Keep the existing declaration -> prepare() -> runtime-binding boundary, and preserve USD-authored transforms/materials unless explicitly overridden.

3. Deferred: deformation binding (#1199)

The original deformable stage is moved to #1199. It retains the explicit simulation-to-Gaussian mapping, center and covariance updates, view-dependent appearance orientation, preparation-time allocation, update/reset synchronization, resource isolation and backend/solver qualification requirements. Completing that stage is tracked independently of closing this issue's delivered scope.

4. Update geometry consumers, visualization and resource ownership

  • Add typed visual-representation access instead of assuming every visual provides vertices and triangles. Keep mesh-specific APIs explicit.
  • Give planning, collision checks and grasp/placement consumers explicit physical or semantic geometry/proxies so appearance replacement preserves their inputs.
  • Deferred: Browser Gaussian resources, updates and protocol extensions are excluded from this issue's closure scope. Existing unsupported Viser composition paths remain explicitly rejected.
  • Qualify RGB, depth and segmentation support per renderer/representation combination, including stable entity/link identity.
  • Share immutable resources across environment instances where supported; keep poses, visibility and deformation state isolated per instance. Avoid copying complete static Gaussian assets each frame.
  • Preserve legacy shape / visual_shape configurations through documented normalization and reject ambiguous overlapping ownership.

Suggested delivery sequence

  1. Shared contracts and render-only objects: typed visuals/bindings, VisualObject, import/binding separation, manager lifecycle integration, and legacy Mesh compatibility.
  2. Static, rigid and link 3DGS: native resource/handle adaptation, mixed representations, transform synchronization, and qualified visualization/camera paths.
  3. Deformable 3DGS: explicit deformation mapping, dynamic Gaussian state, selective reset and supported backend/solver qualification. Moved to [Proposal] Add deformation-aware 3DGS bindings for deformable objects #1199.

Acceptance criteria

Completed items below apply to the delivered render-only/static/rigid/link scope and its documented backend/resource combinations.

  • Render-only imports have an appropriate optional facade and do not acquire unintended physics/collision behavior.
  • Selected entities bind by source path without duplicate native creation; source hierarchy and logical ownership remain traceable.
  • A rigid asset can switch Mesh/3DGS appearance through supported companion visuals while retaining collision, mass/inertia and control behavior.
  • Articulation-link visuals follow named links with correct local offsets and asymmetric rotations; mixed Mesh/3DGS links work.
  • Moved to [Proposal] Add deformation-aware 3DGS bindings for deformable objects #1199: Qualified deformable bindings update centers and covariance/orientation and restore correctly after reset.
  • Multiple environments share eligible immutable resources while selected writes/resets leave other instances unchanged.
  • Planning and semantic geometry queries remain valid when visual representations change.
  • Renderer/sensor capability gaps are explicit and supported native rendering/capture paths are qualified. Browser 3DGS is deferred and excluded from closure scope.
  • Removal, failed initialization and teardown release resources through existing owners without stale bindings.
  • Existing Mesh configurations retain documented behavior, with focused production tests and affected API/context documentation updates.

Completion evidence and supported scope

Reviewed against main at b629c7158, using current code and the existing merged-PR validation records; this tracking update does not claim a new GPU qualification run.

Explicit Gaussian companions are the qualified multi-Arena path. Native USD Gaussian references remain limited to one Arena with Default physics and Hybrid rendering; replacing their source appearance via visuals is unsupported. Gaussian segmentation and other unsupported sensor outputs retain explicit capability limits. Deformable 3DGS remains open in #1199; browser 3DGS is deferred without opening another tracking issue in this closure.

Implementation entry points

Current-code reference: 37a4e532.

  • embodichain/lab/sim/sim_manager.py: USD import/classification, registries and lifecycle.
  • embodichain/lab/sim/spawn/scene.py, spawn/descriptors.py, spawn/usd.py: declaration/binding and native translation. Rigid descriptors already distinguish renders from collisions.
  • embodichain/lab/sim/cfg/, shapes.py, objects/: entity configuration, shared visual interfaces and geometry access. Surface deformables already support an independent visual_shape.
  • embodichain/lab/visualization/protocol.py, scene_exporter.py, backends/: typed resources, updates and backend capabilities.
  • Gym background construction and GenSim usd_scene.py bindings: expose visual entities while retaining existing scene ownership.

References

Checklist

Activity

  1. added
    enhancementImprovements to existing capabilities; use feature for new capabilities.
    renderingImage generation, lighting, material rendering, native renderers and image quality.
    sim-objectSimulation entity configuration, state, creation, destruction and lifecycle.
    assetsRobot and scene assets, meshes, materials, asset catalogs and downloads.
    on Oct 19, 2026
  2. self-assigned this
    on Oct 20, 2026
  3. yuecideng commented on Nov 15, 2026

    @yuecideng
    ContributorAuthor

    Closing the delivered shared visual, render-only, static, rigid and articulation-link scope after #819, #795 and #934. The nine in-scope acceptance items are updated. The remaining deformable 3DGS requirements are retained in follow-up sub-issue #1199. Browser 3DGS is deferred and excluded from this closure. The revised body records supported native/backend limits and existing qualification evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

assetsRobot and scene assets, meshes, materials, asset catalogs and downloads.enhancementImprovements to existing capabilities; use feature for new capabilities.renderingImage generation, lighting, material rendering, native renderers and image quality.sim-objectSimulation entity configuration, state, creation, destruction and lifecycle.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions