Skip to content

[Proposal] Add deformation-aware 3DGS bindings for deformable objects #1199

Open

Description

@yuecideng

Proposal

Implement explicit deformation-aware 3D Gaussian Splatting (3DGS) visual bindings for supported DeformableObject configurations. This is the deferred deformable work from #818, following the shared visual contracts and static/rigid/link Gaussian composition delivered by #819 and #934.

The binding must map the owning simulation state to Gaussian centers, covariance and view-dependent appearance orientation, and participate in the existing preparation, update, selective reset and teardown lifecycle. Browser 3DGS is outside this follow-up's scope.

Motivation

#934 supplies composable Mesh/Gaussian appearance for visual objects, rigid bodies and articulation links, but explicitly excludes deformable Gaussian buffers. Current deformable configuration and rendering remain mesh-based; applying an object or link transform cannot represent stretching and shear.

Closing #818 after its delivered shared/static/rigid/link work must retain the original deformable acceptance requirement here: qualified bindings update centers and covariance/orientation and restore correctly after reset. Visual appearance, simulation discretization, collision geometry and semantic geometry remain independent; Gaussian opacity does not determine physical density.

Additional context

Binding and numerical contract

  • Reuse the shared typed visual configuration and existing deformable ownership. Define a qualified simulation-to-visual mapping, such as tetrahedral embedding, a surface/cage mapping or a solver-provided Gaussian mapping. Gaussian count/order must not be assumed to match simulation nodes.
  • Preserve an authored rest state and define the mapping's validity assumptions. For a local-affine approximation, update both mu_t = phi(mu_0) and Sigma_t = J_phi * Sigma_0 * J_phi^T so stretching, rotation and shear affect the rendered Gaussian shape.
  • Define how view-dependent appearance orientation follows deformation, including the reference frame and rotation convention. Updating centers alone is insufficient.
  • Keep collision, mass/material properties, controls and physical/semantic geometry under their existing owners. Pure numerical mapping algorithms belong in Compute; native adaptation belongs at the Spawn/deformable boundary.

Lifecycle, resources and qualification

  • Build bindings, resolve stable handles and allocate mapping/dynamic buffers during preparation. Synchronize from the owning simulation state after updates and resets, before native visual consumers read it. Reuse DexSim-native implementations where available.
  • Keep native materialization, replication, topology revisions, removal and teardown with Spawn/SimulationManager. Rebuild or invalidate bindings when topology changes; unwind partial initialization and release borrowed views before their native owners.
  • Share eligible immutable Gaussian/rest resources across environments while keeping dynamic centers, covariance/orientation, visibility and reset state isolated. Selected-environment writes and resets must leave other instances unchanged. Avoid reloading/copying immutable point attributes each frame; document dynamic upload/allocation costs.
  • Declare the supported volume/surface, backend, solver, renderer and import combinations. Qualify the supported Newton/CUDA paths separately rather than assuming rigid Gaussian support implies deformable support. Reject unsupported combinations explicitly.
  • Qualify native RGB/depth and source/entity identity for each advertised combination, including same-state capture and reset behavior. Document unsupported sensor outputs. Browser splats and browser protocol extensions are excluded from this delivery.

Acceptance criteria

  • A supported deformable asset can compose Gaussian appearance through an explicit, validated deformation binding without changing its authored physics, collision or semantic geometry.
  • Mapping validation covers incompatible source/rest topology and Gaussian counts/order that differ from simulation nodes; unsupported combinations fail before partial native mutation.
  • Independent numerical references verify centers and covariance under translation, asymmetric rotation, stretch and shear, with a documented and tested appearance-orientation convention.
  • Preparation creates bindings/buffers once per valid topology; update and native capture use the current owning state without stepping physics inside the visual binding.
  • Multiple environments share eligible immutable resources while dynamic visual state remains independent; selected writes and resets preserve unselected rows.
  • Reset restores the corresponding simulation state and authored Gaussian centers, covariance/orientation and appearance consistently, including after deformation.
  • Import/source identity, stable handles, topology rebuilds, failed initialization, removal and teardown are covered without duplicate native entities or stale bindings.
  • Supported backend/solver/renderer/import combinations have native end-to-end RGB/depth and identity qualification; unsupported outputs and combinations are explicit.
  • Existing Mesh deformables and static/rigid/link Gaussian behavior remain compatible, with focused production regression coverage.
  • Representative deformation workloads report latency, dynamic memory/upload costs and bounded resource usage; public API documentation and mapped project context describe the supported contracts and limits.

Implementation entry points

Current reference: main at b629c7158.

  • embodichain/lab/sim/cfg/visuals.py, cfg/deformable.py: typed visual and deformable configuration.
  • embodichain/lab/sim/spawn/_visuals.py, spawn/scene.py, spawn/descriptors.py: native visual resources, binding and lifecycle.
  • embodichain/lab/sim/objects/deformable/: simulation-state reads and selected reset.
  • embodichain/lab/sim/sim_manager.py, sensors/camera.py: publication and native capture identity.
  • tests/sim/spawn/test_visuals.py, tests/sim/test_gaussian_visuals_e2e.py and existing deformable suites: extend production contract coverage.

Checklist

Activity

  1. added
    enhancementImprovements to existing capabilities; use feature for new capabilities.
    renderingImage generation, lighting, material rendering, native renderers and image quality.
    sim-objectSimulation entity configuration, state, creation, destruction and lifecycle.
    assetsRobot and scene assets, meshes, materials, asset catalogs and downloads.
    on Nov 15, 2026
  2. self-assigned this
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

assetsRobot and scene assets, meshes, materials, asset catalogs and downloads.enhancementImprovements to existing capabilities; use feature for new capabilities.renderingImage generation, lighting, material rendering, native renderers and image quality.sim-objectSimulation entity configuration, state, creation, destruction and lifecycle.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions