Repository navigation
Description
Proposal
Implement explicit deformation-aware 3D Gaussian Splatting (3DGS) visual bindings for supported DeformableObject configurations. This is the deferred deformable work from #818, following the shared visual contracts and static/rigid/link Gaussian composition delivered by #819 and #934.
The binding must map the owning simulation state to Gaussian centers, covariance and view-dependent appearance orientation, and participate in the existing preparation, update, selective reset and teardown lifecycle. Browser 3DGS is outside this follow-up's scope.
Motivation
#934 supplies composable Mesh/Gaussian appearance for visual objects, rigid bodies and articulation links, but explicitly excludes deformable Gaussian buffers. Current deformable configuration and rendering remain mesh-based; applying an object or link transform cannot represent stretching and shear.
Closing #818 after its delivered shared/static/rigid/link work must retain the original deformable acceptance requirement here: qualified bindings update centers and covariance/orientation and restore correctly after reset. Visual appearance, simulation discretization, collision geometry and semantic geometry remain independent; Gaussian opacity does not determine physical density.
Additional context
Binding and numerical contract
- Reuse the shared typed visual configuration and existing deformable ownership. Define a qualified simulation-to-visual mapping, such as tetrahedral embedding, a surface/cage mapping or a solver-provided Gaussian mapping. Gaussian count/order must not be assumed to match simulation nodes.
- Preserve an authored rest state and define the mapping's validity assumptions. For a local-affine approximation, update both
mu_t = phi(mu_0)andSigma_t = J_phi * Sigma_0 * J_phi^Tso stretching, rotation and shear affect the rendered Gaussian shape. - Define how view-dependent appearance orientation follows deformation, including the reference frame and rotation convention. Updating centers alone is insufficient.
- Keep collision, mass/material properties, controls and physical/semantic geometry under their existing owners. Pure numerical mapping algorithms belong in Compute; native adaptation belongs at the Spawn/deformable boundary.
Lifecycle, resources and qualification
- Build bindings, resolve stable handles and allocate mapping/dynamic buffers during preparation. Synchronize from the owning simulation state after updates and resets, before native visual consumers read it. Reuse DexSim-native implementations where available.
- Keep native materialization, replication, topology revisions, removal and teardown with Spawn/SimulationManager. Rebuild or invalidate bindings when topology changes; unwind partial initialization and release borrowed views before their native owners.
- Share eligible immutable Gaussian/rest resources across environments while keeping dynamic centers, covariance/orientation, visibility and reset state isolated. Selected-environment writes and resets must leave other instances unchanged. Avoid reloading/copying immutable point attributes each frame; document dynamic upload/allocation costs.
- Declare the supported volume/surface, backend, solver, renderer and import combinations. Qualify the supported Newton/CUDA paths separately rather than assuming rigid Gaussian support implies deformable support. Reject unsupported combinations explicitly.
- Qualify native RGB/depth and source/entity identity for each advertised combination, including same-state capture and reset behavior. Document unsupported sensor outputs. Browser splats and browser protocol extensions are excluded from this delivery.
Acceptance criteria
- A supported deformable asset can compose Gaussian appearance through an explicit, validated deformation binding without changing its authored physics, collision or semantic geometry.
- Mapping validation covers incompatible source/rest topology and Gaussian counts/order that differ from simulation nodes; unsupported combinations fail before partial native mutation.
- Independent numerical references verify centers and covariance under translation, asymmetric rotation, stretch and shear, with a documented and tested appearance-orientation convention.
- Preparation creates bindings/buffers once per valid topology; update and native capture use the current owning state without stepping physics inside the visual binding.
- Multiple environments share eligible immutable resources while dynamic visual state remains independent; selected writes and resets preserve unselected rows.
- Reset restores the corresponding simulation state and authored Gaussian centers, covariance/orientation and appearance consistently, including after deformation.
- Import/source identity, stable handles, topology rebuilds, failed initialization, removal and teardown are covered without duplicate native entities or stale bindings.
- Supported backend/solver/renderer/import combinations have native end-to-end RGB/depth and identity qualification; unsupported outputs and combinations are explicit.
- Existing Mesh deformables and static/rigid/link Gaussian behavior remain compatible, with focused production regression coverage.
- Representative deformation workloads report latency, dynamic memory/upload costs and bounded resource usage; public API documentation and mapped project context describe the supported contracts and limits.
Implementation entry points
Current reference: main at b629c7158.
embodichain/lab/sim/cfg/visuals.py,cfg/deformable.py: typed visual and deformable configuration.embodichain/lab/sim/spawn/_visuals.py,spawn/scene.py,spawn/descriptors.py: native visual resources, binding and lifecycle.embodichain/lab/sim/objects/deformable/: simulation-state reads and selected reset.embodichain/lab/sim/sim_manager.py,sensors/camera.py: publication and native capture identity.tests/sim/spawn/test_visuals.py,tests/sim/test_gaussian_visuals_e2e.pyand existing deformable suites: extend production contract coverage.
Checklist
- I have checked that there is no similar issue in the repo (required). Searches for
deformable 3DGSanddeformable Gaussianfound [Proposal] Decouple visual representations and bindings from physics entities for Mesh and 3DGS #818 as the existing proposal; this issue explicitly extracts its deferred deformable delivery rather than duplicating the completed shared/static/rigid/link work.
Activity