Mesh-based Non-collision Constraints

[zhx06]

Summary

Add mesh-based non-collision constraints via sphere-to-SDF

Detailed description

• Introduces CollisionMode.MESH as an alternative to AABB for no-overlap constraints, using greedy sphere decomposition + differentiable Warp SDF queries against actual collision geometry.
• Solver falls back to AABB for pairs where either object lacks a mesh. Validator mirrors this.
• CLI: --collision_mode mesh enables the new path.

Core files

• relations/warp_sdf_kernels.py — differentiable SDF queries on Warp meshes
• relations/warp_mesh_manager.py — sphere decomposition and mesh caching
• relations/relation_solver.py — vectorized mesh collision loss during optimization
• relations/object_placer.py — mesh collision validation at placement time
• relations/relation_loss_strategies.py — per-pair mesh loss for the strategy API

[isaaclab-review-bot]

🤖 Isaac Lab Review Bot — PR #771

Mesh-based Non-collision Constraints

Summary

This PR adds sphere-to-SDF mesh collision support as an alternative to AABB overlap detection. The architecture is clean — CollisionMode.MESH integrates well into the existing NoCollisionLossStrategy dispatch, and the greedy sphere decomposition + Warp SDF kernel approach is sound. The test suite is thorough (542 lines!) with good coverage of dispatch routing, gradient flow, and integration.

Findings

#	Severity	Finding
1	🟡 Warning	Validator creates fresh WarpMeshManager per call — cache never reused
2	🟡 Warning	Scale applied post-transform in extract_trimesh_from_usd may be incorrect for nested prims
3	🔵 Suggestion	object_base.py abstract method has no explicit return None
4	🔵 Suggestion	Sentinel warning pattern on function object is not thread-safe
5	🔵 Suggestion	Consider documenting the rotated-anchor limitation more prominently

See inline comments for details.

---

Update (5e86ed0a): Reviewed incremental changes since 655ac73.

Addressed Findings

• ✅ Finding #1 resolved — _get_cpu_mesh_manager() now lazily creates and caches the WarpMeshManager on the instance, eliminating redundant allocations per validation call. Good fix.
• ✅ Finding #2 resolved — Removed erroneous .T transpose on ComputeLocalToWorldTransform in usd_helpers.py. USD returns row-major matrices; the transpose was producing incorrect vertex transforms for nested prims.

Other Changes

• Validation logic refactored (_validate_placement): Mesh mode now skips AABB validation entirely (else branch). Previously both checks ran in mesh mode — the AABB check was redundant and could produce false negatives for non-convex shapes. Clean improvement.
• Test suite trimmed: Removed test_sphere_count_respects_budget, test_cache_key_differs_for_different_meshes, test_dispatch_falls_back_when_obj_is_none, and test_mesh_zero_loss_separated_cylinders. These removals look intentional (simplified scope / covered elsewhere), though removing cache-key differentiation test reduces regression coverage on the caching layer.

Remaining Observations

• Findings #3–#5 from original review remain unaddressed (low priority, suggestions only).
• The new _get_cpu_mesh_manager uses hasattr check — works fine but Optional attribute initialized in __init__ would be more explicit.

Overall: Good incremental improvement. The two main warnings from the initial review are resolved. No new concerns.

---

Update (729d892c): Reviewed incremental changes since 5e86ed0a.

Changes in this push (2 files)

1. relation_loss_strategies.py — Added parent_pos_resolved.expand(batch_size, -1) before the per-batch loop. This fixes a shape mismatch when parent_pos_resolved is not already batch-expanded (e.g., single parent broadcast to multiple children). Correct fix.

2. warp_mesh_manager.py — Wrapped getattr(obj, "scale", ...) in tuple() for cache key computation. This prevents unhashable types (e.g., numpy arrays or torch tensors returned by .scale) from breaking the dict lookup. Necessary bugfix.

Assessment

Both changes are small, targeted bugfixes. No new concerns introduced. All previous suggestions (#3–#5) remain low-priority and unaddressed.

[greptile-apps]

Greptile Summary

This PR adds CollisionMode.MESH as an alternative to AABB for no-overlap constraints, using greedy sphere decomposition plus differentiable Warp SDF queries against actual collision geometry. The solver falls back to AABB for pairs where either object lacks a mesh, and the validator mirrors this logic.

• New core modules (warp_sdf_kernels.py, warp_mesh_manager.py): differentiable single- and multi-mesh SDF kernels with per-solve sentinel reset; content-hash-based BVH + sphere caching with correct deep-copy handling.
• Solver integration (relation_solver.py, relation_loss_strategies.py): vectorized mesh collision path precomputes a per-pair cache once per solve, then runs a single multi-BVH kernel launch per batch element; yaw-aware AABB broadphase culls separated pairs before the narrowphase.
• Placement validation (object_placer.py, usd_helpers.py): orientation dict refactored to carry total yaw (marker + sampled); per-instance cached CPU mesh manager avoids BVH rebuilds across attempts.

Confidence Score: 4/5

The MESH collision path is largely correct but the USD mesh extraction applies spawn scale in world space rather than local space, which can silently produce wrong collision geometry for complex multi-prim USD assemblies.

The vectorized solver path and validation logic are well-tested and handle multi-env batching, yaw-aware transforms, and sentinel detection correctly. The USD mesh extractor is the weakest link: post-transform scaling is incorrect when per-prim world transforms are non-identity and scale is non-uniform.

isaaclab_arena/utils/usd_helpers.py (scale ordering in extract_trimesh_from_usd) and isaaclab_arena/tests/test_mesh_collision.py (missing @requires_warp on one test)

Important Files Changed

Filename	Overview
isaaclab_arena/relations/warp_sdf_kernels.py	New file: clean differentiable SDF kernels with per-solve sentinel reset via reset_sdf_sentinel_warning(); backward pass propagates analytical SDF gradients correctly; multi-mesh kernel routes each point to its own BVH.
isaaclab_arena/relations/warp_mesh_manager.py	New file: greedy sphere decomposition + Warp BVH caching; scale converted with tuple() to prevent unhashable-list TypeError; deep-copy drops C-pointer caches correctly.
isaaclab_arena/relations/relation_solver.py	Adds vectorized MESH no-overlap path: precomputed per-pair cache, yaw-aware AABB broadphase, single multi-mesh kernel launch per batch element; correctly detaches anchor/parent positions and flows gradient through active pairs only.
isaaclab_arena/relations/object_placer.py	Refactors orientations to carry total yaw (marker+sampled); adds _get_cpu_mesh_manager() with per-instance BVH cache; _validate_placement chains AABB then mesh check; _centers_in_target_frame correctly applies net yaw.
isaaclab_arena/relations/relation_loss_strategies.py	NoCollisionLossStrategy extended with MESH mode; _compute_mesh_loss now accepts child_yaw/parent_yaw and applies net_yaw rotation; parent_pos.expand(batch_size,-1) resolves the prior IndexError for multi-env anchor cases.
isaaclab_arena/utils/usd_helpers.py	New extract_trimesh_from_usd applies scale in world space after ComputeLocalToWorldTransform; incorrect for non-identity per-prim world transforms with non-uniform scale.
isaaclab_arena/tests/test_mesh_collision.py	Comprehensive new test suite covering sphere decomposition, SDF gradients, yaw-aware transforms, batch solving, broadphase culling, and sentinel safety; one test missing @requires_warp guard.
isaaclab_arena/relations/collision_mode.py	New file: simple CollisionMode enum (BBOX/MESH), no issues.
isaaclab_arena/relations/relation_solver_params.py	Adds collision_mode and num_spheres fields with sensible defaults; no issues.
isaaclab_arena/assets/object.py	Adds lazy-cached get_collision_mesh() with broad exception handling; deepcopy correctly nullifies trimesh to avoid unpicklable C pointers.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ObjectPlacer.place] --> B[_generate_initial_orientations]
    B --> C[_rotate_candidate_bboxes]
    C --> D[RelationSolver.solve]
    D --> E{CollisionMode?}
    E -->|BBOX| F[_compute_no_overlap_loss_aabb]
    E -->|MESH| G[_prepare_mesh_collision_cache]
    G --> H[_compute_no_overlap_loss_mesh]
    G --> I[_compute_no_overlap_loss_aabb skip_mesh_pairs=True]
    H --> J[Loss + gradient]
    I --> J
    F --> J
    D --> K[solved positions]
    K --> L[_validate_placement]
    L --> M[_validate_no_overlap AABB]
    M -->|pass + MESH mode| O[_validate_no_overlap_mesh]
    M -->|pass + BBOX mode| P[accept]
    O --> P
    M -->|fail| Q[reject]
    O -->|fail| Q

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[ObjectPlacer.place] --> B[_generate_initial_orientations]
    B --> C[_rotate_candidate_bboxes]
    C --> D[RelationSolver.solve]
    D --> E{CollisionMode?}
    E -->|BBOX| F[_compute_no_overlap_loss_aabb]
    E -->|MESH| G[_prepare_mesh_collision_cache]
    G --> H[_compute_no_overlap_loss_mesh]
    G --> I[_compute_no_overlap_loss_aabb skip_mesh_pairs=True]
    H --> J[Loss + gradient]
    I --> J
    F --> J
    D --> K[solved positions]
    K --> L[_validate_placement]
    L --> M[_validate_no_overlap AABB]
    M -->|pass + MESH mode| O[_validate_no_overlap_mesh]
    M -->|pass + BBOX mode| P[accept]
    O --> P
    M -->|fail| Q[reject]
    O -->|fail| Q

Loading

_{Reviews (12): Last reviewed commit: "add is_anchor" | Re-trigger Greptile}

[alexmillane]

First partial review.

Looks good. I haven't got to the warp mesh based stuff.

[alexmillane]

What do you think about moving the mesh stuff out of Object and into the solver?

I guess we have it here because we want caching behaviour. We could instead have something like CollisionShapeManager in the solver which manages the caching.

We should also probably move the bounding boxes there too.

[alexmillane]

I feel like the mesh vs. bounding-box decision should be made on the environment level. For example, an environment with a shelf we'd like to place something on qill require mesh-mode. Here we're leaving it up to the user a run time.

That seems like an odd choice to me.

Can you see a way we could configure this in each environment?

[alexmillane]

To me the obvious way would be to make the ObjectPlacerParams/RelationSolverParams an (optional) part of IsaacLabArenaEnvironment. If specified in the environment they're passed along to the solver.

[alexmillane]

Related to the comment above.

We're seeing a reflection of the members of ObjectPlacerParams on the interface to this function.

I think the correct way to get out of this issue (that we have to copy members of ObjectPlacerParams onto the interface), is to make this function accept a full ObjectPlacerParams instance, which is specified per-environment in IsaacLabArenaEnvironment.

We should probably make this change in an MR that preceeds this one.

[alexmillane]

This is a bit ugly and we'd like to avoid it. Or at worst, we could isolate it.

What the motivation here?

Could we solve this in the WarpMeshManager. Potentially by overriding its deep-copy operation?

Aside: why do we need special treatment of _warned_no_mesh? That appears just to be a set[str]?

[alexmillane]

Out of interest, why do we do this lazily?

It seems to me that there's no cost to creating one of these guys, as they just create empty dicts. What's the motivation here?

[zhx06]

This is to avoid loading warp when we only use bbox

[alexmillane]

Two suggestions:

• Part of this code represents transforming vectors in one frame to another. Under a transformation (that happens to only have a yaw component). I'm wondering if we can use more general tools for this. For example, can we use: isaaclab.utils.math for example quat_apply for applying rotations, rather than recoding the code for rotations here.
• It is useful if you use structured notation. For example q_B_A represents the rotation from the frame A to the frame B.

I just wrote a good example of doing this type of this here

[alexmillane]

This might destroy the seeding we do elsewhere.

I'm wondering if we should avoid setting the seed anywhere in our codebase, except for the one place we set it in Isaac Lab.

@xyao-nv @peterd-NV What do you think? Should we outlaw setting seeds in our code?