Add controller haptic feedback to G1 teleoperation environments

[rwiltz]

Description

Adds controller haptic feedback to the IsaacTeleop stack: the XR motion
controller vibrates when a teleoperated G1 hand applies force to an object,
giving operators grasp feedback during teleoperation and demo recording.

Introduces a small device-agnostic feedback seam in isaaclab_teleop — a
HapticFeedbackReceiver protocol (kept off the input-only DeviceBase), a
HapticFeedbackCfg, and a HapticFeedbackDriver that reads per-hand contact
forces from the scene. IsaacTeleopDevice implements the protocol by building
an isaacteleop HapticSink (ControllerHapticDevice + TactileVectorToControllerPulse)
that reuses the existing controller tracker, and feeds per-step force through the
same external-input path already used for the anchor transform.

Both G1 locomanipulation teleop examples (IsaacContrib-PickPlace-Locomanipulation-G1-Abs
and IsaacContrib-PickPlace-FixedBaseUpperBodyIK-G1-Abs) gain per-hand finger
ContactSensors and a haptic_feedback config; teleop_se3_agent.py and
record_demos.py drive the shared driver via the existing hasattr-based
capability discovery.

Requires isaacteleop with the controller haptic device (ControllerHapticDevice,
TactileVectorToControllerPulse, HapticSink).

Fixes # (issue)

Type of change

• New feature (non-breaking change which adds functionality)

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[rwiltz]

@greptile

[greptile-apps]

Greptile Summary

This PR introduces end-to-end controller haptic feedback for the G1 teleoperation environments, wiring per-hand ContactSensor forces through a new HapticFeedbackDriver into an isaacteleop HapticSink that calls xrApplyHapticFeedback on the operator's XR controller. The implementation is device-agnostic: a narrow HapticFeedbackReceiver protocol keeps haptics off the input-only DeviceBase, capability discovery uses the existing hasattr/isinstance idiom, and both teleop scripts share the same driver API with clean update/stop separation.

• New haptic_feedback.py defines HapticFeedbackCfg, HapticFeedbackReceiver, HapticFeedbackDriver, and the create_haptic_feedback_driver factory; the driver reads net_forces_w from ContactSensor data for env 0 and forwards a per-hand scalar to the device.
• session_lifecycle.py builds the HapticSink subgraph (skipped in MCAP replay mode with sinks=[]) and injects cached _haptic_forces into the pipeline via the existing external-input mechanism each step.
• Both G1 env configs add per-hand ContactSensorCfg over finger links and a HapticFeedbackCfg; both teleop scripts (teleop_se3_agent.py, record_demos.py) call haptic_stop() in every non-stepping branch so a paused grip stops vibrating immediately.

Confidence Score: 5/5

Safe to merge — the haptic path is entirely opt-in and degrades gracefully to no-ops on devices that don't support vibration or envs without a haptic config.

The change adds a new opt-in output path that is unreachable unless the env config carries a haptic_feedback attribute AND the active device implements HapticFeedbackReceiver. All existing environments and non-XR devices are unaffected. The stale-force concern raised in a prior review is addressed: both scripts zero the cached force whenever teleop is paused or the session hasn't started. MCAP replay explicitly passes sinks=[] so no live controller writes occur during scripted playback.

No files require special attention — the two nits (inline duplication in record_demos.py and per-step dict allocation in session_lifecycle.py) are purely stylistic and do not affect correctness.

Important Files Changed

Filename	Overview
source/isaaclab_teleop/isaaclab_teleop/haptic_feedback.py	New file: HapticFeedbackCfg, HapticFeedbackReceiver protocol, HapticFeedbackDriver, and create_haptic_feedback_driver factory — clean, well-documented, no issues
source/isaaclab_teleop/isaaclab_teleop/session_lifecycle.py	Adds haptic sink build/teardown and external-input injection; replay path correctly passes sinks=[] so no live controller writes during MCAP replay
source/isaaclab_teleop/isaaclab_teleop/isaac_teleop_device.py	Adds send_haptic() implementing HapticFeedbackReceiver and zeros haptic forces in reset(); passes haptic_cfg through to session lifecycle correctly
scripts/environments/teleoperation/teleop_se3_agent.py	Adds _make_haptic_io helper and calls haptic_stop() in both the action-is-None and paused branches, addressing the stale-force concern
scripts/tools/record_demos.py	Inlines equivalent haptic driver setup (without a shared helper); haptic_stop() wired correctly in all non-stepping branches
source/isaaclab_tasks/isaaclab_tasks/contrib/locomanip_pick_place/locomanipulation_g1_env_cfg.py	Adds per-hand ContactSensorCfg and HapticFeedbackCfg; enables activate_contact_sensors on the robot spawn
source/isaaclab_tasks/isaaclab_tasks/contrib/locomanip_pick_place/fixed_base_upper_body_ik_g1_env_cfg.py	Identical changes to locomanipulation_g1_env_cfg.py — ContactSensors and HapticFeedbackCfg added consistently

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Script as teleop_se3_agent / record_demos
    participant Driver as HapticFeedbackDriver
    participant Sensor as ContactSensor (scene)
    participant Device as IsaacTeleopDevice
    participant LC as TeleopSessionLifecycle
    participant Sink as HapticSink (isaacteleop)
    participant XR as OpenXR (xrApplyHapticFeedback)

    Note over Script: Each sim frame
    Script->>Script: "action = teleop_interface.advance()"
    alt action is None OR teleoperation paused
        Script->>Driver: stop()
        Driver->>Device: send_haptic(endpoint, 0.0) x2
        Device->>LC: push_haptic(endpoint, 0.0) x2
        Note over LC: _haptic_forces zeroed
    else teleoperation active
        Script->>Script: env.step(actions)
        Note over Sensor: Physics updates contact data
        Script->>Driver: update()
        Driver->>Sensor: data.net_forces_w[0]
        Sensor-->>Driver: [num_bodies, 3] tensor
        Driver->>Driver: sum of per-body force norms → scalar
        Driver->>Device: send_haptic(endpoint, force)
        Device->>LC: push_haptic(endpoint, force)
        Note over LC: _haptic_forces[endpoint] = force
    end

    Note over Script: Next advance() → step()
    LC->>LC: _build_external_inputs()
    LC->>LC: wrap _haptic_forces in TactileVector TensorGroup
    LC->>Sink: external_inputs injected
    Sink->>Sink: TactileVectorToControllerPulse (gain/deadband/sat)
    Sink->>XR: xrApplyHapticFeedback (amplitude, freq, duration)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Script as teleop_se3_agent / record_demos
    participant Driver as HapticFeedbackDriver
    participant Sensor as ContactSensor (scene)
    participant Device as IsaacTeleopDevice
    participant LC as TeleopSessionLifecycle
    participant Sink as HapticSink (isaacteleop)
    participant XR as OpenXR (xrApplyHapticFeedback)

    Note over Script: Each sim frame
    Script->>Script: "action = teleop_interface.advance()"
    alt action is None OR teleoperation paused
        Script->>Driver: stop()
        Driver->>Device: send_haptic(endpoint, 0.0) x2
        Device->>LC: push_haptic(endpoint, 0.0) x2
        Note over LC: _haptic_forces zeroed
    else teleoperation active
        Script->>Script: env.step(actions)
        Note over Sensor: Physics updates contact data
        Script->>Driver: update()
        Driver->>Sensor: data.net_forces_w[0]
        Sensor-->>Driver: [num_bodies, 3] tensor
        Driver->>Driver: sum of per-body force norms → scalar
        Driver->>Device: send_haptic(endpoint, force)
        Device->>LC: push_haptic(endpoint, force)
        Note over LC: _haptic_forces[endpoint] = force
    end

    Note over Script: Next advance() → step()
    LC->>LC: _build_external_inputs()
    LC->>LC: wrap _haptic_forces in TactileVector TensorGroup
    LC->>Sink: external_inputs injected
    Sink->>Sink: TactileVectorToControllerPulse (gain/deadband/sat)
    Sink->>XR: xrApplyHapticFeedback (amplitude, freq, duration)

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'develop' into rwiltz/gr1_e..." | Re-trigger Greptile}

[rwiltz]

@greptile