Skip to content

Add a physical grasp validation and generalization benchmark - #1252

Draft
yuecideng wants to merge 3 commits into
mainfrom
codex/physical-grasp-benchmark
Draft

yuecideng wants to merge 3 commits into
mainfrom
codex/physical-grasp-benchmark

Conversation

@yuecideng

@yuecideng yuecideng commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

Add a physics-based feasibility check for a ranked grasp pose and a benchmark for generalization across rigid objects and parallel-jaw grippers. The existing CoffeeCup benchmark measures candidate generation and selection. These new commands measure whether a selected pose can pick up the object, keep it off the table and retain it during transport.

Design

Inputs and execution

  • validate takes a single dynamic rigid-body configuration, a catalog gripper and a ranked JSONL file. Each candidate is a 4×4 object-from-TCP transform in metres. Validation executes every supplied pose.
  • run crosses objects, grippers, placements and seeds from a suite. It either runs GraspKit's antipodal generator on the object's mesh or loads an external algorithm's ranked candidates. It freezes the top ten before physics scoring.
  • GraspValidator owns a headless CUDA scene. Both commands accept --physics default (the default) or --physics newton (DexUni with two-way coupling). Each invocation scores one backend. Each arena contains one target, a table and a native gripper mounted on three driven translation axes. The wrist orientation is set at reset. The gripper's calibrated TCP, approach frame, named pads and force-limited finger commands determine how each pose is executed. Arm reachability and IK are outside this protocol.
  • A batch shares an object and gripper model. Arenas independently execute different candidates and conditions from the same settled object pose, with velocities and contact records reset. Atomic checkpoints preserve candidate order, initial pose, trial identities and results. A worker failure or missing contact observation triggers a retry. Unresolved cases remain visible in the inventory and suppress ranking.

Physical protocol

The open gripper approaches along its calibrated axis from 8 cm away, closes over 0.40 s and lifts vertically by 10 cm. A successful hold requires the lowest collision point to stay at least 2 cm above the table throughout the 3 s hold, continuous retention without table support and contact on both named pads at the end. The same trace yields retention at 0.5, 1 and 3 s.

Each candidate gets three independent trials: nominal, world X and world Y. In each directional trial, the gripper travels 3 cm and returns in two 0.30 s minimum-jerk legs. The commanded peak is 1.92 m/s² (approximately 0.20 g). The trial records the measured TCP acceleration, maximum object translation and rotation relative to the TCP, and time to first loss. Window entry and censoring distinguish a missed pickup from an object still held when observation ended.

Default GPU uses a stationary kinematic table so the contact query can identify its actor. Both backends use a positive measured impulse where available. For DexUni geometry-only rows, a finite, nonpositive signed separation establishes contact. The checkpoint fingerprint includes the backend, solver and contact policy. Use separate output directories to retain both aggregate reports.

Generalization and scoring

Metric Definition for one scene
Success@1 The first-ranked candidate passes the nominal hold.
Robust@1 That same candidate passes the nominal, X and Y trials.
Hit@5 / Hit@10 At least one nominal hold succeeds within the indicated prefix.
Physical Precision@10 Number of successful nominal holds in the top ten divided by ten. Missing candidate slots contribute zero.
Generalization gap Seen-object/seen-gripper Success@1 minus double-held-out Success@1, in percentage points.

The report displays all four object/gripper holdout combinations and rotates the held-out hand by default. CAD variants and near duplicates share a declared family. “Held out” refers to benchmark tuning; training-set novelty requires a verified training inventory. Scores average placements and seeds within each object–gripper pair, then weight pairs equally. Method comparisons reuse the same scenes and bootstrap object families together for paired 95% intervals. Directional retention also reports how many trials successfully reached the disturbance.

The suite produces a Markdown leaderboard, a JSON report and per-case Parquet trial records. The package includes runnable inputs and the candidate-file contract. Usage and suite configuration

Research basis

Cross-embodiment holdouts follow the evaluation question studied by GraspGen-X and AdaGrasp. Fixed-budget physical precision can be computed for user assets without collecting a dense reference set of successful poses. GraspGen-X · AdaGrasp · GraspNet ranking protocol

IPC-GraspSim motivates clearance and terminal jaw contact; MultiGripperGrasp motivates a timed hold and fall-off time. ACRONYM and the grasp-planning benchmarking protocol motivate disturbance testing and recording relative motion. The two horizontal trajectories and their intensity are choices made for this suite. IPC-GraspSim · MultiGripperGrasp · ACRONYM · Benchmarking Protocol for Grasp Planning Algorithms

Validation

  • On an RTX 5090, each backend completed 180 unique physical trials across two objects, three grippers, ten candidates and three conditions. Both reports had zero missing cases. The Parquet trial IDs and backend tags were checked against the JSON reports.

  • The nine-arena GPU control runs on both backends. Two offset candidates miss; the centred candidate establishes a physical grasp. Each condition records retention separately, including slips during the 3 s hold. Measured peak accelerations can differ between solvers for the same commanded trajectory.

  • Backend-specific checkpoints prevent cross-backend resume. The focused grasp test suite includes mixed measured/geometric contact evidence and both real GPU CLI paths (6 tests passed).

  • Focused tests cover native GPU execution, ranking and fixed denominators, an X/Y-specific loss, rotation during retention, family-level aggregation, package resources, GraspKit compatibility and API documentation. Project-context checks cover routing to the new benchmark.

  • Uses the existing simulation, CUDA and reporting dependencies.

Type of change

  • New feature

Checklist

  • Ran black . with Black 26.10.1.
  • Updated benchmark usage and the motion-planning context.
  • Verified public API documentation coverage.
  • Reviewed behavioral coverage and ran focused validation.

@yuecideng yuecideng added physics Physics backends, dynamics, contacts, fluids, deformables and physics solvers. benchmark Benchmark protocols, suites, metrics, evaluation tools and technical reports. feature New user-facing capabilities. labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark Benchmark protocols, suites, metrics, evaluation tools and technical reports. feature New user-facing capabilities. physics Physics backends, dynamics, contacts, fluids, deformables and physics solvers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant