Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
2af3f86
feat(toolkits): add atomic skill trajectory annotation
Nov 9, 2026
2229cfa
Merge branch 'main' into codex/trajectory-skill-annotator
Nov 10, 2026
c80b361
update
Nov 11, 2026
5f6169b
feat(toolkits): select skills and fit annotated core intervals
Nov 12, 2026
4de52af
feat(expansion): add rolling single-world workload execution
yuecideng Nov 14, 2026
2cbe6a5
fix(expansion): own validated metadata inputs
yuecideng Nov 14, 2026
61f922d
fix(sensors): wait for contact sample before permission checks
yuecideng Nov 14, 2026
4723cba
feat(expansion): expose task ports and measured ghost comparison
yuecideng Nov 14, 2026
ab04d97
feat(graspkit): scope pose NMS memory budgets per task
yuecideng Nov 14, 2026
bc5ae3f
perf: reduce single-world planning and row execution overhead
yuecideng Nov 15, 2026
504f44f
docs(expansion): document native ghost preview and WebRTC follow-up
yuecideng Nov 15, 2026
bddf4d7
fix(physics): remove direct GPU API configuration override
yuecideng Nov 15, 2026
24c21fb
feat(expansion): preview confirmed results with native ghost rendering
yuecideng Nov 15, 2026
80d2d51
feat(expansion): collect and preview bounded parameter distributions …
yuecideng Nov 15, 2026
39201fe
fix(motion): preserve planning contexts and phase velocities
yuecideng Nov 15, 2026
85704e1
fix(expansion): preview recorded mesh scenes and native runtime metadata
yuecideng Nov 15, 2026
239a3d5
feat(expansion): compose native single-world workload collection
yuecideng Nov 15, 2026
78dc36c
fix(expansion): isolate workload grasp frames and reset settling
yuecideng Nov 15, 2026
7f97064
feat(expansion): expose declarative collection and preview entrypoints
yuecideng Nov 15, 2026
ed01657
Merge latest main and fix expansion CI regressions
yuecideng Nov 16, 2026
722e385
fix(sim): collect released native views before closing their scene
yuecideng Nov 16, 2026
f3e08ae
fix(expansion): address admission, acceptance and preview review find…
yuecideng Nov 16, 2026
8c244cd
fix(expansion): sync main and preserve placement phase paths
yuecideng Nov 16, 2026
2dfcbed
fix(expansion): stabilize hold timing and joint selection checks
yuecideng Nov 16, 2026
565d981
fix(gensim): clear supports and settle opening before release
yuecideng Nov 16, 2026
4a0f34c
merge: sync main before expansion performance fixes
yuecideng Nov 17, 2026
33fe553
test(gensim): align binding coverage with release clearance defaults
yuecideng Nov 17, 2026
bf4a254
fix(expansion): preserve selected rows and batch measured validation
yuecideng Nov 17, 2026
d083f61
merge: preserve concurrent GenSim binding coverage
yuecideng Nov 17, 2026
c3e6cd1
test(expansion): use native monitors in hold sampling coverage
yuecideng Nov 17, 2026
d887e99
fix(expansion): validate prepared scenes and PickUp path conflicts
yuecideng Nov 17, 2026
cba37a4
fix(expansion): retain reset holds and batch seeded solver paths
yuecideng Nov 17, 2026
4dad707
feat(expansion): reuse idle rows for bounded retry candidates
yuecideng Nov 17, 2026
4759be0
update
Nov 17, 2026
db388b3
fix(expansion): retain owner-proven holds after terminal runtime
yuecideng Nov 17, 2026
b5d16a8
Merge main and preserve portable trajectory fitting workflow
Nov 17, 2026
9e29ad6
fix(expansion): keep speculative retry reuse opt-in
yuecideng Nov 17, 2026
71fda76
docs(context): route trajectory annotation commands
Nov 17, 2026
86303c7
fix(expansion): compare effective scene preparation defaults
yuecideng Nov 17, 2026
5c557e8
feat(expansion): add workspace pose coverage and distribution reports
yuecideng Nov 18, 2026
910db94
feat(expansion): add bidirectional conditional layout sampling
yuecideng Nov 18, 2026
97493f3
fix(expansion): address point-IK and idle-row review feedback
yuecideng Nov 18, 2026
cfdc9f3
merge: sync main into expansion workloads
yuecideng Nov 18, 2026
5b5a307
docs: refresh scene expansion architecture evidence
yuecideng Nov 18, 2026
0557cdc
feat(expansion): support rolling layouts and measured phase coverage
yuecideng Nov 18, 2026
88c2e18
fix(packaging): include expansion distribution report template
yuecideng Nov 18, 2026
4132d86
docs(expansion): correct distribution API heading
yuecideng Nov 18, 2026
95990b8
merge: sync main into expansion workloads
yuecideng Nov 18, 2026
65f7d5d
feat(expansion): separate nominal scene and proposal seeds
yuecideng Nov 18, 2026
657daec
fix(expansion): stabilize installation identity in offline reports
yuecideng Nov 18, 2026
f18d7f7
fix(expansion): enforce task identity and layout admission contracts
yuecideng Nov 18, 2026
6dd084f
merge: integrate concurrent expansion updates
yuecideng Nov 18, 2026
f38aacb
fix(expansion): preserve admission deadlines for active rollouts
yuecideng Nov 18, 2026
c67c721
Merge remote-tracking branch 'origin/codex/expansion-workloads' into …
yuecideng Nov 18, 2026
9fc1fbd
Merge the trajectory fitting dependency (#1221)
han-xudong Nov 19, 2026
c4d9cae
feat(task-program): compile portable fitted Pickup deployments
han-xudong Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
15 changes: 11 additions & 4 deletions agent_context/MAP.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,11 @@ topics:
- embodichain/lab/gym/utils/trajectory_state.py
- embodichain/lab/gym/envs/managers/observation_types.py
- embodichain/lab/gym/envs/demo.py
- embodichain/lab/gym/envs/wrapper/replay.py
- embodichain/lab/gym/utils/profiler.py
- extensions/embodichain_tasks/configs/tasks/manipulation/gaussian_pick_place/
watch_paths: [embodichain/lab/gym/utils/, embodichain/lab/gym/envs/task_program/, embodichain/cli/, extensions/embodichain_tasks/,
tests/gym/envs/, embodichain/lab/evaluation/, tests/lab/evaluation/, tests/lab/scripts/test_evaluate_task_objective.py]
tests/gym/envs/, embodichain/lab/gym/envs/wrapper/replay.py, embodichain/lab/evaluation/, tests/lab/evaluation/, tests/lab/scripts/test_evaluate_task_objective.py]
related_topics: [simulation-system, manager-functor, sim-visualization, task-programs, data-pipeline]
status: active
- id: manager-functor
Expand Down Expand Up @@ -206,7 +207,7 @@ topics:
- id: expansion-engine
title: Expansion Engine
aliases: [Expansion Engine, offline expansion, 扩增引擎, 批量专家采集]
keywords: [ExpansionEngine, ExpansionRunCfg, ExpansionJobCfg, ExpansionSession, ExpansionSessionCfg, ExpansionCoverage, EpisodeCoverage, GymExpansionHost, ArtifactEpisodeSink, MediaSpool, render_observation_variants]
keywords: [ExpansionEngine, ExpansionRunCfg, ExpansionJobCfg, ExpansionSession, ExpansionSessionCfg, ExpansionCoverage, EpisodeCoverage, GymExpansionHost, ArtifactEpisodeSink, MediaSpool, render_observation_variants, WorkloadProgramExecutor, WorkloadExecutionCfg, workload_count, GymExpansionHostExtensions, MeasuredExpansionValidator, MeasuredTrajectory, preview_expansion_comparison, 扩增虚影]
paths: [topics/expansion-engine/expansion-engine.md]
source_of_truth:
- embodichain/lab/expansion/
Expand Down Expand Up @@ -303,6 +304,7 @@ topics:
SimulationSceneExpansionHost, SimulationSceneInitialState, episode_prepare]
paths: [topics/scene-expansion/scene-expansion.md]
source_of_truth:
- embodichain/lab/_scene_config/
- embodichain/lab/sim/scene/feasibility/
- embodichain/lab/sim/scene/expansion/
- embodichain/lab/task_program/integrations/feasibility.py
Expand Down Expand Up @@ -439,13 +441,17 @@ topics:
- id: atomic-actions
title: Atomic Skills
aliases: [atomic skill, atomic skills, atomic action, 原子技能, 原子动作]
keywords: [PlanningBudget, PlanningRecovery, ExecutionPlanLifecycle, AtomicSkillEngine, run-atomic-skill, PlanningContext, ExecutionSession, ExecutionRunner, SkillInvocation,
keywords: [annotate-trajectory, trajectory_annotator, 轨迹切分, 轨迹标注, fit-trajectory, --skill-sequence, scene-skill-fitting-inputs, SkillFitCfg, PickUpMotionGoal, SlideMotionGoal, TwistMotionGoal, HandOverMotionGoal, 视频关节轨迹拟合, 轨迹回填, PlanningBudget, PlanningRecovery, ExecutionPlanLifecycle, AtomicSkillEngine, run-atomic-skill, PlanningContext, ExecutionSession, ExecutionRunner, SkillInvocation,
hand_geometry_cfgs, GraspOpeningTarget, prepare_grasp_query, scoped grasp preparation,
RecoveryPolicy, HeldObjectGuardRequest, PhaseEffectGateRequest, articulation_geometry, control-grid retiming,
AffordanceSamplingContext, AffordancePoseCandidates, AffordanceSample, affordance sampling]
paths: [topics/atomic-actions/atomic-actions.md]
source_of_truth:
- embodichain/lab/sim/atomic_skills/engine.py
- embodichain/lab/sim/atomic_skills/fitting.py
- embodichain/toolkits/skill_fitting/
- embodichain/toolkits/trajectory_annotator/
- embodichain/data_pipeline/skill_fitting.py
- embodichain/lab/sim/atomic_skills/core.py
- embodichain/lab/sim/atomic_skills/goals.py
- embodichain/lab/sim/atomic_skills/execution.py
Expand All @@ -458,7 +464,8 @@ topics:
- embodichain/cli/run_example.py
- examples/sim/atomic_skill/
watch_paths: [embodichain/lab/sim/atomic_skills/, embodichain/lab/sim/atomic_actions/, tests/sim/atomic_skills/,
scripts/tutorials/atomic_skill/, tests/examples/atomic_skill/]
scripts/tutorials/atomic_skill/, examples/toolkits/trajectory_annotation/,
tests/toolkits/trajectory_annotator/, tests/examples/atomic_skill/]
related_topics: [simulation-system, motion-planning, robot-system, ik-solvers, task-programs]
status: active
- id: task-programs
Expand Down
29 changes: 29 additions & 0 deletions agent_context/topics/atomic-actions/atomic-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ Hosts acquire versioned measurements; verification is the boundary that commits
or invalidates observed effects. Preserve required stop/hold boundaries on the
executed control grid instead of relying only on native planner timing.

Row-local planning within a larger World must carry the same explicit
`context.env_ids` through FK/IK, root poses and diagnostics. HandOver applies this
to both source/destination arms, including its continuation and failed-waypoint
diagnostic routes. A one-row planning context does not implicitly select arena
zero or all arenas of the robot.

## Package map

| Concern | Source of truth |
Expand Down Expand Up @@ -99,6 +105,29 @@ are calibrated scene inputs. Preserve their selected owners and fingerprints;
never infer capabilities or general physical success from a sample name. The
host's sampled footprint/support checks remain finite measured acceptance,
separate from geometric proposal qualification.
`CoordinatedPlacement` retains each binding's configured grasp command during
approach/hold and on the support hand through release/retreat. Contact-blocked
measured jaw positions cannot replace these closing targets. Measured robot
observations and held-object transforms still supply geometry and validation.

## Trajectory annotation and fitting

`embodichain annotate-trajectory --bundle` opens the portable input package's
video, joints, calibrated robot and reconstructed scene. `fit-trajectory` and
the website's whole-trajectory fitting panel share `toolkits/skill_fitting/`;
`atomic_skills/fitting.py` adapts the ordinary native planners. Explicit bindings
and observed closing motion select participating control parts; names infer none.
Accepted native spans replace only their selected cores. Remaining source frames
become DemoSkill spans with original DOF-by-frame tensors.

`run-task --skill-sequence skills.yaml` is a direct engine host that plans native
object-bound goals from live state. It preserves closing preload on unbound
joints. A matching Expansion Profile transforms only native free phases;
DemoSkill commands remain unchanged. Execution completion is separate from
physical success. See [fitting contracts and resource ownership](trajectory-fitting.md)
and the [runnable guide](../../../examples/toolkits/skill_fitting/README.md)
for pose frames, grasp symmetry, boundary qualification, collision caches,
Viser comparison and portable input/output packages.

## Recommended change sites

Expand Down
110 changes: 110 additions & 0 deletions agent_context/topics/atomic-actions/trajectory-fitting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Trajectory annotation, fitting and native execution

Video-selected reconstruction enters through `embodichain fit-trajectory`
(`toolkits/skill_fitting/cli.py`) or the annotation website's whole-trajectory
fitting panel (`toolkits/trajectory_annotator/_fitting.py`). Both hosts reuse
`toolkits/skill_fitting/_pipeline.py`; the web host retains one background job
and result, keeps decoding on its existing owner, and invalidates results after
annotation revisions change. Annotation choices cover all built-in skills without
importing simulation; only supported reconstruction types enter fitting. A saved
optional core interval separates loose annotation boundaries from reconstruction,
preserving surrounding motion. `_episode.py` fits each annotation independently
from the immutable recording, combines only its core, and partitions every
remaining source sample into DemoSkill spans. Unsupported authored skills fail
explicitly. The CLI selects this path with `--all-segments --annotations ...`;
its existing single-segment mode remains available.
Its bounded joint charts follow video time. `_viewer.py` owns a local Viser worker
with independent original-material and translucent-blue URDF models; playback
samples all recorded joints on the full clock, including explicit mimic columns.
Model bases use the profile calibration relative to each solver's URDF root,
without TCP offsets. One latest-time slot bounds playback work; replacing the fit
or closing the host stops its viewer. The downloadable NPZ retains full samples.
`_export.py` packages accepted episodes for `run-task --skill-sequence skills.yaml`.
`_native_export.py` writes object-bound Pickup goals, actual participant bindings,
native options/policy and a sealed local resource inventory. `_execution.py` is a
direct AtomicSkillEngine caller: each native invocation plans from measured robot
state and current object pose, then submits ControllerAction commands through Gym.
Only explicitly bound motion/grasp parts take new native-plan commands. Unbound
joints retain the previous commanded targets, not measured positions: a loaded
opposite gripper must keep its closing preload. Demo spans retain full source
command ownership, including deliberate releases. Grasp command/TCP calibration
belongs to the input robot profile; export copies solver TCPs into the Gym robot.
It is not a Task Program lowerer. Only DemoSkill gaps carry original DOF-by-frame
PT matrices and source timestamps; there is no atomic/full-episode trajectory PT.
The host resamples demo clocks and inserts derivative-bounded quintic connections
between adjacent postures. Its execution report separates command completion from
measured object heights and leaves physical success unqualified.
`_sequence_expansion.py` accepts a matching Combined Expansion Profile with
`source.kind=skill_sequence`. This B=1 fixed-scene host applies shared
`atomic_skills/_plan_expansion.py` transforms only to native spans; DemoSkill
source tensors and resampled command paths are excluded. The timeline indices
include every span. Native free-phase permissions, bound motion joints and
protected boundary stencils constrain perturbations. Each recipe resets physics,
and rejected proposals retry without nominal substitution. `_sequence_trace.py`
streams T submitted commands and T+1 measured robot/object states into bounded
HDF5 chunks. These artifacts record command completion with unknown physical
success, not verified Expansion Engine dataset receipts. Editable `expansion.yaml`
is separate from the sealed physical inventory. The
[fitting example](../../../examples/toolkits/skill_fitting/README.md) owns commands
and profile settings.
`_bundle.py` owns `scene-skill-fitting-inputs/v1`: local scene meshes/colliders and
explicit poses, calibrated robot model/profile, original joints, annotations and
frame presentation clock. `fit-trajectory --bundle` requires no image pixels or
original exporter directory; `--package-output` creates portable input resources.
Optional `frame_source` adds a content-bound local video for
`annotate-trajectory --bundle`. This entry resolves all fitting defaults from the
same inventory and persists mutable `annotations.edited.json` separately from
the sealed annotation seed. `--package-video` creates video-enabled resources;
the web fit and exported native runtime need no original recording directory.
A v2 bundle additionally pins a host-owned execution context and declares the
qualified design reference as its fitting trajectory. CLI fitting accepts saved
annotation edits for the same frame source via `--annotations`.
`export_task_program` uses that context to bind a scene-bound Pickup to one
named invocation. It separates the fitted core from the surrounding annotation
samples. Each retained interval needs an explicit named DemoSkill and measured
boundary contract. A retained slice on the same post-step clock can be built
from the pinned source asset with its original commands, shifted phases, events
and scoped evidence. The output links recorded event ticks and asset hashes to
compiled calls. The Expansion Engine owns physical qualification and writes a
new receipt before dataset conversion.
`_collision.py` prepares content-addressed scene convex parts; web import and
packaging supply precomputed files for decomposition-free runtime startup.
Scene-owned Z compensation precedes fitting; the existing pickup interpolation
factor becomes native duration scaling at an unchanged control cadence.
`RobotProfile.simulation` carries explicit physics overlays without changing
kinematic calibration or custom control names. See the
[web example](../../../examples/toolkits/trajectory_annotation/README.md).
`toolkits/skill_fitting/_scene.py` grounds reconstructed local meshes in explicit
world poses and obtains calibrated, finite-surface-qualified antipodal grasps.
Single-arm scene fitting ranks valid TCP poses against the recorded grasp, then
`atomic_skills/fitting.py` calls the ordinary native planner. PickUp uses an
object-bound `GraspGoal`; Slide/Twist retain their motion/axis contracts with the
selected grasp anchor. Parallel-hand alternatives apply an exact half turn in
local TCP Z; contact-frame calibration is conjugated into the physical surface
query so tilted/offset virtual TCPs do not change the rotation axis. Both variants
must pass target-surface checks before ranking. Fitting uses a wider configurable
approach angle and disables the inferred support-plane filter by default;
these host policies do not change native affordance defaults or certify table
clearance. PickUp approaches along the selected grasp's Z. Source motion supplies approach, phase clocks and motion
parameters rather than pointwise acceptance targets. Quintic free-phase bridges
preserve the outer derivative stencils and the contact core. Pickup may rebalance
its free-phase clock inside the unchanged outer interval; explicit phase hints
remain fixed and observed phases remain in provenance. Qualification covers
every changed joint/derivative stencil, while pre-existing source-limit violations
remain separately visible. Rejected results retain candidates without replacing
the original reference. The native fitter does not execute physics; the host
exports the native invocation deployment described above.

`_participants.py` pairs configured motion/hand groups through URDF ancestry and
selects a unique closing hand; ambiguous evidence requires an explicit choice.
Names do not establish capabilities. A profile must supply explicit runtime TCP
calibration and actual finger collision assets; automatic scene geometry supports
opposing prismatic fingers, including two URDF followers of one aperture driver.
Without a scene, the effect-free world-TCP motion
fitter remains available for PickUp, Slide, Twist and explicitly bound HandOver.
Scene HandOver needs time-resolved object/attachment bindings and is not inferred
from an initial object pose. Both paths retain explicit source identity, inclusive
frame conversion, joint semantics and clock offsets. Automatic RGB contact
recognition and physical-success certification are absent. The
[runnable guide](../../../examples/toolkits/skill_fitting/README.md) owns scene
formats, calibration conventions, phase partitions and result arrays.
6 changes: 6 additions & 0 deletions agent_context/topics/data-pipeline/data-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ tabletop Cartesian transfer needs source TCP poses. Prototype preparation and
planning do not establish collision freedom or physical task success. See
[Real2Sim motion](real2sim.md) for ownership and source binding.

`data_pipeline/skill_fitting.py` separately owns named joint recordings, inclusive
video-to-joint clock conversion and standalone numeric/JSON reconstruction
bundles. It does not relax Real2Sim's object-reference requirements. The
[Atomic Skills overview](../atomic-actions/atomic-actions.md) owns the fitting
entry point and the scene-grounded versus effect-free motion-only boundaries.

Read [online sampling](online-sampling.md) for worker states, refill, shared
errors, action schemas, sample provenance, causal histories and example-runner
boundaries. Read [persistence](persistence.md)
Expand Down
23 changes: 23 additions & 0 deletions agent_context/topics/env-framework/execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,23 @@ not orchestrate multiple experiments or qualify kinematic playback.
and control replay restore complete particle states at each recorded boundary;
dynamic replay restores only the initial particle state and recomputes later
fluid motion from recorded actions.
External expert recordings may declare `meta.state_joint_names` for the full
state columns. Replay verifies the exact robot and active joint sets, then
maps state qpos/qvel and explicit expert commands into native order. Missing,
duplicate or mismatched joints reject before stepping. Legacy files without
this field retain numeric-layout checks; raw policy actions cannot be remapped
as joint commands. Legacy fitted replay packages remain readable through this path.
- **Native fitted sequences**: `run-task --skill-sequence skills.yaml` is a separate
direct AtomicSkillEngine host in `toolkits/skill_fitting/_execution.py`. It uses
the plain physical Gym deployment, replans native goals from live state and
submits ControllerAction through `env.step`; only source DemoSkill gaps use PT.
It cannot combine with replay or Task Program ownership. A matching
`--expansion-profile` delegates fixed-scene B=1 spatial variants to
`toolkits/skill_fitting/_sequence_expansion.py`; DemoSkill paths stay unchanged.
Execution
completion/observed heights are recorded separately from physical success.
The portable input/output contract is owned by
[Atomic Skills](../atomic-actions/atomic-actions.md).
- **Policy projection**: `rollout_buffer` retains actual policy observations,
including sampled noise, and its existing obs/action/reward alignment for
LeRobot/RL. It shares the acquisition boundary with raw history, rather than
Expand All @@ -381,6 +398,12 @@ not orchestrate multiple experiments or qualify kinematic playback.
policy action space; a controller/expert trajectory uses the private
`ExpertActionSpec` layout. One trajectory cannot mix these schemas.
- **CLI**: `run-task --replay --replay_trajectory <path> --replay_mode {kinematic,dynamic,control}`.
Automatic `kinematic` / `dynamic` replay runs without wall-clock pacing and
closes the viewer on completion. For persistent inspection use `control`:
`--headless --viser` exposes a frame slider, and `--no-headless` without
`--viser` opens the native window even when the file sets `headless: true`.
Control replay starts paused; terminal `a` plays at the control interval,
`n` / `p` step, a number plus Enter seeks, `r` resets, and `q` exits.

---

Expand Down
Loading