Skip to content

About

Offline forensics for ROS 2 robot missions: replay, root-cause analysis, reality gaps, RGB+thermal fusion, MCAP/Rerun export. NVIDIA Alpamayo + AlpaSim report adapters.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

221 Commits

Folders and files

PyRoboReplay

Ask a recorded robot mission why it failed. Replay ROS 2 missions, run causal / root-cause analysis, quantify the reality gap, and fuse RGB + thermal evidence. Rust core, Python bindings, CLI-first forensics.

CI PyPI License: Apache 2.0 Python 3.10+ Rust NVIDIA

Why it exists: robot logs tell you what happened. Incident reviews need why — which event preceded the failure, what perception missed, and what the robot should have detected. PyRoboReplay turns recorded missions into causal evidence you can diff, export, and defend.

60-second quickstart

pip install pyroboreplay
pyroboreplay replay mission.bag
pyroboreplay analyze mission.bag --detect-gaps --detail --format json --output investigation.json
from pyroboreplay import Mission

mission = Mission.from_ros_bag("warehouse.bag")
failures = mission.detect_failures()
analysis = mission.analyze_failure(timestamp=1234567890.0)

print(len(failures))
print(analysis.get_primary_hypothesis())
print(f"{analysis.get_diagnostic_confidence():.0%}")

Interactive replay keys: Space play/pause, n / → next event, p / ← previous event, ↑ / ↓ speed, ? help, q / Esc quit.

NVIDIA ecosystem

PyRoboReplay is NVIDIA-optional: replay, causal analysis, CLI, and tests run on CPU. NVIDIA appears where it matters — as a provenance-preserving adapter layer for physical-AI model and simulator runs.

Component Status What it actually does
Alpamayo trajectory + reasoning reports Available Loads offline Alpamayo artifacts (.json, .jsonl, .npz, mapping), computes minADE/minFDE, preserves reasoning traces, and correlates failures already detected in the mission. No torch, no weights, no GPU.
AlpaSim ASL rollout reports Available Parses size-delimited .asl rollouts (lazy upstream reader), preserves session/version/actor/ego-ground-truth metadata, and emits JSON or Markdown. Does not vendor AlpaSim or infer safety from poses alone.
Alpamayo / AlpaSim model execution External Generate artifacts with the upstream NVIDIA stacks; PyRoboReplay analyzes the evidence.
SANA-WM synthetic mission replay Not integrated No SANA-WM loader, generator, or synthetic-scenario provenance ships today. Replay accepts recorded mission sources, not generated video.

The honest headline: this is the forensic layer for NVIDIA physical-AI outputs, not an NVIDIA runtime. The adapters are real and tested; model execution and GPU requirements stay upstream.

At a glance

Area Status
ROS 2 SQLite .db3 / .bag mission ingest Supported
Interactive replay + timeline scrubber Stable
Failure detection / root-cause hypotheses Stable
Reality-gap analysis + JSON reports Stable
Mission comparison + topic listing Stable
Native MCAP export Stable
Rerun .rrd export Optional ([rerun])
DINO detection-gap ingest, RGB + thermal fusion Available (Phase 12/13)
Nav2 root-cause heuristics, navigation sessions Available (Phase 14/15)
Alpamayo + AlpaSim report adapters Available
Phase 16-20 ops JSON bridge only (ops() / run_op())
MCAP import Not wired
Live fleet dashboard / streaming Not a product
SANA-WM synthetic replay Not shipped

What it is

PyRoboReplay ingests recorded robot missions (ROS 2 bags) into a unified timeline and answers why and what-if questions rather than only where and what: which events preceded a failure, what the perception stack missed, and what the robot should have detected. The heavy lifting is a Rust core (cdylib + rlib) built with PyO3, exposed to Python through maturin. The supported entry points are the Python API and a Rust CLI with replay, compare, analyze, list, terrain, forensics, nav2, session, and export.

Features

Exposed via CLI / Python (verified, tested):

  • Mission replay — interactive timeline scrubber with keyboard controls (replay).
  • Reality-gap analysis — probabilistic mismatch scoring, severity classification, evidence aggregation, JSON/text output (analyze --detect-gaps).
  • Failure detection and diagnosis — Mission.detect_failures(), Mission.analyze_failure() with hypotheses, confidence, and prioritized actions.
  • Mission comparison — side-by-side diff of two bags (compare).
  • Topic listing — inspect available topics in a bag (list).
  • Real ROS 2 CDR decoding — standard sensor_msgs / nav_msgs / geometry_msgs binary layouts are decoded from real message bytes (src/adapters/cdr.rs); pose confidence is derived from the real position-covariance trace, so localization-loss detection fires on real data.
  • Retrospective detection gaps (Phase 12) — ingest externally generated DINO detections and compare them with the robot's YOLO detections (forensics --dino, analyze_detection_gaps()).
  • RGB + thermal fusion (Phase 13) — ingest RGB detections and a thermal frame, then fuse them (forensics --rgb --thermal, fuse_rgb_thermal()).
  • Navigation sessions (Phase 14) — build a unified timeline from a recorded ROS 2 bag plus optional Linux logs and Nav2 exports (session, build_navigation_session()).
  • Nav2 root-cause heuristics (Phase 15) — run localization/planner/costmap/obstacle/semantic analyzers and classification on a measurements document (nav2, analyze_nav2_diagnostics()).
  • Recorded-mission exports — native MCAP (export --format mcap) plus optional Rerun .rrd export through the Python API or pyroboreplay-export.
  • PyTerrainMap grid interop — read the documented pyterrainmap-gridmap v1 JSON export directly (terrain, load_pyterrainmap_grid()).
  • Phase 16-20 JSON operation bridge — six real Rust operations (causal graph, counterfactual query, fleet learning, temporal patterns, trend detection, decision tree) via ops() / run_op(); see Advanced operations.

Implemented in the Rust core, not yet wired to CLI or Python (library API only):

  • Persistent world knowledge (entities, locations, facts across missions) and spatial grounding.
  • Multi-mission / longitudinal learning.
  • Pluggable detection backends (YOLO / SAM / template fallback). Phase 12 accepts external DINO detections; it does not run a model itself.

Who it's for / Use cases

  • Warehouse and logistics fleets — reconstruct why a run failed, find missed detections and coverage gaps, compare two strategy revisions.
  • Safety and compliance review — produce auditable, saved JSON findings for an incident (analyze --detect-gaps --format json --output report.json).
  • Perception R&D — compare perception strategies over matched missions and quantify regressions.
  • NVIDIA physical-AI teams — turn Alpamayo / AlpaSim runs into trajectory, reasoning, and provenance reports correlated with real mission failures.
  • Field autonomy teams — investigate localization loss, planner oscillation, and sensor disagreement from recorded missions.

What it is not

  • Not a real-time monitoring or fleet dashboard. It works on recorded missions, not live streams. The streaming modules are internal building blocks, not a supported live fleet product.
  • Not an MCAP importer yet. MCAP export is supported; the Phase 14 MCAP adapter exists but is not wired into Mission.from_ros_bag(). ROS 2 SQLite .db3 / .bag is the supported ingest today.
  • Phases 16-20 are not typed Python APIs. Six implemented Rust operations are reachable through the JSON-only ops() / run_op() bridge (Advanced operations); the rest of those modules, including decision reconstruction, clustering, scenario analysis, and rule extraction, remain Rust-library-only.
  • Phase 12 does not run DINO/SAM models. It ingests detections produced by an external open-vocabulary run and performs gap analysis; the in-crate run_dino_detection method remains a deterministic test stub.
  • Alpamayo / AlpaSim are not executed here. The adapters consume offline artifacts; model weights, licensing, and GPU requirements stay upstream.
  • PyTerrainMap interop is file-based, not a runtime dependency. PyRoboReplay reads the documented pyterrainmap-gridmap v1 JSON export; there is still no Cargo/pip dependency, H3 store integration, or ROS bridge. src/core/pyterrain_bridge.rs remains self-contained terrain-modeling logic.
  • Not published to crates.io — build the Rust crate from source if you need it directly.
  • Not a certified or safety-verified system.

Project status

Beta (Development Status 4). Current source version is 2.13.1; the latest PyPI release is 2.13.1 (published 2026-10-23) and includes the Phase 16-20 operation bridge. The CLI and Python API are the tested, supported surface. 869 cargo test --lib unit tests pass plus public-surface Python tests (including 10 for the Phase 16-20 bridge) and 7 real-fixture ROS 2 CDR tests. cargo fmt --check and cargo clippy --all-targets --all-features -- -D warnings are clean (re-verified 2026-10-20). See docs/internal/ROADMAP_HONEST.md for the implemented/partial/missing breakdown.

Installation

Packaging: the 2.13.x wheel ships both the pyroboreplay and pyroboreplay-export console scripts and declares numpy, so installing the package is enough for both the Python API and the CLI.

Wheel coverage: PyPI publishes a macOS arm64 wheel plus an sdist; the sdist builds on other platforms. Release CI is configured to build and smoke-test Linux x86_64/aarch64, macOS arm64/x86_64, and Windows x86_64 for Python 3.10–3.13, and those wheels are produced when the release workflow's upload job runs. See the platform matrix.

pip install pyroboreplay

# or with uv
uv pip install pyroboreplay

# verify
pyroboreplay --version

Requires Python 3.10+. From source (needs a Rust toolchain):

git clone ../PyRoboReplay
cd pyroboreplay
pip install maturin
maturin develop --release      # install the Python package
# or, for the Rust binary/crate only:
cargo build --release

MCAP export

Export a mission timeline to the indexed MCAP robotics log format. The native CLI needs no extra dependency; the Python helper needs mcap.

pyroboreplay export mission.db3 mission.mcap --format mcap
pip install "pyroboreplay[mcap]"
from pyroboreplay import Mission, export_mcap

mission = Mission.from_ros_bag("mission.db3")
export_mcap(mission, "mission.mcap")

Events are written to /events/<sensor> and failures to /failures as JSON messages with their original timestamps, so Foxglove/Rerun and other MCAP readers can open the recording. MCAP (MIT) is an optional dependency.

Rerun mission playback

Export a mission timeline and detected failures to a Rerun recording:

pip install "pyroboreplay[rerun]"
pyroboreplay-export mission.db3 mission.rrd --format rerun
from pyroboreplay import Mission, export_mission

mission = Mission.from_ros_bag("mission.db3")
export_mission(mission, "mission.rrd")
# rerun mission.rrd

Events are logged under events/<sensor> on a time timeline; failures are logged under failures with severity/confidence text. Rerun is an optional Apache-2.0 viewer dependency, not a vendored component.

Alpamayo forensic reports

Turn an offline Alpamayo run (.json, .jsonl, .npz, or a mapping) into a trajectory and reasoning report. Run timestamps are matched against failures already detected in the mission:

from pyroboreplay import (
    build_alpamayo_report,
    load_alpamayo_run,
    write_alpamayo_report,
)

run = load_alpamayo_run("alpamayo-run.json")
report = build_alpamayo_report(run, mission=mission, ground_truth=ego_future_xyz)
write_alpamayo_report(report, "alpamayo-report.md")

The report includes minADE/minFDE per run, per-sample reasoning traces, and correlated failures. No torch or model weights are required; generate the artifacts with the external Alpamayo inference stack.

AlpaSim ASL reports

Turn an AlpaSim rollout (.asl) into a provenance-rich JSON or Markdown report:

import asyncio

from pyroboreplay.asl import (
    build_asl_report,
    inspect_asl,
    read_asl,
    write_asl_report,
)

info = inspect_asl("rollout.asl")
rollout = asyncio.run(read_asl("rollout.asl"))
write_asl_report(build_asl_report(rollout), "alpasim-report.json")

read_asl() imports AlpaSim's upstream reader lazily; inspect_asl() validates the size-delimited framing with the standard library only. The adapter preserves session metadata, version IDs, actor tracks, and ego ground truth. It does not vendor AlpaSim, require a GPU, or infer safety outcomes from actor poses alone.

Usage

pyroboreplay compare mission_a.bag mission_b.bag        # side-by-side
pyroboreplay list mission.bag                            # topics in a bag
pyroboreplay analyze mission.bag --detect-gaps --detail  # findings to stdout
pyroboreplay export mission.bag mission.mcap --format mcap

# Phase 12/13: ingest external detections and a thermal frame
pyroboreplay forensics --yolo detections.json --dino dino.json \
  --rgb rgb.json --thermal thermal.json --format json

# Phase 15: Nav2 root-cause heuristics from a measurements document
pyroboreplay nav2 nav2-measurements.json --format json

# Phase 14: unified session from recorded sources
pyroboreplay session mission.bag --logs system.log --log-type syslog

# PyTerrainMap grid-map JSON export
pyroboreplay terrain terrain-grid.json --format json

The Python package exports Mission, Event, Failure, Hypothesis, RootCauseAnalysis, Action, FleetStatistics, GeoHotspot, plus load_pyterrainmap_grid, analyze_detection_gaps, fuse_rgb_thermal, analyze_nav2_diagnostics, build_navigation_session, and the Phase 16-20 JSON bridge (ops, run_op; see Advanced operations). World-knowledge and typed advanced-analysis APIs still live in the Rust core only.

For AI agents

Canonical facts for automated tooling and coding agents:

Fact Value
Package / CLI pyroboreplay / pyroboreplay + pyroboreplay-export
Python >=3.10
Core Rust/PyO3 extension; Rust crate also exposes a CLI
Main Python entry points Mission.from_ros_bag(), detect_failures(), analyze_failure(), analyze_detection_gaps(), build_alpamayo_report(), ops()
Supported ingest ROS 2 SQLite .db3 / .bag only (MCAP import not wired)
Source layout src/ (Rust), pyroboreplay/ (Python surface), tests/
Python tests pytest tests/ -v after maturin develop --release (local; not wired into CI)
Rust unit tests cargo test --lib (CI)
Rust integration tests cargo test --tests (present; not wired into CI)
Lint cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings (CI)
GPU needed? No for replay, analysis, adapters, or tests
Alpamayo / AlpaSim execution External; adapters parse artifacts only
SANA-WM Not integrated

Agent-friendly docs: docs/CLAUDE.md, docs/ARCHITECTURE.md, docs/API.md, docs/internal/ROADMAP_HONEST.md.

Documentation

Known issues and deferred work

Shipped:

  • Phases 12-15 exposed through the CLI and Python (forensics, session, nav2).
  • Six Phase 16-20 operations exposed through the JSON ops() / run_op() bridge (not typed Python APIs).
  • MCAP export in the native CLI; Rerun export via pyroboreplay-export.
  • Clippy is clean and gates CI (-D warnings).
  • PyTerrainMap pyterrainmap-gridmap v1 JSON reader.
  • AlpaSim ASL framing, decoded-entry mapping, and JSON/Markdown rollout reports.

Blocked / deferred:

  • Typed Python APIs for the remaining Phase 16-20 library types.
  • Full PyTerrainMap runtime integration (H3 store, ROS bridge).
  • MCAP mission import (adapter exists, not wired).
  • SANA-WM synthetic mission replay / provenance (not shipped).

Rejected / deferred:

  • Live fleet dashboard.

Full shipped/blocked/rejected detail: docs/internal/QUEUE_STATUS.md.

Contributing

Contributions welcome — see CONTRIBUTING.md for setup, conventions, and PR guidelines. Bugs and ideas: GitHub Issues; questions: GitHub Discussions.

Part of the robotics & autonomy family

PyRoboReplay is one of six complementary, composable tools in the Mullassery robotics & autonomy family. Together they form a single stack: perceive → map → simulate → learn → debug → coordinate the fleet.

Tool Role What it does
PyRoboVision Perception Multi-object tracker (Kalman filter + Hungarian algorithm) and trajectory predictor for robot perception.
PyTerrainMap Mapping / SLAM Fuses multi-robot terrain observations into a spatial-temporal store, with H3 spatial indexing and SLAM.
PyRoboSimulator Simulation World simulator for autonomous systems, with a FastAPI backend.
PyRoboFrames Learning data Rust-backed ML dataloader for robot-learning datasets (native LeRobot format, Apple Silicon hardware video decode).
PyRoboReplay Mission forensics This project — forensic-debugging and causal-analysis engine for autonomous robot mission replay.
SyncFleet Fleet coordination Rust + SQLite, peer-to-peer state sync for robotics fleets — mutual-TLS 1.3, version-vector replication, mDNS discovery; conflict resolution on the roadmap.

Each works standalone, but they're complementary by design. Browse all projects →

License

Apache License 2.0 — see LICENSE.

About

Offline forensics for ROS 2 robot missions: replay, root-cause analysis, reality gaps, RGB+thermal fusion, MCAP/Rerun export. NVIDIA Alpamayo + AlpaSim report adapters.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages