Ask a recorded robot mission why it failed. Replay ROS 2 missions, run causal / root-cause analysis, quantify the reality gap, and fuse RGB + thermal evidence. Rust core, Python bindings, CLI-first forensics.
Why it exists: robot logs tell you what happened. Incident reviews need why — which event preceded the failure, what perception missed, and what the robot should have detected. PyRoboReplay turns recorded missions into causal evidence you can diff, export, and defend.
pip install pyroboreplay
pyroboreplay replay mission.bag
pyroboreplay analyze mission.bag --detect-gaps --detail --format json --output investigation.jsonfrom pyroboreplay import Mission
mission = Mission.from_ros_bag("warehouse.bag")
failures = mission.detect_failures()
analysis = mission.analyze_failure(timestamp=1234567890.0)
print(len(failures))
print(analysis.get_primary_hypothesis())
print(f"{analysis.get_diagnostic_confidence():.0%}")Interactive replay keys: Space play/pause, n / → next event, p / ← previous event, ↑ / ↓ speed, ? help, q / Esc quit.
PyRoboReplay is NVIDIA-optional: replay, causal analysis, CLI, and tests run on CPU. NVIDIA appears where it matters — as a provenance-preserving adapter layer for physical-AI model and simulator runs.
| Component | Status | What it actually does |
|---|---|---|
| Alpamayo trajectory + reasoning reports | Available | Loads offline Alpamayo artifacts (.json, .jsonl, .npz, mapping), computes minADE/minFDE, preserves reasoning traces, and correlates failures already detected in the mission. No torch, no weights, no GPU. |
| AlpaSim ASL rollout reports | Available | Parses size-delimited .asl rollouts (lazy upstream reader), preserves session/version/actor/ego-ground-truth metadata, and emits JSON or Markdown. Does not vendor AlpaSim or infer safety from poses alone. |
| Alpamayo / AlpaSim model execution | External | Generate artifacts with the upstream NVIDIA stacks; PyRoboReplay analyzes the evidence. |
| SANA-WM synthetic mission replay | Not integrated | No SANA-WM loader, generator, or synthetic-scenario provenance ships today. Replay accepts recorded mission sources, not generated video. |
The honest headline: this is the forensic layer for NVIDIA physical-AI outputs, not an NVIDIA runtime. The adapters are real and tested; model execution and GPU requirements stay upstream.
| Area | Status |
|---|---|
ROS 2 SQLite .db3 / .bag mission ingest |
Supported |
| Interactive replay + timeline scrubber | Stable |
| Failure detection / root-cause hypotheses | Stable |
| Reality-gap analysis + JSON reports | Stable |
| Mission comparison + topic listing | Stable |
| Native MCAP export | Stable |
Rerun .rrd export |
Optional ([rerun]) |
| DINO detection-gap ingest, RGB + thermal fusion | Available (Phase 12/13) |
| Nav2 root-cause heuristics, navigation sessions | Available (Phase 14/15) |
| Alpamayo + AlpaSim report adapters | Available |
| Phase 16-20 ops | JSON bridge only (ops() / run_op()) |
| MCAP import | Not wired |
| Live fleet dashboard / streaming | Not a product |
| SANA-WM synthetic replay | Not shipped |
PyRoboReplay ingests recorded robot missions (ROS 2 bags) into a unified timeline and answers why and what-if questions rather than only where and what: which events preceded a failure, what the perception stack missed, and what the robot should have detected. The heavy lifting is a Rust core (cdylib + rlib) built with PyO3, exposed to Python through maturin. The supported entry points are the Python API and a Rust CLI with replay, compare, analyze, list, terrain, forensics, nav2, session, and export.
Exposed via CLI / Python (verified, tested):
- Mission replay — interactive timeline scrubber with keyboard controls (
replay). - Reality-gap analysis — probabilistic mismatch scoring, severity classification, evidence aggregation, JSON/text output (
analyze --detect-gaps). - Failure detection and diagnosis —
Mission.detect_failures(),Mission.analyze_failure()with hypotheses, confidence, and prioritized actions. - Mission comparison — side-by-side diff of two bags (
compare). - Topic listing — inspect available topics in a bag (
list). - Real ROS 2 CDR decoding — standard
sensor_msgs/nav_msgs/geometry_msgsbinary layouts are decoded from real message bytes (src/adapters/cdr.rs); poseconfidenceis derived from the real position-covariance trace, so localization-loss detection fires on real data. - Retrospective detection gaps (Phase 12) — ingest externally generated DINO detections and compare them with the robot's YOLO detections (
forensics --dino,analyze_detection_gaps()). - RGB + thermal fusion (Phase 13) — ingest RGB detections and a thermal frame, then fuse them (
forensics --rgb --thermal,fuse_rgb_thermal()). - Navigation sessions (Phase 14) — build a unified timeline from a recorded ROS 2 bag plus optional Linux logs and Nav2 exports (
session,build_navigation_session()). - Nav2 root-cause heuristics (Phase 15) — run localization/planner/costmap/obstacle/semantic analyzers and classification on a measurements document (
nav2,analyze_nav2_diagnostics()). - Recorded-mission exports — native MCAP (
export --format mcap) plus optional Rerun.rrdexport through the Python API orpyroboreplay-export. - PyTerrainMap grid interop — read the documented
pyterrainmap-gridmapv1 JSON export directly (terrain,load_pyterrainmap_grid()). - Phase 16-20 JSON operation bridge — six real Rust operations (causal graph, counterfactual query, fleet learning, temporal patterns, trend detection, decision tree) via
ops()/run_op(); see Advanced operations.
Implemented in the Rust core, not yet wired to CLI or Python (library API only):
- Persistent world knowledge (entities, locations, facts across missions) and spatial grounding.
- Multi-mission / longitudinal learning.
- Pluggable detection backends (YOLO / SAM / template fallback). Phase 12 accepts external DINO detections; it does not run a model itself.
- Warehouse and logistics fleets — reconstruct why a run failed, find missed detections and coverage gaps, compare two strategy revisions.
- Safety and compliance review — produce auditable, saved JSON findings for an incident (
analyze --detect-gaps --format json --output report.json). - Perception R&D — compare perception strategies over matched missions and quantify regressions.
- NVIDIA physical-AI teams — turn Alpamayo / AlpaSim runs into trajectory, reasoning, and provenance reports correlated with real mission failures.
- Field autonomy teams — investigate localization loss, planner oscillation, and sensor disagreement from recorded missions.
- Not a real-time monitoring or fleet dashboard. It works on recorded missions, not live streams. The
streamingmodules are internal building blocks, not a supported live fleet product. - Not an MCAP importer yet. MCAP export is supported; the Phase 14 MCAP adapter exists but is not wired into
Mission.from_ros_bag(). ROS 2 SQLite.db3/.bagis the supported ingest today. - Phases 16-20 are not typed Python APIs. Six implemented Rust operations are reachable through the JSON-only
ops()/run_op()bridge (Advanced operations); the rest of those modules, including decision reconstruction, clustering, scenario analysis, and rule extraction, remain Rust-library-only. - Phase 12 does not run DINO/SAM models. It ingests detections produced by an external open-vocabulary run and performs gap analysis; the in-crate
run_dino_detectionmethod remains a deterministic test stub. - Alpamayo / AlpaSim are not executed here. The adapters consume offline artifacts; model weights, licensing, and GPU requirements stay upstream.
- PyTerrainMap interop is file-based, not a runtime dependency. PyRoboReplay reads the documented
pyterrainmap-gridmapv1 JSON export; there is still no Cargo/pip dependency, H3 store integration, or ROS bridge.src/core/pyterrain_bridge.rsremains self-contained terrain-modeling logic. - Not published to crates.io — build the Rust crate from source if you need it directly.
- Not a certified or safety-verified system.
Beta (Development Status 4). Current source version is 2.13.1; the latest PyPI release is 2.13.1 (published 2026-10-23) and includes the Phase 16-20 operation bridge. The CLI and Python API are the tested, supported surface. 869 cargo test --lib unit tests pass plus public-surface Python tests (including 10 for the Phase 16-20 bridge) and 7 real-fixture ROS 2 CDR tests. cargo fmt --check and cargo clippy --all-targets --all-features -- -D warnings are clean (re-verified 2026-10-20). See docs/internal/ROADMAP_HONEST.md for the implemented/partial/missing breakdown.
Packaging: the 2.13.x wheel ships both the
pyroboreplayandpyroboreplay-exportconsole scripts and declaresnumpy, so installing the package is enough for both the Python API and the CLI.
Wheel coverage: PyPI publishes a macOS arm64 wheel plus an sdist; the sdist builds on other platforms. Release CI is configured to build and smoke-test Linux x86_64/aarch64, macOS arm64/x86_64, and Windows x86_64 for Python 3.10–3.13, and those wheels are produced when the release workflow's upload job runs. See the platform matrix.
pip install pyroboreplay
# or with uv
uv pip install pyroboreplay
# verify
pyroboreplay --versionRequires Python 3.10+. From source (needs a Rust toolchain):
git clone ../PyRoboReplay
cd pyroboreplay
pip install maturin
maturin develop --release # install the Python package
# or, for the Rust binary/crate only:
cargo build --releaseExport a mission timeline to the indexed MCAP robotics log format. The native
CLI needs no extra dependency; the Python helper needs mcap.
pyroboreplay export mission.db3 mission.mcap --format mcappip install "pyroboreplay[mcap]"from pyroboreplay import Mission, export_mcap
mission = Mission.from_ros_bag("mission.db3")
export_mcap(mission, "mission.mcap")Events are written to /events/<sensor> and failures to /failures as JSON
messages with their original timestamps, so Foxglove/Rerun and other MCAP
readers can open the recording. MCAP (MIT) is an optional dependency.
Export a mission timeline and detected failures to a Rerun recording:
pip install "pyroboreplay[rerun]"pyroboreplay-export mission.db3 mission.rrd --format rerunfrom pyroboreplay import Mission, export_mission
mission = Mission.from_ros_bag("mission.db3")
export_mission(mission, "mission.rrd")
# rerun mission.rrdEvents are logged under events/<sensor> on a time timeline; failures are
logged under failures with severity/confidence text. Rerun is an optional
Apache-2.0 viewer dependency, not a vendored component.
Turn an offline Alpamayo run (.json, .jsonl, .npz, or a mapping) into a
trajectory and reasoning report. Run timestamps are matched against failures
already detected in the mission:
from pyroboreplay import (
build_alpamayo_report,
load_alpamayo_run,
write_alpamayo_report,
)
run = load_alpamayo_run("alpamayo-run.json")
report = build_alpamayo_report(run, mission=mission, ground_truth=ego_future_xyz)
write_alpamayo_report(report, "alpamayo-report.md")The report includes minADE/minFDE per run, per-sample reasoning traces, and
correlated failures. No torch or model weights are required; generate the
artifacts with the external Alpamayo inference stack.
Turn an AlpaSim rollout (.asl) into a provenance-rich JSON or Markdown report:
import asyncio
from pyroboreplay.asl import (
build_asl_report,
inspect_asl,
read_asl,
write_asl_report,
)
info = inspect_asl("rollout.asl")
rollout = asyncio.run(read_asl("rollout.asl"))
write_asl_report(build_asl_report(rollout), "alpasim-report.json")read_asl() imports AlpaSim's upstream reader lazily; inspect_asl() validates
the size-delimited framing with the standard library only. The adapter preserves
session metadata, version IDs, actor tracks, and ego ground truth. It does not
vendor AlpaSim, require a GPU, or infer safety outcomes from actor poses alone.
pyroboreplay compare mission_a.bag mission_b.bag # side-by-side
pyroboreplay list mission.bag # topics in a bag
pyroboreplay analyze mission.bag --detect-gaps --detail # findings to stdout
pyroboreplay export mission.bag mission.mcap --format mcap
# Phase 12/13: ingest external detections and a thermal frame
pyroboreplay forensics --yolo detections.json --dino dino.json \
--rgb rgb.json --thermal thermal.json --format json
# Phase 15: Nav2 root-cause heuristics from a measurements document
pyroboreplay nav2 nav2-measurements.json --format json
# Phase 14: unified session from recorded sources
pyroboreplay session mission.bag --logs system.log --log-type syslog
# PyTerrainMap grid-map JSON export
pyroboreplay terrain terrain-grid.json --format jsonThe Python package exports Mission, Event, Failure, Hypothesis, RootCauseAnalysis, Action, FleetStatistics, GeoHotspot, plus load_pyterrainmap_grid, analyze_detection_gaps, fuse_rgb_thermal, analyze_nav2_diagnostics, build_navigation_session, and the Phase 16-20 JSON bridge (ops, run_op; see Advanced operations). World-knowledge and typed advanced-analysis APIs still live in the Rust core only.
Canonical facts for automated tooling and coding agents:
| Fact | Value |
|---|---|
| Package / CLI | pyroboreplay / pyroboreplay + pyroboreplay-export |
| Python | >=3.10 |
| Core | Rust/PyO3 extension; Rust crate also exposes a CLI |
| Main Python entry points | Mission.from_ros_bag(), detect_failures(), analyze_failure(), analyze_detection_gaps(), build_alpamayo_report(), ops() |
| Supported ingest | ROS 2 SQLite .db3 / .bag only (MCAP import not wired) |
| Source layout | src/ (Rust), pyroboreplay/ (Python surface), tests/ |
| Python tests | pytest tests/ -v after maturin develop --release (local; not wired into CI) |
| Rust unit tests | cargo test --lib (CI) |
| Rust integration tests | cargo test --tests (present; not wired into CI) |
| Lint | cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings (CI) |
| GPU needed? | No for replay, analysis, adapters, or tests |
| Alpamayo / AlpaSim execution | External; adapters parse artifacts only |
| SANA-WM | Not integrated |
Agent-friendly docs: docs/CLAUDE.md, docs/ARCHITECTURE.md, docs/API.md, docs/internal/ROADMAP_HONEST.md.
- Documentation index — start here
- Install — Python API vs CLI and platform matrix
- Advanced operations — Phase 16-20 JSON bridge
- Alpamayo and AlpaSim workflow — trajectory evidence and ASL logs
- Project status — current gaps and fixes
- Alternatives — Foxglove/Rerun comparison
- docs/CLAUDE.md — product vision and architecture
- docs/API.md — Python and Rust API reference
- docs/ARCHITECTURE.md — phase-by-phase architecture
- docs/QUICKSTART.md — getting started
- docs/STATS_DASHBOARD.md — live metrics dashboard (
replay --stats-dashboard) - examples/ — runnable demos (
cargo run --example root_cause_analysis_demo) - docs/internal/ROADMAP_HONEST.md — implemented vs planned, technical debt
- CHANGELOG.md — notable changes
Shipped:
- Phases 12-15 exposed through the CLI and Python (
forensics,session,nav2). - Six Phase 16-20 operations exposed through the JSON
ops()/run_op()bridge (not typed Python APIs). - MCAP export in the native CLI; Rerun export via
pyroboreplay-export. - Clippy is clean and gates CI (
-D warnings). - PyTerrainMap
pyterrainmap-gridmapv1 JSON reader. - AlpaSim ASL framing, decoded-entry mapping, and JSON/Markdown rollout reports.
Blocked / deferred:
- Typed Python APIs for the remaining Phase 16-20 library types.
- Full PyTerrainMap runtime integration (H3 store, ROS bridge).
- MCAP mission import (adapter exists, not wired).
- SANA-WM synthetic mission replay / provenance (not shipped).
Rejected / deferred:
- Live fleet dashboard.
Full shipped/blocked/rejected detail: docs/internal/QUEUE_STATUS.md.
Contributions welcome — see CONTRIBUTING.md for setup, conventions, and PR guidelines. Bugs and ideas: GitHub Issues; questions: GitHub Discussions.
PyRoboReplay is one of six complementary, composable tools in the Mullassery robotics & autonomy family. Together they form a single stack: perceive → map → simulate → learn → debug → coordinate the fleet.
| Tool | Role | What it does |
|---|---|---|
| PyRoboVision | Perception | Multi-object tracker (Kalman filter + Hungarian algorithm) and trajectory predictor for robot perception. |
| PyTerrainMap | Mapping / SLAM | Fuses multi-robot terrain observations into a spatial-temporal store, with H3 spatial indexing and SLAM. |
| PyRoboSimulator | Simulation | World simulator for autonomous systems, with a FastAPI backend. |
| PyRoboFrames | Learning data | Rust-backed ML dataloader for robot-learning datasets (native LeRobot format, Apple Silicon hardware video decode). |
| PyRoboReplay | Mission forensics | This project — forensic-debugging and causal-analysis engine for autonomous robot mission replay. |
| SyncFleet | Fleet coordination | Rust + SQLite, peer-to-peer state sync for robotics fleets — mutual-TLS 1.3, version-vector replication, mDNS discovery; conflict resolution on the roadmap. |
Each works standalone, but they're complementary by design. Browse all projects →
Apache License 2.0 — see LICENSE.