diff --git a/README.md b/README.md index 92f70c98..a2df8956 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,7 @@ The documentation contains: - [Functional Architecture](./docs/functional-architecture.md) - [Integration Architecture](./docs/integration-architecture.md) - [Deployment Composition](./docs/deployment-composition.md) +- [System Analysis](./docs/system-analysis.md) ## 🙏 Acknowledgements diff --git a/docs/system-analysis.md b/docs/system-analysis.md new file mode 100644 index 00000000..6958659c --- /dev/null +++ b/docs/system-analysis.md @@ -0,0 +1,70 @@ +# System Analysis + +OpenADS is a distributed system: its modules run as separate ROS 2 nodes in individual containers and exchange data via topics. This makes it hard to tell, from the outside, how long a message takes to travel from sensor input to control output, or where time is lost along the way. This page describes how to record runtime traces of the stack with [ros2_tracing](https://github.com/ros2/ros2_tracing) and how to analyze them, so you can inspect timing and message flow across all modules of the automated driving stack. + +## Why Trace the Message Flow? + +Tracing records when callbacks run and when messages are published and received, across all traced nodes and over time. It helps you to: + +- Pinpoint latency bottlenecks and timing jitter. +- Correlate perception, planning, and control decisions with their sensor inputs. +- Debug race conditions and missing messages across nodes. +- Detect performance regressions after changes. + +## How to Trace the Automated Driving Stack + +1. Start the stack with tracing enabled by setting the `ROS_TRACING` environment variable: + + ```bash + export ROS_TRACING=true + docker compose up -d + ``` + +2. Use OpenADStack as usual. Trace data is continuously recorded into an in-memory ring buffer, but not yet written to disk. + +3. Start capturing a trace snapshot to disk: + + ```bash + utils/tracing/tracing.sh start + ``` + +4. Once you have captured the situation of interest, stop capturing: + + ```bash + utils/tracing/tracing.sh stop + ``` + + The trace data of each container is copied to `utils/tracing/trace///` and removed from the containers. + +5. Start the ROS 2 trace analysis tools: + + ```bash + docker compose -f utils/tracing/docker-compose.yml up + ``` + + The `utils/tracing/trace/` folder is mounted into the analysis container at `/trace`. + +### Traced Services + +By default, `tracing.sh` traces all running containers whose name contains one of the following: + +- `perception.point-cloud-fusion` +- `perception.point-cloud-object-detection` +- `understanding.autoware-multi-object-tracker` +- `understanding.lanelet2-object-list-prediction` +- `planning.simple-planner` +- `planning.trajectory-optimization` +- `control.ackermann-trajectory-control` + +Additional containers can be traced by passing their (partial) names via `EXTRA_TRACING_CONTAINERS`: + +```bash +EXTRA_TRACING_CONTAINERS="my-container-1 my-other-container-1" utils/tracing/tracing.sh start +``` + +> [!NOTE] +> Additional containers must support tracing themselves, i.e., their nodes must be launched with tracing enabled. + +## How to Analyze Trace Data + +The analysis container provides [Eclipse Trace Compass](https://eclipse.dev/tracecompass/) and Jupyter Notebooks for evaluating the recorded traces. See [ros2-trace-analysis](https://github.com/openads-project/ros2-trace-analysis) for usage details. diff --git a/perception/point_cloud_object_detection/.docker-compose.oci-overrides.yml b/perception/point_cloud_object_detection/.docker-compose.oci-overrides.yml index 616f2b4c..152f438a 100644 --- a/perception/point_cloud_object_detection/.docker-compose.oci-overrides.yml +++ b/perception/point_cloud_object_detection/.docker-compose.oci-overrides.yml @@ -10,7 +10,6 @@ services: extends: file: ../../utils/compose/docker-compose.template.yml service: ros2-gpu-service - image: ghcr.io/openads-project/point_cloud_object_detection:v3.1.1-slim environment: # --- name ------ NAMESPACE: /perception diff --git a/perception/point_cloud_object_detection/docker-compose.yml b/perception/point_cloud_object_detection/docker-compose.yml index 3dcfcbcc..65b1e3af 100644 --- a/perception/point_cloud_object_detection/docker-compose.yml +++ b/perception/point_cloud_object_detection/docker-compose.yml @@ -10,8 +10,7 @@ services: extends: file: ../../utils/compose/docker-compose.template.yml service: ros2-gpu-service - # image: ghcr.io/openads-project/point_cloud_object_detection:v3.1.1 - image: ghcr.io/openads-project/point_cloud_object_detection:v3.1.1-slim + image: ghcr.io/openads-project/point_cloud_object_detection:v3.1.1 environment: # --- name ------ # NAMESPACE: / diff --git a/understanding/autoware_multi_object_tracker/.docker-compose.oci-overrides.yml b/understanding/autoware_multi_object_tracker/.docker-compose.oci-overrides.yml index 4f7f2bd2..ff95653c 100644 --- a/understanding/autoware_multi_object_tracker/.docker-compose.oci-overrides.yml +++ b/understanding/autoware_multi_object_tracker/.docker-compose.oci-overrides.yml @@ -2,7 +2,7 @@ # `../../utils/scripts/render_compose_sources.py` include: - - oci://ghcr.io/thinking-cars/autoware_multi_object_tracker/compose:v1.1.1 + - oci://ghcr.io/thinking-cars/autoware_multi_object_tracker/compose:v1.2.0 services: diff --git a/understanding/autoware_multi_object_tracker/docker-compose.yml b/understanding/autoware_multi_object_tracker/docker-compose.yml index 9b8b607c..da060fd4 100644 --- a/understanding/autoware_multi_object_tracker/docker-compose.yml +++ b/understanding/autoware_multi_object_tracker/docker-compose.yml @@ -1,8 +1,8 @@ # Auto-generated by `../../utils/scripts/render_compose_sources.py` # based on `.docker-compose.oci-overrides.yml` with resolved includes # include: -# # https://github.com/thinking-cars/autoware_multi_object_tracker/blob/v1.1.1/deployment/compose/docker-compose.yml -# - oci://ghcr.io/thinking-cars/autoware_multi_object_tracker/compose:v1.1.1 +# # https://github.com/thinking-cars/autoware_multi_object_tracker/blob/v1.2.0/deployment/compose/docker-compose.yml +# - oci://ghcr.io/thinking-cars/autoware_multi_object_tracker/compose:v1.2.0 services: @@ -10,7 +10,7 @@ services: extends: file: ../../utils/compose/docker-compose.template.yml service: ros2-service - image: ghcr.io/thinking-cars/autoware_multi_object_tracker:v1.1.1 + image: ghcr.io/thinking-cars/autoware_multi_object_tracker:v1.2.0 environment: # --- name ------ # NAMESPACE: / @@ -55,6 +55,7 @@ services: INPUT_CHANNELS_PATH: "" PUBLISH_MERGED_OBJECTS: false EGO_SOURCE: tf + ROS_TRACING: ${ROS_TRACING:-false} command: - /bin/bash - -ic @@ -81,6 +82,7 @@ services: $${INPUT_CHANNELS_PATH:+input_channels_path:=$${INPUT_CHANNELS_PATH}} \ publish_merged_objects:=$${PUBLISH_MERGED_OBJECTS} \ ego_source:=$${EGO_SOURCE} \ + ros_tracing:=$${ROS_TRACING} \ input_detection01_objects:=$${INPUT_DETECTION01_OBJECTS} \ input_detection02_objects:=$${INPUT_DETECTION02_OBJECTS} \ input_detection03_objects:=$${INPUT_DETECTION03_OBJECTS} \ diff --git a/utils/tracing/docker-compose.yml b/utils/tracing/docker-compose.yml new file mode 100644 index 00000000..853cc719 --- /dev/null +++ b/utils/tracing/docker-compose.yml @@ -0,0 +1,11 @@ +include: + - oci://ghcr.io/openads-project/ros2-trace-analysis/compose:v1.0.0 + +services: + + ros2-trace-analysis: + extends: + file: ../compose/docker-compose.template.yml + service: x11-service + volumes: + - ./trace:/trace diff --git a/utils/tracing/trace/.gitignore b/utils/tracing/trace/.gitignore new file mode 100644 index 00000000..d6b7ef32 --- /dev/null +++ b/utils/tracing/trace/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore diff --git a/utils/tracing/tracing.sh b/utils/tracing/tracing.sh new file mode 100755 index 00000000..d891e5f9 --- /dev/null +++ b/utils/tracing/tracing.sh @@ -0,0 +1,223 @@ +#!/bin/bash + +# Usage: ./tracing.sh {start|stop} + +if [ $# -ne 1 ]; then + echo "Usage: $0 {start|stop}" + exit 1 +fi + +ACTION=$1 +TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S) +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# List of container name substrings to trace +# Default container substrings +DEFAULT_CONTAINER_SUBSTRINGS=( + "perception.point-cloud-fusion" + "perception.point-cloud-object-detection" + "understanding.autoware-multi-object-tracker" + "understanding.lanelet2-object-list-prediction" + "planning.simple-planner" + "planning.trajectory-optimization" + "control.ackermann-trajectory-control" +) + +# Allow additional container substrings via EXTRA_TRACING_CONTAINERS env var +# e.g. EXTRA_TRACING_CONTAINERS="my-container-1 my-other-container-1" ./tracing.sh start +EXTRA_SUBSTRINGS=() +if [ -n "$EXTRA_TRACING_CONTAINERS" ]; then + IFS=' ' read -ra EXTRA_SUBSTRINGS <<< "$EXTRA_TRACING_CONTAINERS" +fi + +CONTAINER_SUBSTRINGS=("${DEFAULT_CONTAINER_SUBSTRINGS[@]}" "${EXTRA_SUBSTRINGS[@]}") + +DOCKER_USER="dockeruser" +TRACE_ROOT="/home/dockeruser/.ros/tracing" + +format_target_label() { + local target_type=$1 + local target_namespace=$2 + local target_name=$3 + + if [ "$target_type" == "docker" ]; then + echo "$target_name" + else + echo "$target_namespace/$target_name" + fi +} + +format_target_output_dir() { + local target_type=$1 + local target_namespace=$2 + local target_name=$3 + + if [ "$target_type" == "docker" ]; then + echo "$target_name" + else + echo "k8s-${target_namespace}-${target_name}" + fi +} + +exec_in_target() { + local target_type=$1 + local target_namespace=$2 + local target_name=$3 + local command=$4 + + if [ "$target_type" == "docker" ]; then + docker exec --user "$DOCKER_USER" "$target_name" bash -ic "$command" + else + kubectl exec -n "$target_namespace" "$target_name" -- bash -ic "$command" + fi +} + +copy_trace_from_target() { + local target_type=$1 + local target_namespace=$2 + local target_name=$3 + local destination_dir=$4 + + if [ "$target_type" == "docker" ]; then + docker cp "$target_name:$TRACE_ROOT/trace" "$destination_dir" + else + kubectl cp "$target_namespace/$target_name:/root/.ros/tracing/trace" "$destination_dir" + fi +} + +# Arrays to track results +SUCCESS_TARGETS=() +FAILED_TARGETS=() +MISSING_TARGETS=() + +# Validate action +if [ "$ACTION" != "start" ] && [ "$ACTION" != "stop" ]; then + echo "Usage: $0 {start|stop}" + exit 1 +fi + +mapfile -t RUNNING_CONTAINERS < <(docker ps --format '{{.Names}}' 2>/dev/null) +mapfile -t RUNNING_PODS < <(kubectl get pods -A --field-selector=status.phase=Running --no-headers -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name 2>/dev/null) + +for substring in "${CONTAINER_SUBSTRINGS[@]}"; do + MATCHED_TARGETS=() + + for container in "${RUNNING_CONTAINERS[@]}"; do + if [[ "$container" == *"$substring"* ]]; then + MATCHED_TARGETS+=("docker||$container") + fi + done + + for pod_entry in "${RUNNING_PODS[@]}"; do + read -r namespace pod_name <<< "$pod_entry" + if [[ "$pod_name" == *"$substring"* ]]; then + MATCHED_TARGETS+=("k8s|$namespace|$pod_name") + fi + done + + if [ ${#MATCHED_TARGETS[@]} -eq 0 ]; then + echo "Warning: No running target containing '$substring'." + MISSING_TARGETS+=("$substring") + continue + fi + + for target in "${MATCHED_TARGETS[@]}"; do + target_type=${target%%|*} + target_remainder=${target#*|} + target_namespace=${target_remainder%%|*} + target_name=${target_remainder#*|} + ( + TARGET_LABEL=$(format_target_label "$target_type" "$target_namespace" "$target_name") + + if [ "$ACTION" == "start" ]; then + echo "Starting tracing in target: $TARGET_LABEL" + + TRACE_COMMAND="ros2 trace start trace --dual-session" + exec_in_target "$target_type" "$target_namespace" "$target_name" "$TRACE_COMMAND" + + if [ $? -eq 0 ]; then + echo "$TARGET_LABEL" >> /tmp/tracing_success_$$ + else + echo "$TARGET_LABEL" >> /tmp/tracing_failed_$$ + fi + + elif [ "$ACTION" == "stop" ]; then + echo "Stopping tracing in target: $TARGET_LABEL" + STOP_COMMAND="ros2 trace stop trace --dual-session" + exec_in_target "$target_type" "$target_namespace" "$target_name" "$STOP_COMMAND" + + STOP_STATUS=$? + + echo "Copy trace files from target: $TARGET_LABEL" + TARGET_OUTPUT_DIR="$SCRIPT_DIR/trace/$TIMESTAMP/$(format_target_output_dir "$target_type" "$target_namespace" "$target_name")" + mkdir -p "$TARGET_OUTPUT_DIR" + copy_trace_from_target "$target_type" "$target_namespace" "$target_name" "$TARGET_OUTPUT_DIR" + + COPY_STATUS=$? + + echo "Removing trace files from target: $TARGET_LABEL" + REMOVE_COMMAND="rm -rf $TRACE_ROOT" + exec_in_target "$target_type" "$target_namespace" "$target_name" "$REMOVE_COMMAND" + + REMOVE_STATUS=$? + + if [ $STOP_STATUS -eq 0 ] && [ $COPY_STATUS -eq 0 ] && [ $REMOVE_STATUS -eq 0 ]; then + echo "$TARGET_LABEL" >> /tmp/tracing_success_$$ + else + echo "$TARGET_LABEL" >> /tmp/tracing_failed_$$ + fi + fi + ) & + done +done + +# Wait for all background jobs to complete +wait + +# Read results from temporary files +if [ -f /tmp/tracing_success_$$ ]; then + mapfile -t SUCCESS_TARGETS < /tmp/tracing_success_$$ + rm /tmp/tracing_success_$$ +fi + +if [ -f /tmp/tracing_failed_$$ ]; then + mapfile -t FAILED_TARGETS < /tmp/tracing_failed_$$ + rm /tmp/tracing_failed_$$ +fi + +# Print summary +echo "" +echo "============================================" +echo "Summary - $ACTION operation:" +echo "============================================" + +if [ ${#SUCCESS_TARGETS[@]} -gt 0 ]; then + echo "✓ Successfully ${ACTION}ed tracing for ${#SUCCESS_TARGETS[@]} target(s):" + for target in "${SUCCESS_TARGETS[@]}"; do + echo " - $target" + done +else + echo "✓ No targets ${ACTION}ed successfully" +fi + +echo "" + +if [ ${#FAILED_TARGETS[@]} -gt 0 ]; then + echo "✗ Failed to $ACTION tracing for ${#FAILED_TARGETS[@]} target(s):" + for target in "${FAILED_TARGETS[@]}"; do + echo " - $target" + done +else + echo "✗ No failures" +fi + +echo "" + +if [ ${#MISSING_TARGETS[@]} -gt 0 ]; then + echo "⚠ Missing ${#MISSING_TARGETS[@]} target(s) (not running):" + for substring in "${MISSING_TARGETS[@]}"; do + echo " - $substring" + done +fi + +echo "============================================="