Repository navigation
Add ROS 2 Message Flow Tracing #13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from all commits
Commits
Show all changes
817 commits
Select commit
Hold shift + click to select a range
a423392
add lanelet2-route-planning
lreiher 1a46496
add script to render compose files by resolving oci include
lreiher ec86458
heavily simplify render script
lreiher 4961efc
improve folder structure
lreiher a84ef96
optimize auto-gen comment block
lreiher 404599a
Restructure compose source rendering
lreiher 8b453ac
keep original values commented in rendered overridden compose
lreiher 13e1a80
add comment block to input files
lreiher de6b28e
remove non-published stack services
lreiher bfc34fe
add github and gitlab ci to check for changes after calling rendering…
lreiher 3485164
add basic flowchart to readme
lreiher ee723e3
rename oci input files
lreiher 8745684
regenerate with newest oci compose templates, incl map server
lreiher f23aec5
default params file to /params.yml
lreiher 7d367e7
simplify docker-compose-essentials
lreiher 7d889cc
update docker-compose-essentials
lreiher adf5fbe
remove docker-compose-essentials submodule
lreiher b896fb8
restore docker-compose-essentials@969f5aa246f907a2398d2c8107000cba1ad…
lreiher 93abc3d
fix hidden nodes in readme graph
lreiher 137fe74
add lanelet2_object_list_prediction and ackermann_trajectory_control
cgeller 4844073
cleanup
cgeller 8a454b9
add monitoring service
RaphvK 13b1fd9
update oci artifacts
cgeller c3116ce
update lanelet2_route_planning to v1.1.1
RaphvK f2e885f
add first version of documentation
cgeller aaef133
cleanup
cgeller ba263a7
cleanup overview
cgeller 596be3e
add simple-object-tracking
cgeller a32f0d6
add simple-planner
cgeller d584998
update demo
cgeller 06723b3
update demo
cgeller cb71663
update images
jpbusch 4366c0b
update essentials to latest main
jpbusch 0ca97d1
Merge branch 'openads-controller' into 'main'
jpbusch 9b7f5b6
update image
cgeller 2a5b5b8
update images
cgeller aa6b7aa
update stack
cgeller a8ac807
update ackermann_trajectory_control to v1.5.0
gkueppers 3fb201a
update predicted object list topic
cgeller b736453
update ego-state estimation image
gkueppers d9bddbc
fix version of optimization
jpbusch 90bc40a
use tagged image for ego state estimation
jpbusch ff7c2c4
Merge branch 'update-ackermann-control' into 'main'
gkueppers bec74fd
update lanelet2_map_server
cgeller ab86765
Merge branch 'main' into openadstack
cgeller 951f09c
update prediction
cgeller 2e952cd
cleanup
cgeller 00fa9ae
add point_cloud_fusion
cgeller 4a1f27c
add reference_trajectory_planning
cgeller 8723679
update simple_object_tracking
cgeller b6fdd44
small updates
cgeller 1b98b47
add point cloud object detection
cgeller 5a2aa33
add autoware-multi-object-tracker
cgeller ba26da4
update rviz config
cgeller 9159bb3
update object list prediction
cgeller 969069c
add route trigger
cgeller 25d941f
update demo
cgeller 6ccf3ea
update perception demo
cgeller 73b76fb
temporary disable detection
cgeller bbc7195
use gpu for pcod
cgeller 0803677
manually fix image version for optimization
cgeller a3bde0e
add triton-server
cgeller 28c87f0
move docker-compose-essentials and scripts to utils
cgeller e579ea5
cleanup
cgeller c4c5305
update perception demo
cgeller c6436db
fix pcod params
cgeller d4360c8
add license file
RaphvK 91b5c5a
smaller intermediate arrow
RaphvK 45f9a64
revert /params.yml specification
cgeller f85fc08
cleanup demo
cgeller 81d434a
fix demo
cgeller 7c1e241
use markdown alerts for sphinx
RaphvK c1f286e
update readme
cgeller 9388664
update readme
cgeller 8ad772b
update usage
cgeller 315edca
update technical architecture
cgeller 9a339ea
add autoware_tracker params
cgeller 4ca3e93
fix links
RaphvK 7933704
use gif as teaser
RaphvK 35296f8
cleanup demo
cgeller d5916a1
move replay logic to launch file
RaphvK ce282a8
add ros2_graph_export
RaphvK 87b41e3
update simple-planner
cgeller 0f29deb
update readme
RaphvK 72aaf0e
update functional architecture
RaphvK 57ca60f
Merge remote-tracking branch 'origin/openadstack' into openadstack
RaphvK 7551025
add a model
RaphvK fa675ab
update documentation
cgeller 68642cd
cleanup definitions
cgeller 8e67ff9
update technical archtecture
cgeller 93b5a92
update readme
RaphvK 041fdb1
Merge remote-tracking branch 'origin/openadstack' into openadstack
RaphvK 8e27aed
remove links
RaphvK 817a321
update demo
cgeller d718d9b
cleanup
cgeller 5fd3690
restructuring
cgeller feaa691
reset autoware tracking
cgeller 4637ed2
fix autoware tracking
cgeller 260f614
remove simple_object_tracking
cgeller 6198bbd
improve teaser images in markdown
RaphvK 5613ea0
add ros2 graph export to usage guide
RaphvK 2164cb6
fix autoware_multi_object_tracker integration
RaphvK 4e11295
exclude ros nodes from graph export
RaphvK d93ce0e
update autoware_multi_object_tracker
cgeller 7eefa57
update rviz path
cgeller d00d3ee
pcod video teaser
RaphvK 3b6ad46
Apply 1 suggestion(s) to 1 file(s)
RaphvK 4bc5b9b
replace a model
RaphvK 61c9a19
restore license
RaphvK 715b7b1
remove etsi_its_messages from docs
RaphvK bb91ec9
update point cloud object detection
cgeller 065467f
upda pcod mdoel
cgeller 2d86bef
remove teasers, update package descriptions
RaphvK e2aaefd
re-generate graph_export
RaphvK f84e174
remove download hint
RaphvK 9bbf404
fix demo
cgeller f537b05
update ros2 graph export
RaphvK f0ffccf
use ros2_graph_export release tag
RaphvK fbad000
update integration and deployment documentation
cgeller d39f187
update citation
cgeller 6c0d3cd
update service description
cgeller 88b73ad
improve oci rendering
jpbusch d46b58e
update documentation
cgeller de6eaa2
add teaser
cgeller 09274fa
update video
cgeller ac7341d
update teaser
cgeller d8aabac
update documentation
cgeller 19c5b6f
add default rmw router and session configs from jazzy branch fb9c070
lreiher 4f3ce93
transfer zenoh config overrides to config files
lreiher ce47a5b
make env vars more readable in compose essentials
lreiher 6993352
sync zenoh config with docker-compose-essentials
lreiher 9b3c708
switch repo check after rendering in ci to git porcelain
lreiher 0d1a648
fix readme
cgeller d23e084
add demo data bag notes
cgeller 9662cb8
update re
cgeller 74ed3ae
update documentation
cgeller c9b67cd
update demo
cgeller 8a1dc46
add rosbag conversion
cgeller 87b734b
update conversion
cgeller 045bb19
Update docs
cgeller 1439b4f
update demo data
cgeller 4497032
update bag file sizes
cgeller fd2d1cb
rename images
cgeller ffd6b2c
update rack
cgeller c439b45
temporary add simple_object_tracking
cgeller 6686520
update pcodt parameters
bastilam f01028e
update demo requirements for GPU compatibility
bastilam 7e13713
[demo] configure planning and prediction services
jpbusch 973f5a1
combine demo compose files in one and use profiles
jpbusch 8022af3
remove tmp simple tracking
jpbusch c05487a
update youtube video
cgeller aa30fb2
add route planning params
jpbusch d397360
update demo rviz config; fix startup behavior
jpbusch 8d32d7c
simplify topic remapping
cgeller 4bd9eef
update upload-asset
cgeller 10b76fe
use video html
cgeller 3cbbd22
readd ZENOH_ROUTER_CHECK_ATTEMPTS, so nodes don't start doing stuff b…
lreiher ad0ca4e
remove control from demo
cgeller 27cee14
improve demo restart
cgeller 790ec41
bump trajectory optimization
cgeller 98dca96
temporary use latest for optimization
cgeller ba9ef06
improve getting started
cgeller 9fff045
fix teaser video in sphinx docs
RaphvK d83f094
Revert "fix teaser video in sphinx docs"
RaphvK 20ccbaa
use sciebo link
RaphvK 1770f93
Revert "use sciebo link"
RaphvK 26592fa
revert trajectory_frame_id in simple_planner
cgeller 89711cf
update ci workflow
cgeller 1ce97a6
fix urls
cgeller 8e7f9a0
use documentation for openads references
cgeller 3ce9294
update references
cgeller de56eed
Merge branch 'openadstack' into 'main'
cgeller fa1a62c
use markdown links in top-level readme and add docs rebuild downstrea…
RaphvK 6e4e3e6
update links of karl paper
jpbusch 5bc1896
update OpenADS badge
jpbusch e579d97
Merge pull request #5 from openads-project/update-karl-paper
jpbusch 6e51596
handle new compose paths in render script
jpbusch a98272d
update simple_planner, trajectory_optimization and prediction services
jpbusch a295d5d
move triton server and zenoh-router to template services instead of t…
jpbusch f36cebb
update optimization service in demo
jpbusch 5f65525
move triton-server CUDA_VISIBLE_DEVICES to template
cgeller 2e88671
make compose rendering work with gitlab oci artifacts
jpbusch 2e620ad
apply service prefix to redered compose file
jpbusch 156052b
fix logs in compose rendering
jpbusch 0b2487b
update triton server image
jpbusch 7e647e6
use slim image for pcod
jpbusch 8f80201
generate rendered service name from ROS namespace + name
jpbusch ff672a1
use slim image for pcod in override service
jpbusch 4d76dfb
render all compose files
jpbusch 852838e
use jazzy-ros-base-alternative-rmw image for zenoh-router
gkueppers c6ad095
switch back to ZENOH_CONFIG_OVERRIDE 🤡
lreiher e781f1a
fix zenoh-router health check after image swap
lreiher c3cf201
fix width of teaser video
RaphvK 10923b3
add description to compose renderer
jpbusch 889af4b
update simple planner params
jpbusch cb3f0c5
sort triton server service in demo compose
jpbusch aef0208
Merge pull request #6 from openads-project/update-services
jpbusch 4c3289f
simplify model volume mount
cgeller aeb12ba
handle linebreaks in compose renderer
jpbusch 81a931b
fix model mounting
cgeller ef4a435
update model image name
cgeller 27d7065
Merge pull request #7 from openads-project/update/model-provisioning
cgeller 34219d4
add planning architecture paper to publications
jpbusch 8f35694
fix typo in ZENOH_ROUTER_CHECK_ATTEMPTS variable
lreiher 6c1a28b
Merge pull request #9 from openads-project/typo-zenoh-config
lreiher a11d182
updated docker-compose to prioritise nvidia gpu's if multiple are ava…
MikkelDLR 279c42d
Add architecture roadmap (#8)
RaphvK ca2deac
modified readme to include xhost command
MikkelDLR 03e809d
Apply suggestions from code review
cgeller 2837633
Merge pull request #10 from MikkelDLR/bugfix/rviz_dual_gpu_visualization
cgeller dc8976c
update ResearchGate link
jpbusch dac5a15
remove git-attributes
cgeller c715f1e
Merge pull request #11 from openads-project/update-publications
jpbusch 04378ec
simplify credential usage in render compose script; make it reliable …
jpbusch d205947
fix random / missing blank lines generated compose
jpbusch d389200
Merge pull request #12 from openads-project/improve-render-compose
jpbusch ba99121
add tracing folder and docs
RaphvK 3ba3053
add tracing folder and docs
RaphvK b01eebf
use tracing branches for images
RaphvK b1ec391
use tracing branches for images
RaphvK 89bbf0a
add tracing script
RaphvK aaf6f2a
add tracing script
RaphvK a80a661
merge tracing into main images
RaphvK 27acab6
use tracing image for tracking
RaphvK 8c114fc
Merge remote-tracking branch 'origin/main' into tracing
RaphvK bd0e7b9
Merge branch 'tracing' of github.com:openads-project/openadstack into…
RaphvK 8a7274c
use latest image
RaphvK 0e84607
fix urdf path
RaphvK 93a379b
improve start instruction
RaphvK 8c67f6b
update version of ackermann_trajectory_control
RaphvK ab9cf69
exclude removed files
RaphvK 863df2c
update override files
RaphvK adfeba4
revert compose generation script
RaphvK 16c620f
fix override files
RaphvK a191e0f
re-render compose files
RaphvK 75c0d79
fix using tracing images
RaphvK c2fe879
remove file from repo
RaphvK eee29eb
improve tracing documentatino
RaphvK b2c860e
link to github repo
RaphvK fc37697
switch to released images
RaphvK 9fabc80
use released image
RaphvK 0937906
use image from public repo
RaphvK 00cef31
Merge remote-tracking branch 'origin/main' into tracing
RaphvK cb0d89b
revert change
RaphvK 9388233
use IMAGE_VARIANT env to disable slim image during tracing
RaphvK 8b137d6
do not use slim images
RaphvK 5012b91
do not mention kubernetes
RaphvK 96c1a34
use oci compose file
RaphvK 06e0c8c
Apply batched suggestions from code review
RaphvK 9aa60c9
Merge branch 'tracing' of github.com:openads-project/openadstack into…
RaphvK File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
RaphvK marked this conversation as resolved.
Resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # System Analysis | ||
|
|
||
| OpenADS is a distributed system: its modules run as separate ROS 2 nodes in individual containers and exchange data via topics. This makes it hard to tell, from the outside, how long a message takes to travel from sensor input to control output, or where time is lost along the way. This page describes how to record runtime traces of the stack with [ros2_tracing](https://github.com/ros2/ros2_tracing) and how to analyze them, so you can inspect timing and message flow across all modules of the automated driving stack. | ||
|
|
||
| ## Why Trace the Message Flow? | ||
|
|
||
| Tracing records when callbacks run and when messages are published and received, across all traced nodes and over time. It helps you to: | ||
|
|
||
| - Pinpoint latency bottlenecks and timing jitter. | ||
| - Correlate perception, planning, and control decisions with their sensor inputs. | ||
| - Debug race conditions and missing messages across nodes. | ||
| - Detect performance regressions after changes. | ||
|
|
||
| ## How to Trace the Automated Driving Stack | ||
|
|
||
| 1. Start the stack with tracing enabled by setting the `ROS_TRACING` environment variable: | ||
|
|
||
| ```bash | ||
| export ROS_TRACING=true | ||
| docker compose up -d | ||
| ``` | ||
|
|
||
| 2. Use OpenADStack as usual. Trace data is continuously recorded into an in-memory ring buffer, but not yet written to disk. | ||
|
|
||
| 3. Start capturing a trace snapshot to disk: | ||
|
|
||
| ```bash | ||
| utils/tracing/tracing.sh start | ||
|
RaphvK marked this conversation as resolved.
Resolved
|
||
| ``` | ||
|
|
||
| 4. Once you have captured the situation of interest, stop capturing: | ||
|
|
||
| ```bash | ||
| utils/tracing/tracing.sh stop | ||
| ``` | ||
|
|
||
| The trace data of each container is copied to `utils/tracing/trace/<timestamp>/<container>/` and removed from the containers. | ||
|
|
||
| 5. Start the ROS 2 trace analysis tools: | ||
|
|
||
| ```bash | ||
| docker compose -f utils/tracing/docker-compose.yml up | ||
|
cgeller marked this conversation as resolved.
Resolved
|
||
| ``` | ||
|
|
||
| The `utils/tracing/trace/` folder is mounted into the analysis container at `/trace`. | ||
|
|
||
| ### Traced Services | ||
|
|
||
| By default, `tracing.sh` traces all running containers whose name contains one of the following: | ||
|
|
||
| - `perception.point-cloud-fusion` | ||
| - `perception.point-cloud-object-detection` | ||
| - `understanding.autoware-multi-object-tracker` | ||
| - `understanding.lanelet2-object-list-prediction` | ||
| - `planning.simple-planner` | ||
| - `planning.trajectory-optimization` | ||
| - `control.ackermann-trajectory-control` | ||
|
|
||
| Additional containers can be traced by passing their (partial) names via `EXTRA_TRACING_CONTAINERS`: | ||
|
RaphvK marked this conversation as resolved.
Resolved
|
||
|
|
||
| ```bash | ||
| EXTRA_TRACING_CONTAINERS="my-container-1 my-other-container-1" utils/tracing/tracing.sh start | ||
| ``` | ||
|
|
||
| > [!NOTE] | ||
| > Additional containers must support tracing themselves, i.e., their nodes must be launched with tracing enabled. | ||
|
|
||
| ## How to Analyze Trace Data | ||
|
|
||
| The analysis container provides [Eclipse Trace Compass](https://eclipse.dev/tracecompass/) and Jupyter Notebooks for evaluating the recorded traces. See [ros2-trace-analysis](https://github.com/openads-project/ros2-trace-analysis) for usage details. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cgeller marked this conversation as resolved.
Resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| include: | ||
| - oci://ghcr.io/openads-project/ros2-trace-analysis/compose:v1.0.0 | ||
|
|
||
| services: | ||
|
|
||
| ros2-trace-analysis: | ||
| extends: | ||
| file: ../compose/docker-compose.template.yml | ||
| service: x11-service | ||
| volumes: | ||
| - ./trace:/trace |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| * | ||
| !.gitignore |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,223 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Usage: ./tracing.sh {start|stop} | ||
|
|
||
| if [ $# -ne 1 ]; then | ||
| echo "Usage: $0 {start|stop}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| ACTION=$1 | ||
| TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S) | ||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
|
|
||
| # List of container name substrings to trace | ||
| # Default container substrings | ||
| DEFAULT_CONTAINER_SUBSTRINGS=( | ||
| "perception.point-cloud-fusion" | ||
| "perception.point-cloud-object-detection" | ||
| "understanding.autoware-multi-object-tracker" | ||
| "understanding.lanelet2-object-list-prediction" | ||
| "planning.simple-planner" | ||
| "planning.trajectory-optimization" | ||
| "control.ackermann-trajectory-control" | ||
| ) | ||
|
|
||
| # Allow additional container substrings via EXTRA_TRACING_CONTAINERS env var | ||
| # e.g. EXTRA_TRACING_CONTAINERS="my-container-1 my-other-container-1" ./tracing.sh start | ||
| EXTRA_SUBSTRINGS=() | ||
| if [ -n "$EXTRA_TRACING_CONTAINERS" ]; then | ||
| IFS=' ' read -ra EXTRA_SUBSTRINGS <<< "$EXTRA_TRACING_CONTAINERS" | ||
| fi | ||
|
|
||
| CONTAINER_SUBSTRINGS=("${DEFAULT_CONTAINER_SUBSTRINGS[@]}" "${EXTRA_SUBSTRINGS[@]}") | ||
|
|
||
| DOCKER_USER="dockeruser" | ||
| TRACE_ROOT="/home/dockeruser/.ros/tracing" | ||
|
|
||
| format_target_label() { | ||
| local target_type=$1 | ||
| local target_namespace=$2 | ||
| local target_name=$3 | ||
|
|
||
| if [ "$target_type" == "docker" ]; then | ||
| echo "$target_name" | ||
| else | ||
| echo "$target_namespace/$target_name" | ||
| fi | ||
| } | ||
|
|
||
| format_target_output_dir() { | ||
| local target_type=$1 | ||
| local target_namespace=$2 | ||
| local target_name=$3 | ||
|
|
||
| if [ "$target_type" == "docker" ]; then | ||
| echo "$target_name" | ||
| else | ||
| echo "k8s-${target_namespace}-${target_name}" | ||
| fi | ||
| } | ||
|
|
||
| exec_in_target() { | ||
| local target_type=$1 | ||
| local target_namespace=$2 | ||
| local target_name=$3 | ||
| local command=$4 | ||
|
|
||
| if [ "$target_type" == "docker" ]; then | ||
| docker exec --user "$DOCKER_USER" "$target_name" bash -ic "$command" | ||
| else | ||
| kubectl exec -n "$target_namespace" "$target_name" -- bash -ic "$command" | ||
| fi | ||
| } | ||
|
|
||
| copy_trace_from_target() { | ||
| local target_type=$1 | ||
| local target_namespace=$2 | ||
| local target_name=$3 | ||
| local destination_dir=$4 | ||
|
|
||
| if [ "$target_type" == "docker" ]; then | ||
| docker cp "$target_name:$TRACE_ROOT/trace" "$destination_dir" | ||
| else | ||
| kubectl cp "$target_namespace/$target_name:/root/.ros/tracing/trace" "$destination_dir" | ||
| fi | ||
| } | ||
|
|
||
| # Arrays to track results | ||
| SUCCESS_TARGETS=() | ||
| FAILED_TARGETS=() | ||
| MISSING_TARGETS=() | ||
|
|
||
| # Validate action | ||
| if [ "$ACTION" != "start" ] && [ "$ACTION" != "stop" ]; then | ||
| echo "Usage: $0 {start|stop}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| mapfile -t RUNNING_CONTAINERS < <(docker ps --format '{{.Names}}' 2>/dev/null) | ||
| mapfile -t RUNNING_PODS < <(kubectl get pods -A --field-selector=status.phase=Running --no-headers -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name 2>/dev/null) | ||
|
|
||
| for substring in "${CONTAINER_SUBSTRINGS[@]}"; do | ||
| MATCHED_TARGETS=() | ||
|
|
||
| for container in "${RUNNING_CONTAINERS[@]}"; do | ||
| if [[ "$container" == *"$substring"* ]]; then | ||
| MATCHED_TARGETS+=("docker||$container") | ||
| fi | ||
| done | ||
|
|
||
| for pod_entry in "${RUNNING_PODS[@]}"; do | ||
| read -r namespace pod_name <<< "$pod_entry" | ||
| if [[ "$pod_name" == *"$substring"* ]]; then | ||
| MATCHED_TARGETS+=("k8s|$namespace|$pod_name") | ||
| fi | ||
| done | ||
|
|
||
| if [ ${#MATCHED_TARGETS[@]} -eq 0 ]; then | ||
| echo "Warning: No running target containing '$substring'." | ||
| MISSING_TARGETS+=("$substring") | ||
| continue | ||
| fi | ||
|
|
||
| for target in "${MATCHED_TARGETS[@]}"; do | ||
| target_type=${target%%|*} | ||
| target_remainder=${target#*|} | ||
| target_namespace=${target_remainder%%|*} | ||
| target_name=${target_remainder#*|} | ||
| ( | ||
| TARGET_LABEL=$(format_target_label "$target_type" "$target_namespace" "$target_name") | ||
|
|
||
| if [ "$ACTION" == "start" ]; then | ||
| echo "Starting tracing in target: $TARGET_LABEL" | ||
|
|
||
| TRACE_COMMAND="ros2 trace start trace --dual-session" | ||
| exec_in_target "$target_type" "$target_namespace" "$target_name" "$TRACE_COMMAND" | ||
|
|
||
| if [ $? -eq 0 ]; then | ||
| echo "$TARGET_LABEL" >> /tmp/tracing_success_$$ | ||
| else | ||
| echo "$TARGET_LABEL" >> /tmp/tracing_failed_$$ | ||
| fi | ||
|
|
||
| elif [ "$ACTION" == "stop" ]; then | ||
| echo "Stopping tracing in target: $TARGET_LABEL" | ||
| STOP_COMMAND="ros2 trace stop trace --dual-session" | ||
| exec_in_target "$target_type" "$target_namespace" "$target_name" "$STOP_COMMAND" | ||
|
|
||
| STOP_STATUS=$? | ||
|
|
||
| echo "Copy trace files from target: $TARGET_LABEL" | ||
| TARGET_OUTPUT_DIR="$SCRIPT_DIR/trace/$TIMESTAMP/$(format_target_output_dir "$target_type" "$target_namespace" "$target_name")" | ||
| mkdir -p "$TARGET_OUTPUT_DIR" | ||
| copy_trace_from_target "$target_type" "$target_namespace" "$target_name" "$TARGET_OUTPUT_DIR" | ||
|
|
||
| COPY_STATUS=$? | ||
|
|
||
| echo "Removing trace files from target: $TARGET_LABEL" | ||
| REMOVE_COMMAND="rm -rf $TRACE_ROOT" | ||
| exec_in_target "$target_type" "$target_namespace" "$target_name" "$REMOVE_COMMAND" | ||
|
|
||
| REMOVE_STATUS=$? | ||
|
|
||
| if [ $STOP_STATUS -eq 0 ] && [ $COPY_STATUS -eq 0 ] && [ $REMOVE_STATUS -eq 0 ]; then | ||
| echo "$TARGET_LABEL" >> /tmp/tracing_success_$$ | ||
| else | ||
| echo "$TARGET_LABEL" >> /tmp/tracing_failed_$$ | ||
| fi | ||
| fi | ||
| ) & | ||
| done | ||
| done | ||
|
|
||
| # Wait for all background jobs to complete | ||
| wait | ||
|
|
||
| # Read results from temporary files | ||
| if [ -f /tmp/tracing_success_$$ ]; then | ||
| mapfile -t SUCCESS_TARGETS < /tmp/tracing_success_$$ | ||
| rm /tmp/tracing_success_$$ | ||
| fi | ||
|
|
||
| if [ -f /tmp/tracing_failed_$$ ]; then | ||
| mapfile -t FAILED_TARGETS < /tmp/tracing_failed_$$ | ||
| rm /tmp/tracing_failed_$$ | ||
| fi | ||
|
|
||
| # Print summary | ||
| echo "" | ||
| echo "============================================" | ||
| echo "Summary - $ACTION operation:" | ||
| echo "============================================" | ||
|
|
||
| if [ ${#SUCCESS_TARGETS[@]} -gt 0 ]; then | ||
| echo "✓ Successfully ${ACTION}ed tracing for ${#SUCCESS_TARGETS[@]} target(s):" | ||
| for target in "${SUCCESS_TARGETS[@]}"; do | ||
| echo " - $target" | ||
| done | ||
| else | ||
| echo "✓ No targets ${ACTION}ed successfully" | ||
| fi | ||
|
|
||
| echo "" | ||
|
|
||
| if [ ${#FAILED_TARGETS[@]} -gt 0 ]; then | ||
| echo "✗ Failed to $ACTION tracing for ${#FAILED_TARGETS[@]} target(s):" | ||
| for target in "${FAILED_TARGETS[@]}"; do | ||
| echo " - $target" | ||
| done | ||
| else | ||
| echo "✗ No failures" | ||
| fi | ||
|
|
||
| echo "" | ||
|
|
||
| if [ ${#MISSING_TARGETS[@]} -gt 0 ]; then | ||
| echo "⚠ Missing ${#MISSING_TARGETS[@]} target(s) (not running):" | ||
| for substring in "${MISSING_TARGETS[@]}"; do | ||
| echo " - $substring" | ||
| done | ||
| fi | ||
|
|
||
| echo "=============================================" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.