Fix: Add validations to suspend_trigger (BugFix)

[EstebanVg15]

Description

The following jobs were executed as part of the NVIDIA Riverside Stress test plans.

1. Job 1
2. Job 2
3. Job 3

As can be seen the following error is commonly displayed.

Failed to suspend system via logind: There's already a shutdown or sleep operation in progress
Running: rtcwake --verbose --device /dev/rtc0 --mode no --seconds 60
Running: systemctl suspend to suspend the system
Traceback (most recent call last):
  File "/tmp/nest-l6gc9hfq.539cd33af75a03c6be23305220f4d9eb9436c26514c6041892099f42a977f924/suspend_trigger.py", line 85, in <module>
     sys.exit(main())
   File "/tmp/nest-l6gc9hfq.539cd33af75a03c6be23305220f4d9eb9436c26514c6041892099f42a977f924/suspend_trigger.py", line 75, in main
     subprocess.check_call(suspend_cmd)
   File "/snap/checkbox22/current/usr/lib/python3.10/subprocess.py", line 369, in check_call
     raise CalledProcessError(retcode, cmd)
 subprocess.CalledProcessError: Command '['systemctl', 'suspend']' returned non-zero exit status 1.
 --------------------------------------------------------------------------------
 Outcome: job passed

Even though the suspend command failed, the job is marked as Outcome: job passed, and the error is printed cyclically in further iterations.

This PR proposes some validations for the suspend_trigger.py script to make the stress-tests/suspend_cycles_{{suspend_id}}_reboot{{suspend_reboot_id}} test cases more robust.

• Run systemctl list-jobs *suspend* to check if there are suspend jobs running before running one more.
• If systemctl list-jobs *suspend* detects suspend jobs running, the job will wait before proceeding.
• Set set -o pipefail on the Checkbox job definition for the job to be marked as "failed" when exiting on error.
• Update the corresponding Python unit tests to support the systemctl list-jobs *suspend* calls.

Resolved issues

https://warthogs.atlassian.net/browse/PERI-1367

Documentation

Tests

I ran the com.canonical.certification::suspend-cycles-stress-test test plan two times on a nvidia-jetson-orin-nano DUT.

1. On this Checkbox submission there can be seen cases, such as the stress-tests/suspend_cycles_27_reboot1 there where suspend jobs still on-going when trying to suspend the device once more, but the test case waited until being able to send the corresponding commands.

2. I've set a maximum 40 seconds delay (spitted in 10 maximum attempts with a 4 second delay between each one). On the attached tests there were 300 suspend/resume cycles.

• 248 suspend/resume cycles ran properly at the first attempt.
• 31 suspend/resume cycles needed 2 reattempts (+8 seconds) to run.
• 18 suspend/resume cycles needed 3 reattempts (+12 seconds) to run.
• 2 suspend/resume cycles needed 4 reattempts (+16 seconds) to run.
• 1 suspend/resume cycle needed 5 reattempts (+20 seconds) to run.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.93%. Comparing base (aed87f5) to head (0915934).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2500   +/-   ##
=======================================
  Coverage   58.92%   58.93%           
=======================================
  Files         476      476           
  Lines       48031    48050   +19     
  Branches     8574     8579    +5     
=======================================
+ Hits        28303    28317   +14     
- Misses      18835    18840    +5     
  Partials      893      893           

Flag	Coverage Δ
checkbox-ng	76.04% <ø> (-0.02%)	⬇️
checkbox-support	67.90% <ø> (ø)
provider-base	34.14% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pieqq]

I only skimmed through the PR, but I have a suggestion.

[EstebanVg15]

Hello @pieqq ,

I've digging more into this issue, please review my findings.

When running sudo systemctl suspend, the command returns the control to the shell immediately before the hardware actually powers down. This latency allows subsequent instructions to be executed during the transition to a sleep state, below can be seen that there are some messages being printed before the system enters suspend mode.

ubuntu@ubuntu:~$ sudo rtcwake -d "/dev/rtc0" -m no -s 5; sudo systemctl suspend; for i in {1..1000}; do date_now=$(date +"%d-%m-%y %H:%M:%S"); echo "iteration: $i $date_now"; done
rtcwake: assuming RTC uses UTC ...
rtcwake: wakeup using /dev/rtc0 at Mon Jun  1 21:36:17 2026
iteration: 1 01-06-26 21:36:10
iteration: 2 01-06-26 21:36:10
iteration: 3 01-06-26 21:36:10
iteration: 4 01-06-26 21:36:10
iteration: 5 01-06-26 21:36:10
iteration: 6 01-06-26 21:36:10
iteration: 7 01-06-26 21:36:10
iteration: 8 01-06-26 21:36:10
iteration: 9 01-06-26 21:36:10
iteration: 10 01-06-26 21:36:10
iteration: 11 01-06-26 21:36:10
iteration: 12 01-06-26 21:36:10
iteration: 13 01-06-26 21:36:10
(the system enters suspend here and after resuming prints the remaining traces)

The issue has been isolated through the attached scripts (scripts.zip), I recorded this video for illustrative purposes (suspend-tests-video.zip), The helper run-suspend.sh cyclically calls suspend.py (mimicking Checkbox). And as can be seen on the video, when this is done without polling systemctl list-jobs *suspend* , it regularly triggers the reported error: Failed to suspend system via logind: There's already a shutdown or sleep operation in progress. This occurs because a new suspend request is queued while a previous one is still processing.

In Checkbox, this issue appears sporadically during stress tests due to the following execution timing.

1. Checkbox calls a shell script on providers/base/units/stress/suspend_cycles_reboot.pxu.
2. The suspend_trigger.py Python helper is called.
3. The helper executes sudo systemctl suspend.
4. Steps 1-3 are called cyclically until completing test plan execution.

Iterating between steps 1-3 sometimes gives enough time to the system to be suspended, but as as reported on this PR, the issues are sometimes visible.

Therefore, implementing a polling mechanism via systemctl list-jobs *suspend* ensures the previous job finishes before Checkbox triggers the next cycle.