monitor: Configure systemd restart limits to better handle transient …#143
Open
mukilan wants to merge 1 commit into
Open
monitor: Configure systemd restart limits to better handle transient …#143mukilan wants to merge 1 commit into
mukilan wants to merge 1 commit into
Conversation
…GH issues. This patch changes the default restart limits so that the monitor service doesn't get stuck in a failed state by restarting too quickly and exhausting systemd's default rate limits. The new configuration will attempt to restart the service with an [exponential backoff] rate of 2.27 and a maximum of 5 minutes i.e. the restarts will be attempted in the following sequence 5s, 11.3s, 26s, 58.5s, 5m, 5m ... We also set a maximum of 10 restarts in 35 minutes so we don't restart indefinetly. This can be relaxed later if we find that it still doesn't help with recovery from intermittent GH issues. Potentially fixes: #112 Testing: Not tested. The code is based on my understanding of the systemd docs. [exponential backoff]: https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html?#RestartSteps= Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…GH issues.
This patch changes the default restart limits so that the monitor service doesn't get stuck in a failed state by restarting too quickly and exhausting systemd's default rate limits. The new configuration will attempt to restart the service with an exponential backoff rate of 2.27 and a maximum of 5 minutes i.e. the restarts will be attempted in the following sequence 5s, 11.3s, 26s, 58.5s, 2m 21s, 5m, 5m ...
We also set a maximum of 10 restarts in 35 minutes so we don't restart indefinetly. This can be relaxed later if we find that it still doesn't help with recovery from intermittent GH issues.
Potentially fixes: #112
Testing: Not tested. The code is based on my understanding of the systemd docs.