fix: ensure boot order is properly set before deployment reboot (#567) - #620
Open
skatsaounis wants to merge 1 commit into
Open
skatsaounis wants to merge 1 commit into
skatsaounis wants to merge 1 commit into
Conversation
…nical#567) On BMCs where MAAS controls the boot order (`can_set_boot_order`, notably IBM Z DPM), deployment could fail two ways: - The firmware can't fall through from netboot to local disk. If the guest rebooted before MAAS flipped the boot device, it network-IPLed into an empty config and hung (No value found for kernel, IPL failed 110). - IBM Z rejects writes with 409,2 busy while another partition op is in flight (fire-and-forget start/stop settling, post-install re-IPL), causing power/boot activities to fail intermittently. - `deploy.py`: MAAS-driven boot switch. For `can_set_boot_order` machines, replace the bare "set boot order + rely on installer reboot" with an explicit sequence in `_switch_to_local_boot`: power off → set boot order to disk → power on → _confirm_powered_on. The confirm step polls power state (bounded) and persists it, de-bouncing transient BMC/HMC readings (state must repeat on two consecutive polls before it's persisted). - `hmcz.py`: busy handling. New `_retry_on_busy` helper wraps all HMC writes (power on/off, boot-order updates), retrying only on 409,2 and re-raising everything else. `set_boot_order` also waits out transitional states before writing and now accepts `paused`/`terminated`/`degraded` (not just `stopped`/`active`) so it doesn't hang the full 120s. Retry budget is deliberately capped (8×15s ≈ 105s) to stay well under the 300s Temporal activity timeout it shares with `wait_for_status`. Resolves: LP:2158318 --------- Co-authored-by: fheimes <frank.heimes@canonical.com> (cherry picked from commit d6747ef)
Contributor
Author
|
Merge after #601 is merged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On BMCs where MAAS controls the boot order (
can_set_boot_order, notably IBM Z DPM), deployment could fail two ways:The firmware can't fall through from netboot to local disk. If the guest rebooted before MAAS flipped the boot device, it network-IPLed into an empty config and hung (No value found for kernel, IPL failed 110).
IBM Z rejects writes with 409,2 busy while another partition op is in flight (fire-and-forget start/stop settling, post-install re-IPL), causing power/boot activities to fail intermittently.
deploy.py: MAAS-driven boot switch. Forcan_set_boot_ordermachines, replace the bare "set boot order + rely on installer reboot" with an explicit sequence in_switch_to_local_boot: power off → set boot order to disk → power on → _confirm_powered_on. The confirm step polls power state (bounded) and persists it, de-bouncing transient BMC/HMC readings (state must repeat on two consecutive polls before it's persisted).hmcz.py: busy handling. New_retry_on_busyhelper wraps all HMC writes (power on/off, boot-order updates), retrying only on 409,2 and re-raising everything else.set_boot_orderalso waits out transitional states before writing and now acceptspaused/terminated/degraded(not juststopped/active) so it doesn't hang the full 120s. Retry budget is deliberately capped (8×15s ≈ 105s) to stay well under the 300s Temporal activity timeout it shares withwait_for_status.Resolves: LP:2158318
(cherry picked from commit d6747ef)