CASMTRIAGE-9122: Flood of SQUASHFS errors seen on compute nodes and i… by aasha-hpe · Pull Request #6550 · Cray-HPE/docs-csm

aasha-hpe · 2026-04-14T20:06:41Z

Description

This is the WAR document for the iSCSI issue (CASMTRIAGE-9122) seen after worker node rebuild during the upgrade.

If I added any command snippets, the steps they belong to follow the prompt conventions (see [example][1]).
If I added a new directory, I also updated .github/CODEOWNERS with the corresponding team in [Cray-HPE][2].
My commits or Pull-Request Title contain my JIRA information, or I do not have a JIRA.

…SCSI multipathing is broken after worker node rebuild during upgrade

which has automated steps to logout iSCSI sesion, do iscsiadm discovery and login to iSCSI session.

jpdavis-prof · 2026-04-22T18:53:39Z

+CONFIG_FILE="/etc/iscsi/iscsid.conf"
+#CONFIG_FILE="/root/Asha/iscsid.conf"


Remove debug artifact. Leftover development path should not ship.

Suggested change

CONFIG_FILE="/etc/iscsi/iscsid.conf"

#CONFIG_FILE="/root/Asha/iscsid.conf"

CONFIG_FILE="/etc/iscsi/iscsid.conf"

jpdavis-prof · 2026-04-22T18:53:40Z

+cp "$CONFIG_FILE" "${CONFIG_FILE}.bak"
+
+# Set iscsid.safe_logout value 'No'
+sed -i 's/^\(\s*iscsid\.safe_logout\s*=\s*\)[Yy][Ee][Ss]/\1No/' "$CONFIG_FILE"


Variable brace consistency. Use ${CONFIG_FILE} everywhere — line 43 already uses ${CONFIG_FILE}.bak.

Suggested change

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$[Yy][Ee][Ss]/\1No/' "$CONFIG_FILE"

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$[Yy][Ee][Ss]/\1No/' "${CONFIG_FILE}"

jpdavis-prof · 2026-04-22T18:53:40Z

+PORTAL=$(iscsiadm -m session | grep $NCN_WORKER | awk '{print $3}' | sed 's/3260.*/3260/')
+IQN=$(iscsiadm -m session | grep $NCN_WORKER | awk '{print $4}')


Bug: PORTAL/IQN can be empty or multi-line. If the worker name doesn't match any session, both variables are empty and subsequent iscsiadm commands produce cryptic errors. Also, grep $NCN_WORKER matches worker10 when searching for worker1. Quote and brace all variables, and validate.

Suggested change

PORTAL=$(iscsiadm -m session | grep $NCN_WORKER | awk '{print $3}' | sed 's/3260.*/3260/')

IQN=$(iscsiadm -m session | grep $NCN_WORKER | awk '{print $4}')

PORTAL=$(iscsiadm -m session | grep "${NCN_WORKER}" | awk '{print $3}' | sed 's/3260.*/3260/')

IQN=$(iscsiadm -m session | grep "${NCN_WORKER}" | awk '{print $4}')

if [[ -z "${PORTAL}" ]] || [[ -z "${IQN}" ]]; then

echo "Error: No iSCSI session found for worker ${NCN_WORKER}"

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "${CONFIG_FILE}"

exit 1

fi

jpdavis-prof · 2026-04-22T18:53:40Z

+
+# Logout the iSCSI session
+
+iscsiadm -m node -T $IQN -p $PORTAL -u


Quote and brace variables to prevent word splitting and stay consistent.

Suggested change

iscsiadm -m node -T $IQN -p $PORTAL -u

iscsiadm -m node -T "${IQN}" -p "${PORTAL}" -u

jpdavis-prof · 2026-04-22T18:53:40Z

+  echo "Logging out of iSCSI session with $NCN_WORKER failed, so exiting by resetting iscsid.safe_logout to 'Yes' "
+  sed -i 's/^\(\s*iscsid\.safe_logout\s*=\s*\)No/\1Yes/' "$CONFIG_FILE"


Brace consistency in the error path.

Suggested change

echo "Logging out of iSCSI session with $NCN_WORKER failed, so exiting by resetting iscsid.safe_logout to 'Yes' "

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "$CONFIG_FILE"

echo "Logging out of iSCSI session with ${NCN_WORKER} failed, so exiting by resetting iscsid.safe_logout to 'Yes'"

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "${CONFIG_FILE}"

jpdavis-prof · 2026-04-22T18:53:40Z

+# Perform iscsiadm discovery
+
+iscsiadm -m discovery -t sendtargets -p $PORTAL
+
+# Login to iSCSI session
+
+iscsiadm -m node -T $IQN -p $PORTAL -l


Bug: No error checking on discovery or login. If either fails (network issue, target not ready), the script exits 0 — the operator thinks it worked but the iSCSI session is broken.

Suggested change

# Perform iscsiadm discovery

iscsiadm -m discovery -t sendtargets -p $PORTAL

# Login to iSCSI session

iscsiadm -m node -T $IQN -p $PORTAL -l

# Perform iscsiadm discovery

iscsiadm -m discovery -t sendtargets -p "${PORTAL}"

if [ $? -ne 0 ]; then

echo "Error: Discovery failed for portal ${PORTAL}"

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "${CONFIG_FILE}"

exit 1

fi

# Login to iSCSI session

iscsiadm -m node -T "${IQN}" -p "${PORTAL}" -l

if [ $? -ne 0 ]; then

echo "Error: Login failed for ${IQN} at ${PORTAL}"

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "${CONFIG_FILE}"

exit 1

fi

jpdavis-prof · 2026-04-22T18:53:40Z

+
+# Set back iscsid.safe_logout from 'No' to 'Yes'
+
+sed -i 's/^\(\s*iscsid\.safe_logout\s*=\s*\)No/\1Yes/' "$CONFIG_FILE"


Brace consistency.

Suggested change

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "$CONFIG_FILE"

sed -i 's/^$\s*iscsid\.safe_logout\s*=\s*$No/\1Yes/' "${CONFIG_FILE}"

github-actions · 2026-05-13T11:27:52Z

This pull-request has not had activity in over 20 days and is being marked as stale.

ravikanth-nalla-hpe · 2026-06-02T11:33:16Z

+[`iscsi_post_rollout.sh`](../../scripts/operations/iscsi_sbps/iscsi_post_rollout.sh) and it is required
+to copy the script onto iSCSI initiator (Compute/UAN) nodes and run using `pdsh` command from the master
+node:
+


Reword it as:

The solution is to re-establish the iSCSI connection after a worker node rebuild by logging out of the existing iSCSI session, rediscovering the LUNs, and logging back in using the iscsiadm command. These steps are automated in a script [iscsi_post_rollout.sh (../../scripts/operations/iscsi_sbps/iscsi_post_rollout.sh) and must be copied to the iSCSI initiator nodes (Compute/UAN) and executed from the master node using the pdsh command.

How about this text:

The solution is to re-establish the iSCSI session after a worker node rebuild by logging out the existing
stale iSCSI session with rebuilt worker node, rediscover the LUNs and logging in back using the iscsiadm
command. The steps are automated the script iscsi_post_rollout.sh and must be copied to the iSCSI initiator nodes (Compute/UAN) and executed from the master node using
pdsh command.

Please reword "The solution is to re-establish the iSCSI session after a worker node rebuild by logging out the existing stale iSCSI session with rebuilt worker node,..." as it is confusing after and with rebuild context here.

"The steps are automated in the script"

Please remove "and executed from the master node using pdsh command." as it is very confusing.

Reword below....

"Run the script scp_iscsi_scr.sh to copy iscsi_post_rollout.sh onto compute nodes. Then to run iscsi_post_rollout.sh on compute nodes:"

to something like below...

Steps 1: Run *src.h
Note: <copies *rollout.sh script to computes>
Step 2: : Run *rollout.sh

ravikanth-nalla-hpe · 2026-06-11T06:18:57Z

+iSCSI client nodes became `un-responsive` where most of the commands failed with
+`Bus error`.
+
+Example command:


Add node type prefix as per the standard through out the doc.

ravikanth-nalla-hpe · 2026-06-11T06:35:21Z

+
+# This is the script to copy iscsi_post_rollout.sh to all compute nodes.
+
+#!/bin/bash


ravikanth-nalla-hpe · 2026-06-11T06:36:41Z

+))
+
+for alias in "${aliases[@]}"; do
+  echo "$alias-nmn"


Remove echo statement.

ravikanth-nalla-hpe · 2026-06-11T06:37:57Z

+
+for alias in "${aliases[@]}"; do
+  echo "$alias-nmn"
+  scp iscsi_post_rollout.sh root@$alias-nmn:/root/


Please avoid copying to "/" o or "/root", instead use "/tmp" or your own new dir created.

ravikanth-nalla-hpe · 2026-06-11T06:38:57Z

+# shellcheck disable=SC2207
+
+aliases=($(
+  sat status --filter role=compute \


As you have mentioned please take care for UAN too.

aasha-hpe added 3 commits April 14, 2026 20:01

CASMTRIAGE-9122: Flood of SQUASHFS errors seen on compute nodes and i…

fe9f44e

…SCSI multipathing is broken after worker node rebuild during upgrade

Fixed spelling errors

0842aa5

Fixed spelling

ad5571b

aasha-hpe requested review from harishkhpe, rajeshranganathan-hpe and ravikanth-nalla-hpe April 14, 2026 20:25

Moving the doc to iscsi_sbps folder

f3b9219

aasha-hpe requested review from a team as code owners April 16, 2026 06:32

aasha-hpe requested review from GunashekarKM, SavioJoshua, atifsyedali, brookshire, don-bahls-hpe, haroldlongley, jnowicki-hpe, kkelling-hpe, nrockershousen and sfkramer April 16, 2026 06:32

aasha-hpe added 4 commits April 20, 2026 12:14

Added ../../scripts/operations/iscsi_sbps/iscsi_pre-requiste.sh script

e562f5c

which has automated steps to logout iSCSI sesion, do iscsiadm discovery and login to iSCSI session.

Fixing shfmt errors

4827a1e

Fixed shfmt errors

e5e1c9b

Fixed shfmt errors

ce555d4

jpdavis-prof reviewed Apr 22, 2026

View reviewed changes

aasha-hpe and others added 3 commits April 23, 2026 07:48

Script review comments addressed

4b10374

Merge branch 'release/1.7' into CASMTRIAGE-9122-WAR

4cf806a

Fixed shfmt error

9085650

rajeshranganathan-hpe approved these changes Apr 23, 2026

View reviewed changes

github-actions Bot added the Stale Hasn't had activity in over 30 days label May 13, 2026

ravikanth-nalla-hpe added 3 commits June 1, 2026 09:11

Modifications done to doc/script based on test results

e4f6d36

Fixed formatting errors and edited text

518a694

Fixed formatting errors

a04f082

github-actions Bot removed the Stale Hasn't had activity in over 30 days label Jun 1, 2026