Workflow failure: Setup SSH fails

[anton-seaice]

Sometimes with repro workflows there are intermittent failures where Setup SSH fails

e.g. from https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/24299835552/job/70951235137

Run for host in $HOSTS; do
  for host in $HOSTS; do
    ssh-keyscan $host >> ~/.ssh/known_hosts
  done
  chmod 600 ~/.ssh/known_hosts
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    ARTIFACT_LOCAL_LOCATION: /opt/artifact
    EXPERIMENT_LOCATION: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/schedule/dev-preindustrial+emissions/a7ad7679f735de3d50c5effbfd370701307c5a47
    HOSTS: ***
  ***
  
Error: Process completed with exit code 1.

These are minor errors, but do lead to annoying false negatives in scheduled tests. Typically it passes on re-run

[anton-seaice]

I wonder if a retry is possible, or waiting longer helps ?

[anton-seaice]

Here is another example:

https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/24375666239/job/71189261565

[anton-seaice]

And https://github.com/ACCESS-NRI/access-om3-configs/actions/runs/24494989738/job/71589008127

[jo-basevi]

Ah no, it's not doing great.. I think the first attempt could be to add to a retry to the ssh-keyscan in the action.

https://github.com/ACCESS-NRI/actions/blob/06fb21867433f6c82add1ee40b53ecc38666851c/.github/actions/setup-ssh/action.yml#L88

[anton-seaice]

Getting so many of these !

https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/24622206333/job/71995247705

[jo-basevi]

I've just merged in a change which adds a retry directly to the setup-ssh action. Let me know if it makes no difference and it's still erroring at that step

[anton-seaice]

Is ACCESS-NRI/build-cd#377 the same issue ? Maybe they use the same action ? (unless the action is versioned ?)

[anton-seaice]

@jo-basevi - still seeting these (but in build-cd now!)

[jo-basevi]

Ok looks like it is retrying ssh-keyscan at least so that confirms that is where the issue is

Run access-nri/actions/.github/actions/setup-ssh@main
Run mkdir -p ~/.ssh
Run path="$HOME/.ssh/$KEY_NAME"
Run # Try to decrypt the private key with the provided passphrase (if any)
Run for host in $HOSTS; do
Retrying ssh-keyscan for *** in 5s...
Error: Process completed with exit code 1.

Ok we could add more retries and maybe add a longer timeout for ssh-keyscan command which defaults to 5s.

[jo-basevi]

Ok I've merged a change with several more retries and a longer timeout now so hopefully that helps.