Pre-requisites
What happened? What did you expect to happen?
When upgrading from Argo Workflows v3 to v4, daemon pods that exit non-zero started being marked NodeFailed even when the overall workflow succeeded.
In v3, assessNodeStatus unconditionally marked daemon pods as NodeSucceeded when their Kubernetes phase was PodFailed:
if tmpl != nil && tmpl.IsDaemon() {
new.Phase = wfv1.NodeSucceeded // daemon failure silently ignored
}
In v4 (commit 9b73126), this guard was removed as part of adding retry strategy support for daemon containers. PodFailed now calls inferFailedReason for all pods, including daemons. The stale comment "ignore pod failure for daemoned steps" remains but no longer matches the behaviour.
The root cause: Argo intentionally terminates daemon pods by killing argoexec (the main container) once downstream steps complete. Kubernetes sends SIGTERM to argoexec, which exits 143 (128+15). When the pod phase transitions to PodFailed, inferFailedReason examines the main container's exit code. For non-daemon pods, any non-zero exit on the main container correctly surfaces as NodeFailed. But for daemon pods, SIGTERM/SIGKILL on the main container is expected and should be treated the same way sidecar containers already are (exit codes 137 and 143 are ignored for sidecars).
We confirmed this by:
- Running dind (Docker-in-Docker) as a daemon pod in a real workflow
- Trying dockerd-entrypoint.sh, preStop lifecycle hooks, --shutdown-timeout=0, and wrapping dockerd with || true — none resolved the issue because the problem was argoexec (main) being SIGTERMed, not the dind container itself
- Reading inferFailedReason and confirming there was no daemon-awareness for the main container exit code path
Fix: in the IsMainContainerName case of inferFailedReason, check if the template is a daemon and the exit code is 137 or 143. If so, treat it the same as a sidecar SIGKILL/SIGTERM — log it and mark mainContainerSucceeded so the node resolves to NodeSucceeded. Any other non-zero exit code on the main container of a daemon pod still surfaces as NodeFailed, preserving detection of real daemon crashes.
Version(s)
:latest
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container
Pre-requisites
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below.What happened? What did you expect to happen?
When upgrading from Argo Workflows v3 to v4, daemon pods that exit non-zero started being marked NodeFailed even when the overall workflow succeeded.
In v3, assessNodeStatus unconditionally marked daemon pods as NodeSucceeded when their Kubernetes phase was PodFailed:
In v4 (commit 9b73126), this guard was removed as part of adding retry strategy support for daemon containers. PodFailed now calls inferFailedReason for all pods, including daemons. The stale comment "ignore pod failure for daemoned steps" remains but no longer matches the behaviour.
The root cause: Argo intentionally terminates daemon pods by killing argoexec (the main container) once downstream steps complete. Kubernetes sends SIGTERM to argoexec, which exits 143 (128+15). When the pod phase transitions to PodFailed, inferFailedReason examines the main container's exit code. For non-daemon pods, any non-zero exit on the main container correctly surfaces as NodeFailed. But for daemon pods, SIGTERM/SIGKILL on the main container is expected and should be treated the same way sidecar containers already are (exit codes 137 and 143 are ignored for sidecars).
We confirmed this by:
Fix: in the IsMainContainerName case of inferFailedReason, check if the template is a daemon and the exit code is 137 or 143. If so, treat it the same as a sidecar SIGKILL/SIGTERM — log it and mark mainContainerSucceeded so the node resolves to NodeSucceeded. Any other non-zero exit code on the main container of a daemon pod still surfaces as NodeFailed, preserving detection of real daemon crashes.
Version(s)
:latest
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
N/ALogs from the workflow controller
Logs from in your workflow's wait container