Workflow controller occurs `fatal error: stack overflow` when specific named workflow is reconciled

[okzw999]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When a workflow with a specific combination of workflow name, step name, and referenced template name is reconciled, the workflow-controller crashes with fatal error: stack overflow in childrenFulfilled and childrenFulfilledHelper.
https://github.com/argoproj/argo-workflows/blob/v3.7.15/workflow/controller/operator.go#L2616-L2648

In our reproduction, the workflow stays in Running and the controller repeatedly restarts. This affects not only the problematic workflow but also the whole controller process.

The issue occurs with the following structure:

• Workflow name is custom-job-thbh7
• The workflow has a step named custom-job
• That step references a WorkflowTemplate or ClusterWorkflowTemplate template also named custom-job
• The referenced template is a steps template
• Inside that referenced template, the executed step is named custom-job-main
• That step references a pod/container template also named custom-job-main

We confirmed that the same problem occurs whether the referenced template is defined in a namespaced WorkflowTemplate or in a ClusterWorkflowTemplate.

In the actual incident, the workflow name was defined with generateName. The problem was discovered when a workflow was accidentally submitted with the generated name custom-job-thbh7. For the minimal reproduction below, we use a fixed metadata.name: custom-job-thbh7 so that the same generated node IDs can be reproduced reliably.

With this combination, the workflow status contains a cyclic node relationship. For example:

Nodes:
  custom-job-thbh7:
    Children:
      custom-job-thbh7-1277661780

  custom-job-thbh7-1277661780:
    Boundary ID: custom-job-thbh7
    Children:
      custom-job-thbh7-3328663899

  custom-job-thbh7-3328663899:
    Boundary ID: custom-job-thbh7
    Children:
      custom-job-thbh7-3969105591

  custom-job-thbh7-3969105591:
    Boundary ID: custom-job-thbh7-3328663899
    Children:
      custom-job-thbh7-1277661780

The last node points back to custom-job-thbh7-1277661780, which is an ancestor node. After this cyclic node graph is created, the controller repeatedly recurses through the children while checking whether child nodes are fulfilled, and eventually crashes:

The Argo UI also shows an abnormal loop-like graph for this workflow. The custom-job node is connected in a way that visually returns to an earlier node instead of progressing to the pod node.

runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper
    /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633
...
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilled
    /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2647

For comparison, if we submit the same workflow structure with a different workflow name, for example custom-job-abcde, the workflow succeeds normally. Its node graph is acyclic and includes the expected pod node:

custom-job-abcde:
  Children:
    custom-job-abcde-3032388424

custom-job-abcde-3032388424:
  Children:
    custom-job-abcde-3045544495

custom-job-abcde-3045544495:
  Children:
    custom-job-abcde-783001307

custom-job-abcde-783001307:
  Children:
    custom-job-abcde-2932551912

custom-job-abcde-2932551912:
  Type: Pod
  Phase: Succeeded

We also tested several variations to narrow down the condition. The issue did not occur in the following cases:

• The workflow name is changed from custom-job-thbh7 to another value, such as custom-job-abcde.
• The workflow step does not reference another steps template, and instead directly references a pod/container template.
• The workflow step name or the referenced template name is changed from custom-job to another value.
• The inner step name or the pod/container template name is changed from custom-job-main to another value.
• Multiple inner steps are defined, but the executed, non-skipped step does not use the custom-job-main step/template name combination.

Therefore, the crash seems to require both the nested steps template structure and this specific set of generated node names, but it does not seem to depend on whether the referenced template is namespaced or cluster-scoped. This suggests that the controller can create an invalid cyclic node graph from otherwise valid workflow/template names, instead of simply rejecting the workflow or processing it normally.

We also checked the same problematic workflow on v3.6. In v3.6, submitting the workflow did not cause the workflow-controller to hit the fatal stack overflow, and the workflow completed. However, the node relationship was still cyclic in the same way. It looks like the invalid cyclic node graph can already be created, but the fatal crash is triggered by the childrenFulfilled / childrenFulfilledHelper logic that was added in v3.7.

Expected behavior:

• Reusing the same name for a step and the template it references should not create a cyclic node graph.
• The workflow should either run to completion or fail gracefully.
• The workflow-controller should not crash or enter a restart loop because of a malformed or cyclic node relationship.
• If such a cyclic node graph is detected, the controller should handle it defensively, for example by marking the workflow as failed with a clear error instead of recursing until stack overflow.

Version(s)

v3.7.15, v4.0.6

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: custom-job-thbh7
  labels:
    workflows.argoproj.io/archive-strategy: "false"
spec:
  arguments:
    parameters:
      - name: MESSAGE
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: custom-job
        templateRef:
          name: workflow-template-debug
          template: custom-job
        arguments:
          parameters:
          - name: message
            value: "{{workflow.parameters.MESSAGE}}"
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-debug
spec:
  templates:
    - name: custom-job
      inputs:
        parameters:
          - name: message
      steps:
      - - name: custom-job-main
          template: custom-job-main
          arguments:
            parameters:
            - name: message
              value: "{{inputs.parameters.message}}"
    - name: custom-job-main
      inputs:
        parameters:
        - name: message
      container:
        image: foo # in-house image based on rocky linux 9.

        command: [echo]
        args: ["{{inputs.parameters.message}}"]
        resources:
          limits:
            memory: "1000Mi"
            cpu: "1"
          requests:
            memory: "500Mi"
            cpu: "500m"

Logs from the workflow controller

workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:09.621Z" level=info msg="update pod event" pod=custom-job-thbh7-custom-job-main-1277661780
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:10.800Z" level=info msg="update pod event" pod=custom-job-thbh7-custom-job-main-1277661780
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:12.360Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2169762057 namespace=argo workflow=custom-job-thbh7
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:12.361Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=custom-job-thbh7
workflow-controller-65768587fc-jjldk workflow-controller runtime: goroutine stack exceeds 1000000000-byte limit
workflow-controller-65768587fc-jjldk workflow-controller runtime: sp=0xc02b1b04e0 stack=[0xc02b1b0000, 0xc04b1b0000]
workflow-controller-65768587fc-jjldk workflow-controller fatal error: stack overflow
workflow-controller-65768587fc-jjldk workflow-controller 
workflow-controller-65768587fc-jjldk workflow-controller runtime stack:
workflow-controller-65768587fc-jjldk workflow-controller runtime.throw({0x2ff8174?, 0x435c9b?})
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/panic.go:1094 +0x48 fp=0xc000195e98 sp=0xc000195e68 pc=0x4812e8
workflow-controller-65768587fc-jjldk workflow-controller runtime.newstack()
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/stack.go:1159 +0x5bd fp=0xc000195fc8 sp=0xc000195e98 pc=0x46489d
workflow-controller-65768587fc-jjldk workflow-controller runtime.morestack()
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/asm_amd64.s:620 +0x7d fp=0xc000195fd0 sp=0xc000195fc8 pc=0x4875bd
workflow-controller-65768587fc-jjldk workflow-controller 
workflow-controller-65768587fc-jjldk workflow-controller goroutine 487 gp=0xc000f40fc0 m=4 mp=0xc000180008 [running]:
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/pkg/apis/workflow/v1alpha1.Nodes.Get(0xc0036c7b90, {0xc004e88f80, 0x1b})
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/pkg/apis/workflow/v1alpha1/workflow_types.go:1820 +0x1ca fp=0xc02b1b04f0 sp=0xc02b1b04e8 pc=0x1af250a
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc016d92a80, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2629 +0x1a5 fp=0xc02b1b07f8 sp=0xc02b1b04f0 pc=0x26ff205
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc016d92900, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc02b1b0b00 sp=0xc02b1b07f8 pc=0x26ff225
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc016d92780, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc02b1b0e08 sp=0xc02b1b0b00 pc=0x26ff225
...
workflow-controller-65768587fc-jjldk workflow-controller ...691732 frames elided...
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc00573c600, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc04b1a4920 sp=0xc04b1a4618 pc=0x26ff225
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc00573c480, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc04b1a4c28 sp=0xc04b1a4920 pc=0x26ff225
...
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilled(...)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2647
...
workflow-controller-65768587fc-jjldk workflow-controller runtime.goexit({})
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000f03fe8 sp=0xc000f03fe0 pc=0x4892e1
workflow-controller-65768587fc-jjldk workflow-controller created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).Run in goroutine 108
workflow-controller-65768587fc-jjldk workflow-controller 	/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:392 +0x19bc
workflow-controller-65768587fc-jjldk workflow-controller 
workflow-controller-65768587fc-jjldk workflow-controller goroutine 515 gp=0xc000a56540 m=nil [sync.Cond.Wait, 8 minutes]:
workflow-controller-65768587fc-jjldk workflow-controller runtime.gopark(0xc005592ae0?, 0xc0009f6a80?, 0xe0?, 0xc9?, 0xc000690230?)
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc008e53468 sp=0xc008e53448 pc=0x48140e
workflow-controller-65768587fc-jjldk workflow-controller runtime.goparkunlock(...)
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/proc.go:466
workflow-controller-65768587fc-jjldk workflow-controller sync.runtime_notifyListWait(0xc00042bcd0, 0x2a582)
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/runtime/sema.go:606 +0x159 fp=0xc008e534b8 sp=0xc008e53468 pc=0x482bf9
workflow-controller-65768587fc-jjldk workflow-controller sync.(*Cond).Wait(0x22f7c8f?)
workflow-controller-65768587fc-jjldk workflow-controller 	/usr/local/go/src/sync/cond.go:71 +0x73 fp=0xc008e534f0 sp=0xc008e534b8 pc=0x493293
workflow-controller-65768587fc-jjldk workflow-controller k8s.io/client-go/util/workqueue.(*Typed[...]).Get(0x35147a0)
workflow-controller-65768587fc-jjldk workflow-controller 	/go/pkg/mod/k8s.io/client-go@v0.33.1/util/workqueue/queue.go:277 +0x86 fp=0xc008e53558 sp=0xc008e534f0 pc=0x147a9c6
workflow-controller-65768587fc-jjldk workflow-controller k8s.io/client-go/util/workqueue.(*Typed[...]).Get()
workflow-controller-65768587fc-jjldk workflow-controller 	/go/pkg/mod/k8s.io/client-go@v0.33.1/util/workqueue/queue.go:273 +0x25 fp=0xc008e53578 sp=0xc008e53558 pc=0x147d7c5
workflow-controller-65768587fc-jjldk workflow-controller k8s.io/client-go/util/workqueue.(*delayingType[...]).Get()
workflow-controller-65768587fc-jjldk workflow-controller 	<autogenerated>:1 +0x24 fp=0xc008e53590 sp=0xc008e53578 pc=0x147f3c4

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[coderabbitai]

🔗 Related PRs

#15090 - fix: workflow controller to detect stale workflows [merged]
#15345 - chore: enable errorlint linter [merged]
#15442 - fix: requeue workflow if expected variables are missing. Fixes #15513 [merged]
#15585 - feat: add tracing span instrumentation to controller and executor [merged]
#15736 - fix: add stepgroup and taskgroup to scope. Fixes #15737 [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!