Mage-Flow-Edit-Turbo ~1.6x slower on ROCm after graph-plan caching change (#1963)

[basementops]

Description

After bisecting a performance regression on Mage-Flow-Edit-Turbo (img-edit workflow) on ROCm/HIP, I found it's caused by commit 5ebce93 ("fix: reuse graph plans when scale parameters change", #1963). Sampling time for 16 steps at 1024×1024 goes from ~162s to ~266s (~1.64×) — output is pixel-identical (same seed), so this is purely a performance regression, not a correctness issue.

Notably ROCm-specific: The same test on the Vulkan backend shows no regression at all (154–156s before and after this commit).

Environment

• OS: Debian 13, AMD Ryzen AI Max+ 395 (Strix Halo), gfx1151, ROCm 7.2.4
• Build: -DSD_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release

Bisect result

5ebce93342718ff7c0afa69b9b51d3fc0a1fd991 is the first bad commit
commit 5ebce93342718ff7c0afa69b9b51d3fc0a1fd991
    fix: reuse graph plans when scale parameters change (#1963)
 src/core/ggml_graph_cut.cpp | 16 ++++++++++++++--

Parent commit and everything before it: ~150–165s (matches the known-good baseline at 97d2990, three commits before the regression). 5ebce93 itself and everything after (tested up to current master c678dfe): ~230–266s.

Steps to reproduce

Weights: Mage-Flow-Edit-Turbo (Comfy-Org/Mage-Flow, bf16), Qwen3VL-4B-Instruct (llm + llm_vision), own VAE.

sd-cli \
  --diffusion-model mage_flow_edit_turbo_bf16.safetensors \
  --llm Qwen3VL-4B-Instruct-Q8_0.gguf \
  --llm_vision mmproj-Qwen3VL-4B-Instruct-F16.gguf \
  --vae mage_flow_vae_bf16.safetensors \
  --sampling-method euler --steps 16 --cfg-scale 1.0 -W 1024 -H 1024 --diffusion-fa \
  --ref-image <any image> \
  -p "change the background to a beach at sunset" \
  -o out.png

Hypothesis

The commit stops including op_params for GGML_OP_SCALE nodes in the graph-layout cache signature (intended to allow reusing a cached plan when only a scale value changes). For Mage-Flow-Edit-Turbo's graph, this may cause structurally different graphs to collide on the same cache key, forcing extra reconciliation work — apparently only costly on the HIP/ROCm backend path, not Vulkan. I haven't dug further into ggml_graph_cut.cpp internals to confirm the exact mechanism.

Additional notes

Related to (but distinct from) #2009 (SIGFPE in Wan video path) — same bisect investigation surfaced this as a second issue. Happy to test a patch.

[leejet]

Thanks for the detailed bisect and the Vulkan comparison. I traced the change, but I haven't confirmed the root cause on ROCm yet.

One detail worth checking: 5ebce93 changes the signature used by both the graph-cut plan cache and allocator reservation reuse. It still compares tensor shapes, strides, view offsets, and source connections, so I haven't found evidence of structurally different graphs colliding.

I also ran a local graph-only check using Mage-Flow with BF16 weight metadata, one reference latent, flash attention enabled, and 16 different timesteps. The graph had 2,018 nodes, including 128 SCALE nodes. All SCALE parameters, graph-layout signatures, and binding signatures stayed identical across timesteps. Mage's timestep embedding is passed as tensor data, rather than a changing SCALE parameter. This suggests that this graph should already reuse its plan before the change. This was a CPU-side graph construction check without loading weights or running inference, so it does not reproduce or rule out the ROCm regression.

Could you repeat the same benchmark on both the parent commit and 5ebce93, with GGML_CUDA_DISABLE_GRAPHS=1 set and --log-level debug added? Despite its name, this environment variable also disables HIP graphs. Please keep the seed and all other arguments unchanged.

The HIP backend has a separate graph capture/replay cache whose warmup depends on tensor metadata and addresses. If disabling HIP graphs removes the performance gap, that would narrow the investigation to this path. Otherwise, we can focus on allocation and execution differences. Logs from both versions would help, especially the occurrences of CUDA graph warmup complete, CUDA graph warmup reset, and build cached graph cut plan done.

[basementops]

Thanks for the detailed follow-up. I ran the requested comparison, but the result is more surprising than expected — I want to report it honestly rather than send a misleading conclusion.

Setup: rebuilt separately at the parent commit (7f986a9) and at 5ebce93, both -DSD_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1151, same test as the original report (Mage-Flow-Edit-Turbo, 1024×1024, 16 steps, euler, cfg 1.0, --diffusion-fa, seed 42).

With GGML_CUDA_DISABLE_GRAPHS=1 --log-level debug:

Commit	Sampling time
7f986a9 (parent)	160.39s
5ebce93	148.52s

No CUDA graph warmup complete/reset lines in either log (expected, graphs disabled). build cached graph cut plan done appears 5 times in both logs, identical pattern (mage_vae encode, 2× qwen3vl conditioning, mage_flow, mage_vae decode), each taking 0–4ms — plan is built once and reused across all 16 mage_flow executing segment steps in both commits, matching your own graph-construction check.

Then I went back and re-ran the control (graphs enabled, default) on these same fresh builds — and couldn't reproduce the original regression at all:

Condition (commit 5ebce93, graphs enabled)	Sampling time
Isolated, idle system	147.36s
Under real concurrent load (a 35B LLM actively generating on the same GPU, ~120s overlap)	157.79s
Different resolution (832×1216 instead of 1024×1024)	144.95s
Original report	~265.8s

CUDA graph warmup complete does appear in these logs (confirming graphs were genuinely active), but sampling time stays in the same 145–160s range regardless — no sign of the ~1.64× slowdown under any of these four conditions.

I don't have a solid explanation for the discrepancy with the original measurement. One thing worth mentioning for transparency: the original number was measured before a full host reboot (unrelated incident, a GPU memory/driver issue on our side), and everything I've tested since was measured after — I can't rule out that the original run was affected by some environmental state that a clean boot resets, though I also can't confirm that's what happened.

I'm hesitant to invest in a much more involved test (a persistent sd-server process handling two differently-shaped requests in sequence, to specifically try to trigger the cache-key collision you described) without more confidence it would actually reproduce something. Happy to try that if you think it's worth it, or if you have a more specific repro that differs from what I've done here — otherwise I'd currently lean toward this being an artifact of the original measurement rather than a real, reliably-reproducible regression. Sorry for the back-and-forth — wanted to give you the full picture rather than a tidy but possibly wrong answer.