Skip to content

perf: run one-frame Wan VAE convolutions as 2D convolutions - #2038

Merged
leejet merged 1 commit into
leejet:masterfrom
nanguoyu:perf/wan-vae-single-frame-conv2d
Sep 23, 2026
Merged

leejet merged 1 commit into
leejet:masterfrom
nanguoyu:perf/wan-vae-single-frame-conv2d

Conversation

@nanguoyu

@nanguoyu nanguoyu commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Problem

On Metal, the VAE decode of Qwen Image 2.1 is slow: 74.5 s for a 512x512 image on an M1 Pro. The Wan VAE's convolutions go through ggml_ext_conv_3d(); Metal has no IM2COL_3D, so since #1731 they fall back to GGML_OP_CONV_3D, whose Metal kernel accumulates each output element in a scalar loop.

For images this fallback is avoidable. Image VAEs built on the Wan VAE keep their Conv3d weights one frame deep (every decoder convolution of Qwen Image 2.1's VAE is 1x3x3 or 1x1x1), and an image is a single frame, so each of these convolutions is a 2D convolution.

Change

WAN::CausalConv3d::forward() runs a single-frame input through a one-frame-deep kernel with ggml_ext_conv_2d() (im2col + mul_mat) and everything else through ggml_ext_conv_3d() as before. VAEs with deeper temporal kernels (Wan 2.1/2.2, Qwen Image) are unchanged, as is video decoding. On backends that implement IM2COL_3D the 2D path computes the same products; the activations go through im2col in f16 either way.

Verification

Apple M1 Pro (32 GB), macOS 26, -DSD_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON, on master 2dc7f54, Qwen Image 2.1 Q4_K_M, 512x512, 20 steps, seed 42:

sd-cli --diffusion-model qwen-image-2.1-Q4_K_M.gguf --llm Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf \
    --vae qwen_image_2.1_vae_bf16.safetensors -p "a lovely cat holding a sign that says 'sd.cpp'" \
    --cfg-scale 6.0 --sampling-method euler --steps 20 -W 512 -H 512 --diffusion-fa -s 42 -v
decode_first_stage image
master 74.5 s reference
this PR 7.3 s max difference 5/255, PSNR 66.5 dB

Left: master. Right: this PR.

Not tested on CUDA, Vulkan or CPU.

pr2-images

Image VAEs built on the Wan VAE (Qwen Image 2.1 among them) keep Conv3d
weights one frame deep, and an image is a single frame, so each such
convolution is a 2D convolution. Metal has no IM2COL_3D, so these fall back
to GGML_OP_CONV_3D (leejet#1731), whose Metal kernel accumulates every output
element in a scalar loop: a 512x512 Qwen Image 2.1 decode takes over a
minute on an M1 Pro.

Run them through ggml_ext_conv_2d (im2col + mul_mat) instead. Backends that
implement IM2COL_3D compute the same products through the 2D path.
@leejet
leejet merged commit 70c1dbc into leejet:master Sep 23, 2026
9 checks passed
@leejet

leejet commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants