Whether the lav_paint supports the wan2.1 model

[linwenzhao1]

Hello, please ask if the background replacement demo also supports the Wan2.1 model?

[YujieOuO]

CogVideoX and Wan2.1 currently do not support background generation. Because the 3D Causal-VAE they utilize may not directly support inpainting operations😟.

Any relevant suggestions are welcome for discussion.

[linwenzhao1]

CogVideoX and Wan2.1 currently do not support background generation. Because the 3D Causal-VAE they utilize may not directly support inpainting operations😟.

Any relevant suggestions are welcome for discussion.

Does vae mainly do the work of space compression and will it include inpaint operation? Maybe I don't have a deep understanding

[YujieOuO]

1. AnimateDiff uses the Image VAE and we can mask the background latent with noise, then repaint a new background after denoising step.
2. Wan2.1 employs the 3D Cause VAE, which is not suitable for directly masking background areas. Because there is a risk of information leakage between frames. For instance, the masked region in the second frame may be filled with information from the first frame. As a result, the original video information is retained, making it impossible to achieve inpainting by simply denoising the masked areas.

[linwenzhao1]

1. AnimateDiff uses the Image VAE and we can mask the background latent with noise, then repaint a new background after denoising step.
2. Wan2.1 employs the 3D Cause VAE, which is not suitable for directly masking background areas. Because there is a risk of information leakage between frames. For instance, the masked region in the second frame may be filled with information from the first frame. As a result, the original video information is retained, making it impossible to achieve inpainting by simply denoising the masked areas.

There is another question. Do you try to relight the picture with instruct pix2pix? pix2pix has better relight effect than IC-LIGHT for some scenes.

[YujieOuO]

No, we haven't done relighting on InstructPix2Pix. Will InstructPix2Pix perform better than IC-Light on the relighting task? It seems that InstructPix2Pix can't maintain the original ID information. Maybe you could provide some cases.