Optimization issue

[NeuroLord]

Are there any ways to reduce VRAM consumption to 12 gigabytes? Perhaps offload models partially/completely to the CPU? Thanks in advance!

[River-Zhang]

I’m not certain if this will reduce VRAM consumption, but you might want to try using DeepSpeed. It’s typically used for large language models, so its effectiveness for your case might be limited. However, it could potentially help with VRAM management.