Demand unet convergence

[JZZZ1314]

https://github.com/Kwai-Kolors/Kolors
in kolors, use sdxl unet,but These two more 'encoder_hid_proj.weight', 'encoder_hid_proj.bias'
Do you have a plan that you can adjust? and the convergence scheme for its LLM

[ljleb]

Sure I can add a config for kolors. I need to look at the state dict keys, do you know if all of the sdxl keys are the same except for those with hid_proj in the name?

[ljleb]

Looking at the hugging face, there seems to only be a diffusers release of the model. This means the state dict comes split into multiple different files, which is currently not something sd-mecha can work with.

If this is really the case, then to support the kolors model type, we need to first add support for the diffusers directory format, which has not been done yet.

To this end, we can add a new custom dictionary type in sd_mecha/streaming.py which will take care of forwarding the keys to save to the right file by managing all the output files at the same time. Something similar has to be done for model loading, and if there is a namespace conflict of the same key in different state dicts then it could require a lot more work to take care of key disambiguation.

[ljleb]

If you intend to merge kolors models with other sdxl models like pony, this would be a lot more work though. There are different implementations of sdxl (diffusers, sgm, etc.), which means the same exact model will have different keys when converted from sgm to diffusers for example. This means that we need to be able to create 2 configurations for the same architecture and to convert a model from one implementation to the other.

There is no facility for this currently in sd-mecha. I am considering adding a module sd_mecha.extensions.conversion which will allow users to specify custom conversion functions from one implementation to the other (or one model architecture to another if applicable (sd1.5 to sdxl for example)) and a function sd_mecha.convert() which will integrate conversion to the recipe plan.

[JZZZ1314]

Sure I can add a config for kolors. I need to look at the state dict keys, do you know if all of the sdxl keys are the same except for those with hid_proj in the name?

Yes, from their paper, we can see that the essence is that the unet based on SDXL adds the processing of encode, by finding that there are indeed two layers,

[JZZZ1314]

If you intend to merge kolors models with other sdxl models like pony, this would be a lot more work though. There are different implementations of sdxl (diffusers, sgm, etc.), which means the same exact model will have different keys when converted from sgm to diffusers for example. This means that we need to be able to create 2 configurations for the same architecture and to convert a model from one implementation to the other.

There is no facility for this currently in sd-mecha. I am considering adding a module sd_mecha.extensions.conversion which will allow users to specify custom conversion functions from one implementation to the other (or one model architecture to another if applicable (sd1.5 to sdxl for example)) and a function sd_mecha.convert() which will integrate conversion to the recipe plan.

I'm looking forward to this day

[ljleb]

There are like a hundred other things that I need to do before starting to look into this... I'm not sure when I'll be able to start looking at this. It might take a while. I'm open to contributions if you are interested in this and don't want to wait for me.