Question on FAMO loss behavior

[MendezEd]

Hello and thanks for the fantastic work on FAMO!

I've been experimenting with FAMO in some personal projects and have observed an interesting behavior related to loss values. It appears that when individual losses are less than 1, the FAMO loss becomes negative, and when they are greater than 1, the FAMO loss is positive. This seems to stem from the logarithmic operation in get_weighted_loss.

Interestingly, when all individual losses are uniformly less than 1 or greater than 1, the aggregated FAMO loss seems to converge towards zero. I'm not entirely clear on the mechanics behind this behavior.

I am curious about how FAMO is expected to perform in scenarios where some losses exceed 1 while others do not. Additionally, I've encountered issues when attempting to implement learning rate (LR) scheduling under these conditions. It seems to fail.

Could you provide any insights or recommendations for handling such cases? Maybe there's a different approach or adjustment to the LR scheduling that might help?

[Cranial-XIX]

Yes, as ln(1) = 0, the log loss will be negative/positive when the original loss is greater/smaller than 1. I don't know what is happening exactly in your case. From your description, I would play with the min_losses (making them -1 or smaller so that we fake all losses to be greater than 1) and then play with the weight decay gamma (for w_opt). When weight decay gamma is large and min_losses are small enough, you should recover the original average loss training.

[MendezEd]

Thanks! All of them are MSEs, so min_losses should be 0 as default. Would setting them to -1 mess up the gradients? Should I set all min_losses to -1? If it doesn't make any difference, could this be the default setting?

Well, my problem has some losses that are bigger than 1, while others are smaller. I was playing with ReduceLROnPlateau to schedule ADAM's LR. However, some losses were increasing while others were decreasing, and the scheduler monitors just one of the directions. This was resulting in a series of suboptimal plateaus in the aggregated loss. However, if faking the loss to be always positive works, this shouldn't be a problem anymore.