About MoE Loss

[yang-0201]

In the currently open-sourced project, the loss function being used is still v8DetectionLoss. I could not find the definition, implementation, or weighting strategy of the MoE loss. In addition, the paper mentions that performance degradation occurs when DFL and MoE loss are used simultaneously, but in the current open-source codebase, the DFL loss has not been removed.

Could you please clarify whether a fully reproducible version, including the complete MoE loss implementation and the corresponding loss configuration, will be released in a future update?

[yang-0201]

find！

[isLinXu]

Our MoE modules are still undergoing refactoring and decoupling; once the review is complete we’ll open-source them.

[isLinXu]

You can find the file at: https://github.com/isLinXu/YOLO-Master/blob/main/ultralytics/cfg/default.yaml

# Hyperparameters ------------------------------------------------------------------------------------------------------
lr0: 0.01 # (float) initial learning rate (SGD=1e-2, Adam/AdamW=1e-3)
lrf: 0.01 # (float) final LR fraction; final LR = lr0 * lrf
momentum: 0.937 # (float) SGD momentum or Adam beta1
weight_decay: 0.0005 # (float) weight decay (L2 regularization)
warmup_epochs: 3.0 # (float) warmup epochs (fractions allowed)
warmup_momentum: 0.8 # (float) initial momentum during warmup
warmup_bias_lr: 0.1 # (float) bias learning rate during warmup
box: 7.5 # (float) box loss gain
cls: 0.5 # (float) classification loss gain
dfl: 1.5 # (float) distribution focal loss gain
moe: 1.5 # (float) moe loss gain

[yang-0201]

Bug in the new MoE loss implementation: the MoE loss does not seem to take effect and is difficult to optimize, as the loss does not consistently decrease.

I conducted two experiments using the same configuration file to train on the COCO dataset, one with the old version of the project and one with the latest version. Structurally, both experiments only use ES_MOE in the backbone network.

In the first experiment (old version), no MoE loss was applied. In the second experiment (new version), MoE loss was enabled using the default parameters. However, after training for more than 170 epochs, I observed that the results at the same epoch were identical in both experiments: the mAP, as well as the other three losses, were exactly the same.

Meanwhile, the reported MoE loss starts from approximately 0.008, increases to 0.01, then 0.02, and remains stable thereafter. This strongly suggests that the MoE loss is not actually affecting training or backpropagation.

Could you please clarify what might be causing this issue? Is the MoE loss currently disconnected from the main optimization pipeline, or is there an intended design choice that I may be missing?