How are efficiency performance and efficiency score calculated?

[DennisAdenHollander]

I have uploaded the first submission for the efficiency, but i receive an efficiency performance of 0.000. I was wondering on what basis the performance and score is calculated also for my research paper

[TimJaspers0801]

The Efficiency score is based on the number of Gflops of your model (you want to minimize this value). The Efficiency performance is calculated as the mean Dice score divided by the number of Gflops. However, there is an additional condition: your model must achieve at least 80% of the baseline performance (TA_baseline) to receive an efficiency score. If it falls below this threshold, the efficiency performance is set to 0.000. This is also the reason why you received a performance of 0.000. That said, don’t worry—this challenge is designed to be difficult, and achieving a good efficiency score is not necessarily easy. Keep working on it, and let us know if you need any guidance!"

[kmilas]

So the efficiency score does not take into account model parameters, inference speed or memory consumption ? Also I use two tools to calculate the flops of my model and get different results. For example with thop https://github.com/ultralytics/thop i get 10.68 GMACs and from flops counter https://github.com/sovrasov/flops-counter.pytorch I get 13.36 GMACs. Also we need to care only about gpu inference or also cpu inference and can we know the image resolution that you use for testing ?

[TimJaspers0801]

You're right to raise these points. While model parameters, inference speed, and memory consumption are all important practical metrics, they are often highly correlated—especially with the number of FLOPs. For this challenge, we consider the number of FLOPs to be the most representative and comparable measure of efficiency across submissions.

The image resolution used for testing is equal to the resolution you use for training your model. In the process_data.py file, you have access to the preprocess function. Here, you can simply resize the images to the intended resolution.

[DennisAdenHollander]

I do not get how my more accurate model (trained using a combined loss function of CE loss and DICE loss) gets a lower mean dice score than the worst model in my arsenal? I mean even when I check the validation set the standard model performs very poorly and the secondary model performs decent, however the score does not represent this at all...

[TimJaspers0801]

Don't you perform any normalization? than you only need to resize to the correct resolution.

[DennisAdenHollander]

i have only changed figure size as was instructed, for the rest I have not edited this file at all...

[TimJaspers0801]

no, except if you came up with something special in the preprocessing of the data.

[DennisAdenHollander]

I have not done this. I have compared the original files with my current ones and the only difference is:

img = transforms.functional.resize(img, size=(256, 256), interpolation=transforms.InterpolationMode.LANCZOS)
inside preprocess(img):

For the rest I noticed a discrepancy in the competition instructions, where it states I have to deliver preprocess.py instead of process_data.py.

Next to that I noticed my models getting high scores on codalab for dice 1, 2 and 10, where the other dices are super low (approaching zero), does this mean anything (it getting the number of classes wrong)? My model is checked next to the base UNet model, and sends out the same output shape atleast.

There is one difference between the UNet and my model, my model states n_classes = 19 whereas the UNet states n_classes = 1

Sorry that I take so much time, but I cannot check whether I fixed the issue due to me throwing away submissions and training coins for this.

[TimJaspers0801]

No worries at all, that’s exactly what this discussion page is for!

We’re aware of the discrepancy in the competition instructions; someone else has already pointed it out. For now, just stick with the process_data.py file, which includes the correct preprocess function.

From what I can see, it looks like you're on the right track. Don’t worry too much about the number of submissions—that limit is mainly there to prevent the server from getting overloaded. When you get a decent score, I can remove previous submissions for you, and your number of submissions will be back to 1.

That said, it’s a bit tricky for me to pinpoint the exact issue without seeing more details. If you have some time, feel free to drop by our office (Flux 5.092), and we can take a look at your code together.