GEV is unusually high? Verbose file incorrect?

[armenbagdasarov]

Hi Denis! Armen again.

It seems unusual to me, the GEV after stage 2 / group clustering is very high (see Screenshot 1). I have never seen GEV values so high at this stage (>.90). Does this seem unusual to you, too?

Also, the verbose file at stage 2 does not seem to be outputting with the correct information. Specifically, I used the resampling method as clearly seen in Screenshots 2 and 3 (i.e., see folder structure). However, as seen in Screenshot 4, the verbose file says no resampling was used.

[DenisBrunet]

Hi again Armen
I know I checked for the GEV computation implementation last year, but if memory serves, didn't found anything at that time.
I am going to double check that, and run some tests on some existing data of mine..

[DenisBrunet]

Didn't have time to fully investigate yet, but found I did modify the GEV implementation back in 2024-05-14. But I did (supposedly) thorough tests back in the day. I will double check these modifications.

[armenbagdasarov]

I just ran a quick test myself. I looked at some of our baby data that I had already processed with an older version (not open source) of Cartool. The GEV for 5 microstates was .77 in the older version and now with the new version (exact same data, all the same inputs) it is 97.69 (n = 43 babies). So, something definitely appears to be wrong in the calculation!

[DenisBrunet]

Hi Armen, there are quite a few things happening here:

• GEV formula implementation seems totally fine. I checked against an older version, and checked the code, too. But you said you found some discrepancies with the same dataset, which is weird. Which older version did you use for comparison?
• The verbose output might be confusing you due to some poorly chosen choice of words (of mine). "Whole epoch" was meant to be "Whole time-line". As well as the "GFP resampling" which could be confused to the data resampling. Therefor, I updated the verbose ouput to make that less ambiguous.
• On your last screenshot, it shows "no resampling", because this is the results of a "local" clustering, so to speak. The input data has been first resampled by the main segmentation, saved to local files, then each of these files has been fed to a "child" segmentation process. This very verbose you see is from the child segmentation. So this is fine, although a bit confusing. Will see if I can clarify that, too...

In the meantime, could you try the 5.04 release to see if the other issues got fixed?
Could you maybe send me the verbose files of your (stage 2?) segmentation from which you find different GEV results?
Best,
Denis

[armenbagdasarov]

I was comparing version 4.11 and 5.03

It appears that Version 4.11 never produced verbose files at Stage 2 for each like version 5.03 does except for in the resampling folder for each resampled epoch which is not very relevant.

I guess here is where I am confused. At Stage 2, version 5.03 outputs a verbose file for some of the k solutions as long as they meet some threshold.

You can visualize this with the error.data file. For example, here, k = 6, 7, 10, and 11 produce a positive MetaCriterion; though, clearly 6 is the winner. And verbose files are only available for 6 and 7 since the MetaCriterion for 10 and 11 are too low. k = 7 is very low too but passes the threshold I suppose.

When you open each verbose file, the GEV are as follows: k = 6 (GEV = 96.2008), k = 7 (GEV = 99.7968). But when you open the error.data file (I usually copy and paste into Excel), the GEV are as follows (and to me, more reasonable): k = 6 (GEV = 0.8430306), k = 7 (GEV = 0.8616478).

Why are the GEV values different? Are they representing different things?

Here are the verbose files for k = 6 and 7.

https://duke.box.com/s/re6l3qfw683i563eiuikq127kv57ytuc
https://duke.box.com/s/vem25pctrxiylewassaukml9p9fybl3w

[DenisBrunet]

OK, now I see where you get confused, because it is confusing with all these steps... I will see if I can improve this state of affair.

In the meantime, here is a full recap of all the steps involved, all while using the resampling option:

Resting States, Stage 1 (subjects level):

• Input: preprocessed EEG x number of subjects
• Creates n (say 40) resampled files for each subject
• Locally calling the Clustering on each subsampled files, for each subject
• Output: (templates + error.data + verbose for optimal clustering) x 40 x number of subjects

Restings States, Stage 2 (group level):

• Input: all the resulting templates from stage 1 above, concatenated into a single, big bag of data
• Creates (say) 100 resampled files out of it
• Locally calling the Clustering 100 times:
  • Input: 1 resampled file
  • Output, in the /Resampling local directory: (templates + error.data + verbose for optimal clustering) x 100
  • Verbose:
    • Verbose will say it uses "all of the data", because locally, this is what it is actually doing, though the data comes from the parent's resampling
    • GEV is correct, and is logically quite high, as the input is already locally optimal templates
• Final merging step:
  • Merging the 100 .error.data, applying the Meta-Criterion voting method. It generates the "Stage2.Resampled.error.data" file that you showed in your snapshot. It is not the results of a single clustering, but a sort of histogram of the 100 Local Maxes.
  • Consequently, there is no verbose file attached to this error.data file, as it is not the product of an actual clustering
  • The .ep templates files are generated from another round of local re-clusterings, taking all existing resampled results with equal outcomes. Cartool re-clusters together all templates for 4 maps results (only), then 5, then 6 etc... But this clustering is done only to provide the corresponding templates to the merged error.data above.
  • The verbose files you see come from this ultimate re-clustering. They are a bit confusing, because they do not reflect the final quality of the clustering, which you can only have a glimpse of from the error.data GEV line. I think I will get rid of them...

[DenisBrunet]

On a related topic, it might be handy to add another Preset in the Segmentation Toolbox, something like:

"Stage 1 with these parameters + followed by Stage 2 with default parameters"

User provides the parameters for Stage 1, which might need some tuning, but then Stage 2 could be run straight away with the default from Cartool. This would do a single operation for a single condition...
What do you think? Or do you prefer to always control the 2 stages manually?

[armenbagdasarov]

Ah ok! Yes, cleaning up some of the areas that may cause confusion sounds like a great idea. I understand now.

So, which GEV value do we report in papers? Only at the individual level from the backfitting? I am not sure everyone is reporting the GEV values from the same step in their papers.

I think a single operation could be helpful for some individuals, though I like to control the stages manually sometimes.

[DenisBrunet]

Ah ok! Yes, cleaning up some of the areas that may cause confusion sounds like a great idea. I understand now.

I just made a release (5.04.02) that cleans-up the spurious verbose files. That was a side-effects of recent changes, and was not really intended... It has the Back-Fitting bug fixed, too (hopefully).

So, which GEV value do we report in papers? Only at the individual level from the backfitting? I am not sure everyone is reporting the GEV values from the same step in their papers.

If you were to report a number, I would go to the GEV track of the final error.data, f.ex. here 85.2%:

But this is only the mean GEV of the 100 Stage 2 segmentations GEV's, which are computed from the resampled templates that comes from the stage 1 segmentations, which are themselves computed from the GFP Peaks of the original data. Quite a lot of steps, and it does not really say how much your original data is explained, which is given by the Back-Fitting GEV...