Optimize END xgb#17
Merged
Merged
Conversation
* updated MIMIC preprocessing to be compatible with short term outcomes * updated meta data processing for END * updated imaging missingness computation and table 1 * rerun shap figures * refactored inference plotting into a reusable function
…into optimize_end_xgb
…into optimize_end_xgb
JulianKlug
added a commit
that referenced
this pull request
Apr 24, 2026
) * Early neurological deterioriation - working preprocessing pipeline - working dataloader - trial of gridsearch * Fixed wrong length of dataset during training __len__ method should return num batch count * imbalance factor: take into account ratio pos:neg samples in loss * Cluster hyperoptimisation for short term outcomes transformer * deactivate conda before launching sbatch * Access remote redis cluster * use storage_host in all subprocesses to access other nodes * adapt to yggdrasils specs * add database process on slurm * clarify naming scheme for gridsearch logs * Timeseries prediction using Encoder/Decoder * updated slurm configs * Gridsearch with decoder * mean aggreation of cosine similarity * aggregate xgb for end prediction * xgb gridsearch eval * Added val evaluation for xgb * Try AUPRC as eval metric in XGB * Validation set evaluation function * Time to event prediction pipeline * Ensure training right model... * adapted callbacks for all metrics * ensure direction for early stopping * use correct direction for early stopping * use optuna dashboard * Added server readme instructions * Added TTE gridsearch evaluation * Evaluate validation classification accuracy while training on TTE transformer * Added evluation for tte prediction * Added validation set evaluation function * Corrected Encoder validation function * added filter function for old checkpoints * Hand tuned END Transformer Encoder * updated transformer encoder eval function * Meta data (Table 1) for short term outcomes and transformer encoder / decoder evaluation exploration * encoder-decoder inference * Decoder gridsearch finetuned for ROC AUC * adapted master launcher for cluster gridseearch (ROC in decoder evaluation * updated logs * implemented oversampling for data sampler * add loss function as hyperopt parameter * add oversampling to hyperopt config * segregate loss functions * Updated requirements * new implementation of AP loss * enable weighted cosine similarity computation * update aploss and cosine similarity metric * log val auprc * specify CV fold for encoder decoder evaluation * Cleaning imports * import normalisation data on cluster eval of encoder/decoder * Encoder Decoder evaluation - find right CV fold * Preprocessing: allow multiple END events * Added ability to predict within interval events and not only exact timing * ensure imbalance_factor is in config * log trial meta parameters * increase cluster process memory requirement * fix config loading in encoder validation * Use loss function from config on model eval decoder * tune xgb * fix in decoder inference * xgb: save auroc + auprc * Preapre xgb gridsearch for cluster usage * Prepare encoder hyperopt eval for multiple datasets * change file name & xgb eval * testing and shap compution for xgb model * END: xgb shap analysis - top features * identify patients for imaging extraction * added neutrophil count & phospate, log variable selection * extract analysed imaging results from registry * updated imaging preprocessing to accept long format and integrated imaging data from registry * automate selection of parameters to normalize * make winsorising optional * fixed shap computation * optimisation of the end XGB model * evaluating combos * updated SLURM cluster launcher * updated report * added rolling window features to XGB * document XGB feature aggregration method * added retrain function for xgb * added computation of final XGB model, testing of model, computation of baseline and comparison of xgb with baseline * figure creation functions * updated MIMIC preprocessing to be compatible with short term outcomes * updated meta data processing for END * computation of shap values * XGB end shap figures * updated top predictor selection * added shap plot for imaging features * external validation for short term outcomes * creation of combined shap plot * compute and plot end event rates * updated imaging missingness computation and table 1 * Updated MIMIC preprocessing (#15) * updated MIMIC preprocessing to be compatible with short term outcomes * updated meta data processing for END * updated imaging missingness computation and table 1 * create END table 1 for mimic * rerun shap figures * refactored inference plotting into a reusable function * compare end rates between datasets * Optimize END xgb (#17) * updated MIMIC preprocessing to be compatible with short term outcomes * updated meta data processing for END * updated imaging missingness computation and table 1 * rerun shap figures * refactored inference plotting into a reusable function * updated inference plotting to account for new aggregation method * update local analyses * added ppv npv npv plot * Added auroc over time plots for END analysis * add overall column to table1 * compute complete end event rates * added duration of monitoring analysis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.