Feat: convert MATLAB/RAVEN CI workflows to Python#1027
Merged
Conversation
edkerk
changed the title
3 tasks
Converts the three MATLAB/RAVEN GitHub Actions workflows to Python using the raven-toolbox, so they no longer need MATLAB or the self-hosted runner: - yaml-conversion: testYamlConversion.py (read/write round-trip, GLPK). - check-metabolictasks: testMetabolicTasks.py (check_tasks with close_boundaries). - gene-essentiality: estimateEssentialGenes.py + taskEssentialGenes.py + evaluateHart2015Essentiality.py + geneEssentiality.py, building cell-line tINIT models from the Hart 2015 RNA-seq data and comparing predictions with the Hart 2015 CRISPR fitness screen (Gurobi via the Gurobi_Eduard secret). Adds a reevaluate-gene-essentiality workflow that regenerates the summary from the committed per-gene matrix without re-solving, and the Python requirements files.
edkerk
force-pushed
the
worktree-matlab-to-python-workflows
branch
from
July 11, 2026 19:55
5698cf0 to
591b41a
Compare
edkerk
changed the title
checkout v4->v7, setup-python v5->v6, git-auto-commit-action v5->v7.
edkerk
force-pushed
the
worktree-matlab-to-python-workflows
branch
from
July 11, 2026 21:09
7b06a96 to
aa72e00
Compare
edkerk
force-pushed
the
worktree-matlab-to-python-workflows
branch
from
July 11, 2026 21:48
aa72e00 to
8be90bd
Compare
Classic get_init_model kept every isozyme gene of each retained reaction, unlike MATLAB getINITModel2 (called with removeGenes=true), so context models carried unexpressed alternatives that masked genes essential for the expressed isozyme and depressed the essentiality metrics. Apply remove_low_score_genes to each context model with the tissue gene scores, keeping the highest-scoring isozyme per OR rule.
edkerk
force-pushed
the
worktree-matlab-to-python-workflows
branch
from
July 11, 2026 21:53
8be90bd to
888f4e8
Compare
find_task_essential_reactions runs sequentially per task with no solver time limit, so a single hard task can stall the step (as happens on develop's model, 6h timeout). Add diagnoseTaskEssential.py to time each task with a per-solve cap and an overall budget, exposed via a 'diagnose' workflow_dispatch input, to locate which task/reaction is pathological.
…EMOTE Combines yaml-validation and yaml-conversion into one Model file checks workflow: - Removes the "Basic MEMOTE" step (memoteTest.py) whose two checks (no duplicate reactions; reactions have metabolites) are a strict subset of the full MEMOTE suite the Model QC workflow runs, and are also covered by MACAW's duplicate_test. yaml-validation no longer needs the memote-docker container. - Round-trip now runs with both cobra (no RAVEN toolbox) and raven-toolbox, so the committed .yml is verified canonical for both toolchains. - Keeps the YAML lint and model/annotation-file consistency (sanityCheck) checks.
…ity comment command - Drop the reevaluate-gene-essentiality workflow (the recompute script reevaluateGeneEssentiality.py is kept, e.g. for a future /reevaluate command). - Add pr-command.yml: a collaborator-gated issue_comment dispatcher so maintainers can start the (hours-long) gene-essentiality run by commenting '/run gene-essentiality' on a pull request. Same-repo branches only; forks get a clear message. Structured so more commands are easy to add.
Recompute gene-essential_summary.md from the committed matrix (no solve) on the PR branch when a maintainer comments '/reevaluate gene-essentiality'. Same-repo only.
Reconcile the CI migration with the QC consolidation (#1056) and combined QC workflow (#1059) merged into develop meanwhile: - gene-essentiality.yml: keep this branch's pull_request guard and updated comment wording, but post via github-script (develop dropped the deprecated comment-pr action and the commentGeneEssential.md template). - yaml-validation.yml: accept its removal; model-file-checks.yml now does the YAML lint. - model-file-checks.yml: drop the model/annotation consistency step - sanityCheck.py was removed on develop and its check is now in qcModelChecks.py (Model QC workflow). - README.md: use develop's model QC / MEMOTE / MACAW result mentions.
Model quality report
Structural checksDuplicate keys (model unloadable) and no growth block the merge; the other rows are non-blocking.
Model QC reports
MACAW and mass/charge balance
Model file and metabolic tasks
MEMOTETotal score: 20.2% (core subset) 0
Per-test scores
The score above is the fast core subset. Comment Gene essentiality (Hart 2015)Not run automatically (it takes hours). Comment ❌ = a count rose vs the target branch (regression) · Full workflow run · this comment is edited as results come in |
Consolidate the model-file-checks and check-metabolictasks workflows into the one Model QC workflow, reported in the single QC comment: - add YAML round-trip (cobrapy + RAVEN), YAML lint and the metabolic tasks as steps; each writes a one-line pass/fail status the comment shows in a new 'Model file and metabolic tasks' section, and the final gate fails the build on any of them. - combine the two metabolic-task lists: testMetabolicTasks.py now runs essential and verification in one job (they were a 2-way matrix of the same script, not two tests). - delete model-file-checks.yml and check-metabolictasks.yml. Gene essentiality (hours long) is now run on demand only: remove its pull_request trigger, and the QC comment states it is not run with the /run command to run it. /run gene-essentiality passes the PR number so the result still posts its own comment. (Full MEMOTE on demand is not wired yet - noted for follow-up.)
The full MEMOTE suite (hours long) no longer runs automatically on PRs to main; Model QC runs only the fast core subset on every pull request. Comment /run memote to run the full suite: pr-command dispatches memote-full.yml on the PR branch with the PR number, it runs the full suite, commits the score and updates the Model QC comment in place (the comment now says the row is the subset and how to get the full score). The post-qc-comment action gains an issue-number input so a workflow_dispatch run, which has no pull-request context, can target the PR.
Pull MEMOTE out of the Model QC reports table into its own section: the total score with its mode (core subset / full), a per-section score table with a delta versus the target branch, and a collapsible per-test breakdown. buildReport parses the section and detailed tables that memoteSnapshot.py already writes.
|
Gene essentiality was run on this pull request via GH Actions. Summary: Gene essentiality vs Hart 2015 fitness genes
The full per-gene essentiality matrix is committed to
|
Install raven-toolbox from fix/ftinit-raven-parity (PR #69) so the gene essentiality workflow picks up the corrected task-essential-reaction discovery, which now matches RAVEN's checkTasks (397 essential reactions on Human-GEM v2.0.0 instead of 259). Revert to the default branch after that PR merges.
Rewrite estimateEssentialGenes.py to reproduce the DevelopWBM Gen_ftINIT_models Human2 pipeline instead of classic tINIT2: - Port prepHumanModelForftINIT: remove drug reactions and amino-acid-triplet reactions (getAATripletReactions matches MATLAB at 735) and MAR13081, load spontaneous reactions from model/reactions.tsv, then run prep_init_model once (== prepINITModel) with ext_comp='e' and the protein/pool custom-ignore list. - Build each cell-line model with ftinit(series='1+1', gene_scores=..., fill_gaps=True), i.e. ftINIT with removeGenes=true and useScoresForTasks=true, scoring reactions per cell line from the Hart 2015 expression. - Essential genes still via find_task_essential_genes (== checkTasksGenes). Raise the recursion limit for cobra's recursive deepcopy of the genome-scale model during gap-filling.
find_task_essential_genes keyed each task's pFBA flux set by task.id, but the metabolic task list reuses 5 ids across 57 tasks (ER/BS/SU/IC/GR), so same-id tasks overwrote each other's flux set. The per-task filter then tested genes against the wrong task's flux distribution and skipped genes essential for the overwritten tasks, under-counting by ~50 (226 vs RAVEN checkTasksGenes 276 on the same context model). Pair each task with its own flux set by position; the count now matches RAVEN exactly (276/276 on the DLD1 model).
…table gene-essential.csv now holds, per cell line, the confusion class of each gene's prediction against the Hart 2015 fitness call (TP/TN/FP/FN, or P/. when Hart did not score the gene) instead of a bare yes/no prediction, so the experimental truth and the correctness are visible per gene. All model genes are kept as stable rows, so a diff between runs shows only changed cells and a gene gaining FN/FP marks a regression. The Hart-to-Ensembl symbol mapping is factored into a shared experimental_status() helper reused by the summary evaluator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main improvements in this PR:
Converts the three MATLAB/RAVEN GitHub Actions workflows to Python using the raven-toolbox, so they no longer require MATLAB or the self-hosted runner.
ubuntu-latest)yaml-conversiontestYamlConversion.mtestYamlConversion.pycheck-metabolictaskstestMetabolicTasks.mtestMetabolicTasks.pygene-essentialityestimateEssentialGenes.m+evaluateHart2015Essentiality.mcheck_tasks(model, tasks, close_boundaries=True)(replacing the MATLABaddBoundaryMetsstep), pinned to GLPK. 69 essential and 21 verification tasks pass.estimateEssentialGenes#970). Writesdata/testResults/gene-essential.csv.reevaluate-gene-essentialityworkflow that regenerates the summary from the committed per-gene matrix without re-solving, plus the Python requirements files.The MATLAB CI scripts are kept in
code/test/alongside the Python ports for now.Action required before merge: the
gene-essentialityworkflow needs a full Gurobi license in theGurobi_Eduardsecret (WLS or named-user); the size-limited license bundled with pipgurobipycannot handle a genome-scale model. The workflow writes the secret to$HOME/gurobi.lic, pointsGRB_LICENSE_FILEat it, and fails early with a clear message if the secret is absent.I hereby confirm that I have:
data/deprecatedIdentifiers/. (No reactions or metabolites are changed in this PR.)developas target branch, and will be resolved with a squash-merge.mainas target branch, and will be resolved with a merge commit.