A spatial biomarker is only useful if you can also say how confident you are, and why.
This is the layer that says when to trust a spatial prediction - and when to abstain.
Problem - Results - What this is / isn't - How it works - Step-by-step - Recommendations - Setup - Reproduce
SpatialTrust-IQ stress-tests spatial biology predictions and reports, for a given input, whether the answer can be trusted or should be withheld. It is built and demonstrated on the real Keren et al. (2018) MIBI-TOF triple-negative breast cancer cohort. It reproduces the field's most-cited spatial biomarker - the tumour–immune mixing score - then shows exactly where it breaks under honest scrutiny, and wraps any predictive model in a calibration + out-of-distribution abstention gate.
This is a portfolio demonstration of how I approach building trustable spatial-omics workflows. It is not a clinical device.
Spatial proteomics keeps producing prognostic scores that look convincing in one cohort and then fail to hold up in the next. The mixing score from Keren et al. 2018 is the textbook case: it separated survival cleanly in the original MIBI cohort and became one of the most-cited spatial biomarkers in the field, then did not independently validate on later CyCIF/IMC cohorts (JCI Insight 2025).
Spatial foundation models hit the same wall from the other direction: commonly cited figures put accuracy dropping from roughly 89% in-distribution to around 54% on novel antibody panels and unseen tissues - with nothing warning the user that it is happening.
Both failures are the same shape: strong performance in familiar conditions, silent degradation in unfamiliar ones. This repository implements the layer that measures that gap and guards against it, and reports the result honestly - including the negative results, which are the point.
On the real 38-patient outcome cohort, under honest cross-validation and multiple-testing correction:
| Claim under test | Verdict | Evidence |
|---|---|---|
| Expert phenotypes re-derivable from shared tables | No | 29% balanced recall |
| Mixing score is a stable survival predictor | No | significant in 11% of resamples; LOO HR 1.0–1.5, p = 0.71 |
| Mixing-score HR robust to the contact-radius choice | No | HR 0.96 -> 1.37 across ~4–39 µm |
| Any spatial feature survives multiple testing | No | 0 / 20 at Benjamini–Hochberg q < 0.1 |
| Spatial features predict recurrence (n = 38) | No | LOO AUC 0.49 (95% CI 0.29–0.68), permutation p > 0.5 |
| A biomarker replicates across disjoint cohort halves | No (fragile) | frac_immune HR 0.58 vs 0.86, pooled crosses 1 |
| CD8–tumour engagement is more trustworthy than mixing | Yes | same-direction in 98% of resamples vs 62% |
| Abstaining buys accuracy where it matters | Yes | selective accuracy 0.53 -> 1.0 as coverage falls |
| The trust gate catches an out-of-distribution batch | Yes | 100% abstention on a novel-panel batch |
Every figure below is generated from the real cohort by the pipeline (figures/).
The expert phenotypes cannot be re-derived from the shared summary tables (balanced recall ~29%), so cell types are treated as a fixed input and the spatial layer - which can be recomputed - is what gets audited.
Reproduced faithfully, then stress-tested by bootstrap and leave-one-out. The mixing score stays significant in only 11% of resamples and is barely distinguishable from no effect (LOO HR 1.0–1.5, p = 0.71). CD8–tumour engagement, on the same patients, keeps its direction in 98% of resamples.
The mixing-score hazard ratio slides from 0.96 (protective) to 1.37 (harmful) purely with the contact-radius parameter, and no spatial feature survives FDR correction.
Per-patient SHAP, an honest leave-one-out AUC with a bootstrap CI and a permutation null, and an out-of-distribution gate that escalates 4 of 38 patients to a human instead of guessing.
Splitting the real cohort into disjoint patient halves and re-testing each biomarker on patients it never saw: the effect is fragile and does not replicate convincingly, echoing the published cross-platform non-replication from within the original dataset.
Wrapping any predict_proba with marker-overlap, OOD, and calibrated-confidence gates: selective accuracy rises as the gate abstains, and a genuinely out-of-distribution batch is 100% abstained - the layer that guards the commonly cited 89% -> 54% foundation-model failure.
What this is
- A trust and abstention layer for spatial-omics predictions, demonstrated end-to-end on real MIBI-TOF data.
- An honest stress-test of a famous spatial biomarker, reporting where it holds and where it breaks.
- A reproducible local pipeline: ingest, QC, provenance check, spatial features, trust audit, robustness, explainable model, cross-cohort replication, and a model-agnostic trust gate - each stage runnable on its own.
What this is not
- A new biomarker. The deliverable is the diligence machinery, not another score on the pile.
- A state-of-the-art predictor. On 38 patients the model is honestly at chance, and it is reported as such.
- A cross-platform validation study. External cohorts are stubbed via a platform-agnostic adapter; the within-cohort replication probe runs on real data only.
- A clinical or diagnostic tool.
| Stage | What happens | Why it matters |
|---|---|---|
| 1. Ingest | Load the real MIBI single-cell tables and clinical labels (~197,640 cells, 41 patients) | Real, messy data - including a mislabelled patient file handled explicitly |
| 2. QC | Documented per-cell and per-image gates: coordinates, nuclear signal, phenotype, min cells | A score is only trustworthy if the cells feeding it are real, not segmentation debris |
| 3. Provenance | Try to re-derive the expert cell types from shared markers (fails at 29% recall) | If the labels aren't reproducible, treat them as fixed input rather than pretend otherwise |
| 4. Spatial features | Compute the Keren mixing score plus interpretable neighbourhood features | A faithful reproduction of the biomarker to be audited |
| 5. Trust audit | Bootstrap stability and leave-one-out sensitivity of the score against survival | Distinguishes a stable association from one riding on a few patients |
| 6. Robustness | Sweep the contact radius; screen ~20 features with Benjamini–Hochberg FDR | Exposes parameter-driven flips and multiple-testing artefacts |
| 7. Model | Explainable recurrence model, leave-one-out CV, per-patient SHAP, OOD gate | Honest performance with a CI and a permutation null; abstains on unfamiliar patients |
| 8. Reproduce | Split into disjoint patient halves; per-cohort bootstrap CIs; meta-analysis | Tests replication on patients the biomarker never saw |
| 9. Trust gate | Wrap any predict_proba with marker-overlap, OOD, and calibrated-confidence gates |
The reusable layer that guards the foundation-model OOD failure |
The method in nine steps. Each maps to a stage in the pipeline diagram above and to a specific location in the code.
Step 1. Ingest the real single-cell data.
Load each patient's per-cell table (x, y, cell type, 36 markers) and the clinical outcome table. The source ships patient 1's expression file mislabelled; it is handled explicitly rather than dropped.
-> data.load_all_cells(...), data.load_clinical(...)
Step 2. Apply documented QC gates.
Drop cells with invalid coordinates, cells with no captured nucleus (bottom 1st percentile of nuclear signal, per image), cells with no assigned phenotype, and whole images with fewer than 200 cells. Every drop is counted and reported.
-> qc.run_qc(...)
Step 3. Check phenotype provenance before trusting anything.
Train a classifier to re-derive the paper's expert cell types from the shared markers and cross-validate it. Balanced recall is 29%, so cell types are frozen as input and the spatial layer is what gets scrutinised.
-> phenotype.validate_against_expert(...)
Step 4. Compute the mixing score and interpretable features.
mixing_score = immune–tumour contacts / immune–immune contacts within a contact radius, plus per-type fractions, contacts per cell, and CD8–tumour engagement.
-> spatial.features_table(...), spatial.mixing_score_for_image(...)
Step 5. Audit the trust of the headline score.
Fit the score against survival, then stress-test it: bootstrap resampling (sign consistency and how often it stays significant) and leave-one-patient-out (how far the HR moves).
-> trust.audit_mixing_score(...)
Step 6. Stress-test robustness and multiple testing.
Sweep the contact radius to expose parameter-driven direction flips, and screen ~20 features against survival with Benjamini–Hochberg FDR correction.
-> screen.radius_sensitivity(...), screen.screen_features(...)
Step 7. Build an explainable, uncertainty-aware model.
A gradient-boosted recurrence model with leave-one-out CV (honest for n≈38), per-patient SHAP attribution, a bootstrap AUC CI, a permutation null, and an IsolationForest OOD gate that escalates unfamiliar patients.
-> model.fit_explainable_model(...)
Step 8. Test cross-cohort / within-cohort replication.
Partition the real cohort into disjoint patient halves, re-fit each biomarker per half with bootstrap CIs, and combine with a fixed-effect inverse-variance meta-analysis (pooled HR, I²).
-> cohort.split_cohort(...), reproducibility.run_reproducibility(...)
Step 9. Guard predictions with a model-agnostic trust gate.
Wrap any probability output with three independent gates - marker overlap, OOD novelty, and isotonic-calibrated confidence - each producing an auditable abstention reason, and measure selective accuracy versus coverage.
-> trustgate.TrustGate(...).fit(...).evaluate(...)
Practices that separate a defensible spatial-biomarker result from one driven by leakage, chance, or an arbitrary parameter.
| Practice | Rationale |
|---|---|
| Gate every cell before it enters a score | A spatial coordinate does not make a segmentation artefact a real cell |
| Check whether cell types are even re-derivable | If phenotyping isn't reproducible, audit the layer you can recompute and treat labels as fixed |
| Reproduce a biomarker, then stress-test it | Bootstrap and leave-one-out separate a stable effect from one riding on a few patients |
| Sweep arbitrary parameters (e.g. contact radius) | The same score can flip clinical direction with a knob nobody reports |
| Correct for multiple testing (BH FDR) | Twenty spatial hypotheses will hand you a false positive without correction |
| Report leave-one-out AUC with CI and a permutation null | On small cohorts a single point estimate hides the uncertainty |
| Calibrate probabilities and allow abstention | Selective accuracy on answered cases beats confidently answering everything |
| Check marker-panel overlap before scoring a new batch | The literal foundation-model failure mode is absent training markers |
| Practice | Consequence |
|---|---|
| Reporting a biomarker from a single cohort as validated | It may not replicate across cohorts, or even across halves of the same one |
| Fixing the contact radius silently | The hazard ratio, and the clinical conclusion, depend on it |
| Screening many features without FDR | Uncorrected noise masquerades as signal |
| Quoting an AUC without a CI or null on small n | Chance performance can look like a result |
| Letting a model extrapolate onto out-of-distribution inputs | Silent degradation - the 89% -> 54% failure with no warning |
| Trusting cell-type labels you cannot reproduce | Downstream spatial claims inherit an unverified foundation |
Keren et al., A Structured Tumor-Immune Microenvironment in TNBC Revealed by MIBI-TOF, Cell 2018. This project uses the single-cell intermediate tables (per-cell x/y, cell type, 36 markers) and clinical survival/recurrence labels redistributed by the peer-reviewed reanalysis of Patwa et al., Multiplexed Imaging Analysis of the Tumor-Immune Microenvironment, Communications Biology 2021 (github.com/aalokpatwa/rasp-mibi).
- 41 patients, ~197,640 cells, 36 immuno-oncology markers (CD8, CD4, CD3, FoxP3, PD-1, PD-L1, IDO, Lag3, CD68, Pan-Keratin, Vimentin, SMA, and others).
- Clinical outcome (overall survival, recurrence) for 38 patients.
Cell-type codes (from the source): 2 Endothelial, 3 Mesenchyme, 4 Tumor, 5 Treg, 6 CD4_T, 7 CD8_T, 8 CD3_T, 9 NK, 10 B, 11 Neutrophil, 12 Macrophage, 13 DC, 14 DC_Mono, 15 Mono_neutrophil, 16 Other.
The endpoint here is overall survival / recurrence, not a literal immune-checkpoint-blockade response label. The trust and validation methodology is identical; an ICB-response cohort is the direct extension.
git clone https://github.com/ankurgenomics/spatial-trust-iq.git
cd spatial-trust-iq
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"Requirements: Python ≥ 3.10. Data is vendored under data/raw/rasp-mibi (see scripts/fetch_data.sh to refetch).
python -m spatial_trust_iq.pipeline run --all # full pipeline + all figures + HTML reportpython -m spatial_trust_iq.pipeline ingest
python -m spatial_trust_iq.pipeline qc
python -m spatial_trust_iq.pipeline provenance # figures/phenotype_provenance.png
python -m spatial_trust_iq.pipeline niche # figures/niches.png
python -m spatial_trust_iq.pipeline spatial-stats # figures/spatial_stats.png
python -m spatial_trust_iq.pipeline mixing # figures/mixing_trust.png
python -m spatial_trust_iq.pipeline robust # figures/robustness.png (FDR + radius sweep)
python -m spatial_trust_iq.pipeline model # figures/explainable_score.png + trust report
python -m spatial_trust_iq.pipeline reproduce # figures/reproducibility.png (cross-cohort)
python -m spatial_trust_iq.pipeline reproduce-real # figures/reproducibility_real.png (real within-cohort splits)
python -m spatial_trust_iq.pipeline trustgate # figures/trustgate.png (OOD / abstention wrapper)All figures land in figures/; the per-patient report in figures/reports/trust_report.html.
python3 scripts/make_diagrams.py # figures/pipeline.png + figures/architecture.pngpytest -qsrc/spatial_trust_iq/
data.py # ingest the real MIBI single-cell tables + clinical labels
qc.py # documented per-cell / per-image QC gates (incl. nuclear-signal gate)
phenotype.py # marker-based phenotyping as a data-provenance / trust check
spatial.py # mixing score + interpretable neighbourhood features (pixel units)
niche.py # cellular neighbourhoods + neighbourhood enrichment + Ripley's K
trust.py # resampling stability: where the mixing score replicates vs breaks
screen.py # FDR-corrected feature screen + radius-sensitivity sweep
model.py # explainable recurrence model + SHAP + OOD abstention + AUC CI/perm
cohort.py # platform-agnostic Cohort + adapters (Keren, spora-io, synthetic)
reproducibility.py # cross-cohort biomarker reproducibility engine + meta-analysis
trustgate.py # model-agnostic OOD / calibration / abstention wrapper
report.py # per-patient HTML trust report
figures.py # all publication figures
pipeline.py # CLI orchestrator
scripts/
make_diagrams.py # pipeline + architecture diagrams
fetch_data.sh # refetch the vendored MIBI data
tests/ # pytest suite (synthetic unit tests + guarded real-data smoke test)
figures/ # generated figures + reports/
Code: MIT. Data © original authors (Keren et al. 2018; Patwa et al. 2021), used for non-commercial research demonstration under their terms.