Merge pull request #14 from nancytanaka1/chore/tidy-up-post-public

docs+refactor: reframe Mullenbach 2018 from benchmark to inspiration; deprecate compare_to_mullenbach

Author: nancytanaka1

DECISIONS.md

@@ -13,6 +13,7 @@ Format: `[Decision]: [What was chosen] — [Why] — [Alternatives considered]`
 - **2026-04-20** — Chunk-and-max-pool over full notes; Longformer deferred
 - **2026-04-21** — Defer date un-shifting to Silver; EDA chart flagged as diagnostic-only
 - **2026-04-23** — TF-IDF + LR baseline ships on the F1 story; P@k floor lowered to informational
+- **2026-04-26** — Reframing Mullenbach 2018 from benchmark to inspiration
 
 ---
 
@@ -136,3 +137,30 @@ would be portfolio-irrelevant. Real F1 numbers will come from
 **Conclusion** — loop validated end-to-end. Tokenization, chunking, fp16 training,
 MLflow logging, and TrainingArguments wiring all confirmed working. Ready to scale
 on Databricks GPU.
+
+
+## 2026-04-26 — Reframing Mullenbach 2018 from benchmark to inspiration
+
+Earlier drafts of the README presented numerical deltas between this work's
+results on MIMIC-IV/ICD-10 top-50 and Mullenbach et al. 2018's results on
+MIMIC-III/ICD-9 top-50 (Table 5). On review, the comparison is
+methodologically invalid: different dataset, different coding system,
+different cohort, different label space. Numerical proximity ("Micro F1
+within 0.003") is confounded by all four factors and does not constitute
+benchmark equivalence.
+
+Decision: drop all numerical comparisons to Mullenbach 2018 from the README,
+the metric targets, and the MLflow logged metrics. Retain the methodological
+inheritance (multi-label framing, patient-level splits, P@k as
+coder-assist-relevant, top-50 label cardinality) and cite the paper as the
+intellectual antecedent.
+
+A future apples-to-apples reproduction on MIMIC-III/ICD-9 — using
+Mullenbach's published cohort logic and label set — would make benchmark
+comparison meaningful. Tracked as future work; out of scope for the current
+job-search-window deliverable.
+
+Affected:
+- README.md (headline, §1, §6, §14)
+- src/mimic_icd_coder/evaluate.py (compare_to_mullenbach function deprecated)
+- MLflow logged metrics (delta_vs_caml_* removed from baseline runs going forward)

README.md

@@ -6,7 +6,7 @@
 **Evaluation data:** held-out patient-level test split (n=12,091 admissions, 6,567 patients, seed 42)
 > **TL;DR** — 30-second skim.
 >
-> - **Baseline beats Mullenbach 2018 CAML on F1.** Micro F1 0.617 (+0.003 vs. CAML), Macro F1 0.584 (+0.052). Trails on P@5 (0.526 vs. 0.609) — a deliberate trade from `class_weight="balanced"` + F1-optimal thresholds.
+> - **Baseline ships strong absolute numbers.** Micro F1 0.617, Macro F1 0.584 on held-out patient-level test split (n=12,091, 6,567 patients). P@5 = 0.526 — a deliberate trade from `class_weight="balanced"` + F1-optimal per-label thresholds.
 > - **Signature failure pattern:** Z87.891 (personal history of nicotine dependence) is systematically over-fired in 9 of the top 10 confusion pairs at 36–49% FP rates on unrelated conditions. Root cause is calibration, not representation — a transformer encoder alone will not fix this.
 > - **Pre-registered transformer predictions:** F17.210 (current vs. former smoker), K59.00 (buried constipation meds), and Z23 (buried immunization) should exit the worst-10 after Bio_ClinicalBERT fine-tuning. Specific F1 targets and falsification conditions are at the bottom of this document.
 

reports/EDA_Report.docx

@@ -15,19 +15,19 @@ format:
 
 *To be filled after baseline training completes.*
 
-One paragraph: model choice, cohort, headline metrics, delta versus Mullenbach et al. 2018.
+One paragraph: model choice, cohort, absolute headline metrics, key MLOps disciplines (patient-level splits, per-label thresholds, reproducibility, calibration analysis).
 
 # Research Question
 
-Given a free-text discharge summary, predict the set of ICD-10 codes assigned to that admission. Benchmark against Mullenbach et al. 2018 CAML (MIMIC-III top-50 ICD-9).
+Given a free-text discharge summary, predict the set of ICD-10 codes assigned to that admission. The multi-label framing, patient-level evaluation discipline, and top-50 cardinality are inherited from Mullenbach et al. 2018 (CAML, MIMIC-III/ICD-9 top-50); this work targets MIMIC-IV/ICD-10 and does not claim apples-to-apples benchmark equivalence — see [`DECISIONS.md`](../DECISIONS.md) 2026-04-26.
 
 # Data
 
 Cohort construction, composition, and limitations are documented in [`reports/data_card.md`](data_card.md). Headline numbers:
 
 - MIMIC-IV-Note v2.2 + MIMIC-IV v3.1 Hosp, joined on `hadm_id`.
 - Top-50 ICD-10 codes covering 91.04% of admissions; long tail spans 19,440 distinct codes.
-- Modelable cohort: **122,288 admissions**.
+- Modelable cohort: **122,283 admissions**.
 - Splits: patient-level 80/10/10 by `subject_id`, seed = 42.
 
 # Methods
@@ -47,14 +47,16 @@ Full rationale in [`DECISIONS.md`](../DECISIONS.md).
 
 ## Headline Metrics (Test Split)
 
-| Metric | Baseline (TF-IDF + LR) | Chunked Bio_ClinicalBERT | Mullenbach 2018 CAML (MIMIC-III top-50) | Target | Floor | Target Δ vs. CAML |
-|---|---|---|---|---|---|---|
-| Micro F1 | **0.617** | TBD | 0.614 | ≥ 0.70 | 0.55 | +0.086 |
-| Macro F1 | **0.584** | TBD | 0.532 | ≥ 0.55 | 0.40 | +0.018 |
-| P@5 | 0.526 | TBD | 0.609 | ≥ 0.70 | — | +0.091 |
-| P@8 | 0.433 | TBD | n/a (not reported for top-50) | ≥ 0.65 | — | — |
+Test-split results on MIMIC-IV-Note v2.2 + MIMIC-IV v3.1 Hosp, top-50 ICD-10 codes. Patient-level held-out test split, n=12,091 admissions, seed=42.
 
-CAML baseline values are from Mullenbach et al. 2018, Table 5 ("Results on MIMIC-III, 50 labels"), CAML row. P@8 is not reported in Table 5 (the top-50 setting); the paper's P@8 values (0.709 for MIMIC-III full, 0.523 for MIMIC-II full) are from different evaluation settings and are not used here. See [`src/mimic_icd_coder/evaluate.py::MULLENBACH_CAML_TOP50`](../src/mimic_icd_coder/evaluate.py) for the citation.
+| Metric | Baseline (TF-IDF + LR) | Chunked Bio_ClinicalBERT | Target | Floor |
+|---|---|---|---|---|
+| Micro F1 | **0.617** | TBD | ≥ 0.70 | 0.55 |
+| Macro F1 | **0.584** | TBD | ≥ 0.55 | 0.40 |
+| P@5 | 0.526 | TBD | ≥ 0.70 | — |
+| P@8 | 0.433 | TBD | ≥ 0.65 | — |
+
+Targets are absolute, not benchmark-relative. They reflect the operational threshold for "the transformer branch is delivering value over the baseline." Numerical comparison to Mullenbach et al. 2018's published numbers is not methodologically valid because this work targets MIMIC-IV/ICD-10 (different dataset, different coding system, different cohort, different label space) — see [`DECISIONS.md`](../DECISIONS.md) 2026-04-26.
 
 ## Per-Label Error Analysis
 
@@ -74,23 +76,26 @@ Macro F1 reported by:
 
 Temporal fairness deferred pending Silver-stage date un-shifting. Race-stratified analysis is deferred pending Silver/Gold integration of the demographic columns — race is already ingested to Bronze (per `feat/ingest-demographic`), but the fairness slicing pass has not been implemented. See [`DECISIONS.md`](../DECISIONS.md) 2026-04-21.
 
-# Comparison to Mullenbach 2018
+# Methodological inheritance from Mullenbach 2018
+
+Mullenbach et al. 2018 (CAML) established the multi-label ICD-coding benchmark on MIMIC-III/ICD-9 top-50. This work inherits its methodological framing — multi-label classification with sigmoid + per-label decisions, patient-level evaluation, P@k as a coder-assist-relevant metric, top-50 label cardinality as a tractable problem size. These are methodological inheritances, not benchmark equivalences.
 
-Metric deltas are reported with the following caveats:
+Numerical comparison between this work's results and Mullenbach's published numbers is **not methodologically valid**. The work differs from Mullenbach 2018 along all four axes that matter:
 
-- Cohort differences — size (47K MIMIC-III vs. 122K here), era (pre-2015 vs. 2008–2019), code system (ICD-9 vs. ICD-10).
-- ICD-10 sub-code fragmentation redistributes common conditions (CHF, CKD, COPD, T2DM) across several codes, so top-50 label spaces are not identical.
-- CAML trained on full-length notes with custom attention; this work uses chunked Bio_ClinicalBERT.
+- **Different dataset.** Mullenbach used MIMIC-III v1.4; this work uses MIMIC-IV v3.1 + MIMIC-IV-Note v2.2.
+- **Different coding system.** Mullenbach used ICD-9-CM; this work uses ICD-10-CM. Label spaces are non-overlapping; the top-50 codes in each are different sets covering different clinical concepts.
+- **Different cohort.** Different size (47K MIMIC-III vs. 122K here), different era (pre-2015 vs. 2008–2019), different documentation conventions.
+- **Different model architecture.** CAML used full-length notes with custom label-attention; this work uses chunked Bio_ClinicalBERT.
 
-Deltas are informative but not strictly apples-to-apples.
+A future apples-to-apples reproduction on MIMIC-III/ICD-9 — using Mullenbach's published cohort logic and label set — would make benchmark comparison meaningful. Tracked as future work in [`DECISIONS.md`](../DECISIONS.md) 2026-04-26; out of scope for the current deliverable.
 
 # Limitations
 
 See [`reports/data_card.md`](data_card.md). Key points relevant to eval interpretation:
 
 - Cohort is 48% of all ICD-10 admissions in v3.1 Hosp, limited by MIMIC-IV-Note v2.2 sampling.
 - Temporal and race-stratified fairness evaluation are deferred, not absent.
-- Top-50 K is comparability-driven, not coverage-optimal; raising K would improve coverage but weaken the CAML comparison.
+- Top-50 K is methodologically inherited from Mullenbach 2018 as a tractable problem size, not coverage-optimal. Raising K would improve coverage but the choice here keeps the methodological framing aligned with prior published work even though direct numerical comparison is invalid (see *Methodological inheritance from Mullenbach 2018* above).
 
 # References
 

reports/baseline_error_analysis.md

@@ -18,38 +18,40 @@ Template: Mitchell et al. 2019.
 
 ## Intended Use
 
-- **Primary use:** Production-grade benchmark for clinical NLP ICD-10 auto-coding on MIMIC-IV, deployed as an Azure Databricks Model Serving endpoint for reproducible inference. Demonstrates the full MLOps lifecycle — feature engineering, model validation against a published benchmark (Mullenbach et al. 2018 CAML), Unity Catalog Model Registry, drift monitoring.
+- **Primary use:** Reproducible end-to-end clinical NLP + MLOps reference build for ICD-10 auto-coding on MIMIC-IV, deployed as an Azure Databricks Model Serving endpoint for reproducible inference. Demonstrates the full MLOps lifecycle — feature engineering, multi-label model training and evaluation discipline, Unity Catalog Model Registry, drift monitoring. Methodologically inspired by Mullenbach et al. 2018 (CAML) on MIMIC-III/ICD-9; this work targets MIMIC-IV/ICD-10 and does not claim apples-to-apples benchmark equivalence (see [`DECISIONS.md`](../DECISIONS.md) 2026-04-26).
 - **Out of scope:** Clinical decision support, real-world coding workflows, any production health-system deployment. The MIMIC-IV license restricts use to research and benchmarking on credentialed PhysioNet data only.
 - **Users:** Data scientists and ML engineers evaluating production clinical NLP patterns; research teams benchmarking against this implementation on MIMIC-IV top-50 ICD-10.
 
 ## Training Data
 
 Cohort definition, composition, and limitations are documented in [`reports/data_card.md`](data_card.md). Model-specific details:
 
-- Modelable cohort: **122,288 admissions**.
+- Modelable cohort: **122,283 admissions**.
 - Splits: patient-level 80/10/10 by `subject_id`, seed = 42.
 - Labels: top-50 ICD-10 codes, covering 91.04% of cohort admissions.
 
 ## Evaluation Data
 
 - **Test split:** Held-out 10% of patients, no overlap with train or val.
 - **Metrics:** Micro F1, Macro F1, P@5, P@8, per-label F1.
-- **Reference baseline:** Mullenbach et al. 2018 CAML on MIMIC-III top-50 ICD-9 — Table 5 of the paper. Micro F1 = 0.614, Macro F1 = 0.532, P@5 = 0.609. P@8 is not reported for the top-50 setting.
+- **Methodological inspiration:** Mullenbach et al. 2018 (CAML) established the multi-label ICD-coding benchmark on MIMIC-III/ICD-9 top-50 — its multi-label framing, patient-level evaluation discipline, and top-50 cardinality are inherited here. Numerical comparison is *not* methodologically valid because this work targets MIMIC-IV/ICD-10 (different dataset, different coding system, different cohort, different label space). See [`DECISIONS.md`](../DECISIONS.md) 2026-04-26.
 
-Full evaluation methodology and comparison caveats are in [`reports/eval_report.qmd`](eval_report.qmd).
+Full evaluation methodology is in [`reports/eval_report.qmd`](eval_report.qmd).
 
 ## Performance
 
-| Metric | Target | Floor | Mullenbach 2018 CAML (MIMIC-III top-50) | Target Δ vs. CAML | Result |
-|---|---|---|---|---|---|
-| Micro F1 | ≥ 0.70 | 0.55 | 0.614 | +0.086 | **0.617** (baseline) / TBD (transformer) |
-| Macro F1 | ≥ 0.55 | 0.40 | 0.532 | +0.018 | **0.584** (baseline) / TBD (transformer) |
-| P@5 | ≥ 0.70 | — | 0.609 | +0.091 | 0.526 (baseline) / TBD (transformer) |
-| P@8 | ≥ 0.65 | — | n/a (Mullenbach Table 5 reports P@5 only) | — | 0.433 (baseline) / TBD (transformer) |
+Test-split results on MIMIC-IV-Note v2.2 + MIMIC-IV v3.1 Hosp, top-50 ICD-10 codes. Patient-level held-out test split, n=12,091 admissions, seed=42.
 
-Baseline = TF-IDF (1–2 gram, min_df=5, max_features=200k) + One-vs-Rest Logistic Regression (`class_weight="balanced"`, liblinear solver, max_iter=1000) with per-label F1-optimal thresholds. Evaluated on held-out patient-level test split, n=12,091 admissions, seed=42. MLflow run `4e577699a67a4027bc27628e9b237ac5`. Baseline beats Mullenbach CAML on both F1 metrics; P@k is intentionally de-emphasized for the baseline and is the transformer branch's primary gate — see [`DECISIONS.md`](../DECISIONS.md) 2026-04-23.
+| Metric | Target | Floor | Result |
+|---|---|---|---|
+| Micro F1 | ≥ 0.70 | 0.55 | **0.617** (baseline) / TBD (transformer) |
+| Macro F1 | ≥ 0.55 | 0.40 | **0.584** (baseline) / TBD (transformer) |
+| P@5 | ≥ 0.70 | — | 0.526 (baseline) / TBD (transformer) |
+| P@8 | ≥ 0.65 | — | 0.433 (baseline) / TBD (transformer) |
 
-CAML values from Mullenbach et al. 2018 Table 5 (MIMIC-III, 50 labels). See [`src/mimic_icd_coder/evaluate.py::MULLENBACH_CAML_TOP50`](../src/mimic_icd_coder/evaluate.py) for the citation.
+Targets are absolute, not benchmark-relative. They reflect the operational threshold for "the transformer branch is delivering value over the baseline" rather than a literature comparison.
+
+Baseline = TF-IDF (1–2 gram, min_df=5, max_features=200k) + One-vs-Rest Logistic Regression (`class_weight="balanced"`, liblinear solver, max_iter=1000) with per-label F1-optimal thresholds. MLflow run `4e577699a67a4027bc27628e9b237ac5`. P@k is intentionally de-emphasized for the baseline and is the transformer branch's primary gate — see [`DECISIONS.md`](../DECISIONS.md) 2026-04-23.
 
 ## Fairness Analysis
 

reports/eval_report.qmd

@@ -175,19 +175,36 @@ def evaluate_multilabel(
     return result
 
 
-def compare_to_mullenbach(result: EvalResult) -> dict[str, float]:
-    """Return absolute deltas from the Mullenbach 2018 CAML top-50 baseline.
+def compare_to_mullenbach(result: EvalResult) -> dict[str, float]:  # noqa: ARG001
+    """**Deprecated.** Returns numeric deltas from Mullenbach 2018 CAML top-50.
+
+    .. deprecated:: 2026-04-26
+        See ``DECISIONS.md`` 2026-04-26 — "Reframing Mullenbach 2018 from
+        benchmark to inspiration." Mullenbach's published numbers are on
+        MIMIC-III/ICD-9 top-50; this project trains on MIMIC-IV/ICD-10
+        top-50. Different dataset, different coding system, different
+        cohort, different label space. Numerical proximity does not
+        constitute benchmark equivalence and the deltas are
+        methodologically invalid as a comparison.
+
+        Retained as a no-op-with-warning so any downstream caller (e.g.
+        ``pipeline.run_evaluate_test``) surfaces the issue at runtime
+        instead of silently logging meaningless ``mullenbach_*_delta``
+        metrics. Will be removed once
+        ``pipeline.run_evaluate_test`` stops calling it.
 
-    Positive = we beat the baseline.
+    Returns:
+        Empty dict. Issues a ``DeprecationWarning``.
     """
-    deltas = {
-        "micro_f1_delta": result.micro_f1 - MULLENBACH_CAML_TOP50["micro_f1"],
-        "macro_f1_delta": result.macro_f1 - MULLENBACH_CAML_TOP50["macro_f1"],
-    }
-    if 5 in result.precision_at_k and "p_at_5" in MULLENBACH_CAML_TOP50:
-        deltas["p_at_5_delta"] = result.precision_at_k[5] - MULLENBACH_CAML_TOP50["p_at_5"]
-    # Mullenbach Table 5 does not report P@8 for the 50-label setting — no
-    # apples-to-apples baseline exists, so we deliberately skip the delta.
-    if 8 in result.precision_at_k and "p_at_8" in MULLENBACH_CAML_TOP50:
-        deltas["p_at_8_delta"] = result.precision_at_k[8] - MULLENBACH_CAML_TOP50["p_at_8"]
-    return deltas
+    import warnings
+
+    warnings.warn(
+        "compare_to_mullenbach is deprecated as of 2026-04-26. The numeric "
+        "deltas are methodologically invalid (different dataset, different "
+        "coding system, different cohort vs. Mullenbach 2018). See "
+        "DECISIONS.md 2026-04-26. This function now returns an empty dict; "
+        "remove the call site to silence this warning.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+    return {}

reports/model_card.md

@@ -419,10 +419,15 @@ def run_evaluate_test(cfg: AppConfig, paths: Paths) -> dict[str, float]:
     scoreboard.
 
     Returns:
-        A flat dictionary of test metrics. Also includes ``mullenbach_*_delta``
-        keys — how far our numbers are from the published CAML baseline.
+        A flat dictionary of test metrics: micro/macro F1, AUC, AUPRC, P@k.
+
+    Note:
+        ``mullenbach_*_delta`` keys were previously included here. Removed
+        per ``DECISIONS.md`` 2026-04-26 — Mullenbach 2018's published numbers
+        are MIMIC-III/ICD-9 top-50; this work trains on MIMIC-IV/ICD-10
+        top-50, so the numerical deltas are methodologically invalid.
     """
-    from mimic_icd_coder.evaluate import compare_to_mullenbach, evaluate_multilabel
+    from mimic_icd_coder.evaluate import evaluate_multilabel
     from mimic_icd_coder.models.baseline import BaselineModel
 
     silver = pd.read_parquet(paths.silver / "notes.parquet")
@@ -442,7 +447,5 @@ def run_evaluate_test(cfg: AppConfig, paths: Paths) -> dict[str, float]:
         top_k_list=cfg.evaluation.top_k_metrics,
     )
     metrics = result.to_dict()
-    deltas = compare_to_mullenbach(result)
     logger.info("test.metrics", **metrics)
-    logger.info("test.vs_mullenbach", **deltas)
-    return {**metrics, **{f"mullenbach_{k}": v for k, v in deltas.items()}}
+    return metrics

src/mimic_icd_coder/evaluate.py

@@ -145,6 +145,11 @@ def test_pipeline_baseline_beats_random(synthetic_workspace: tuple[AppConfig, Pa
 
     test_metrics = run_evaluate_test(cfg, paths)
     assert "micro_f1" in test_metrics
-    assert "mullenbach_micro_f1_delta" in test_metrics
+    # mullenbach_*_delta keys were removed per DECISIONS.md 2026-04-26
+    # (different dataset / coding system / cohort vs. Mullenbach 2018);
+    # asserting on test_metrics["micro_f1"] alone is the right contract now.
+    assert not any(
+        k.startswith("mullenbach_") for k in test_metrics
+    ), "mullenbach_* keys must not be present after the 2026-04-26 reframe"
     # Test metrics on a tiny synthetic corpus may trail val; sanity bound only
     assert test_metrics["micro_f1"] > 0.1