Skip to content

DECISIONS.md

Latest commit

 

History

History
164 lines (136 loc) · 14 KB

DECISIONS.md

File metadata and controls

164 lines (136 loc) · 14 KB
Raw

DECISIONS.md

Record every non-obvious architectural or methodological choice. Format: [Decision]: [What was chosen] — [Why] — [Alternatives considered]

Index

  • 2026-04-20 — Target top-50 ICD-10 codes
  • 2026-04-20 — Patient-level train/val/test split
  • 2026-04-20 — Use MIMIC-IV v3.1 labels joined to MIMIC-IV-Note v2.2 notes
  • 2026-04-20 — ICD-10-only cohort
  • 2026-04-20 — Bio_ClinicalBERT as primary model family; Clinical-Longformer as fallback
  • 2026-04-20 — Chunk-and-max-pool over full notes; Longformer deferred
  • 2026-04-21 — Defer date un-shifting to Silver; EDA chart flagged as diagnostic-only
  • 2026-04-23 — TF-IDF + LR baseline ships on the F1 story; P@k floor lowered to informational
  • 2026-04-26 — Reframing Mullenbach 2018 from benchmark to inspiration

2026-04-20 — Target top-50 ICD-10 codes

  • What: Predict the 50 most frequent ICD-10 codes in the MIMIC-IV Hosp diagnoses table.
  • Why: Published benchmark (Mullenbach et al. 2018) used top-50 on MIMIC-III. Direct comparability; keeps the label space tractable for BERT fine-tuning.
  • Alternatives: All ~18K ICD-10 codes (intractable), top-10 (too easy), disease-category roll-up (less useful clinically).
  • Update 2026-04-22: Cohort-aware top-50 verified after mic gold. Z20.822 (COVID exposure, post-hoc, insufficient cohort support in MIMIC-IV-Note v2.2 window) dropped; N18.3, J18.9, Y92.239, Z23 added in its place. Final cohort 122,288 admissions across 65,665 patients (per notebooks/01_eda.ipynb §10 Table B4 and reports/EDA_Report.docx); data/gold/label_names.json confirmed.

2026-04-20 — Patient-level train/val/test split

  • What: Partition by subject_id, not by hadm_id. 80/10/10 stratified by label presence.
  • Why: Admission-level splits leak patient-specific language patterns (writing style, comorbidity clusters) across train/test and inflate metrics.
  • Alternatives: Temporal split by admit date. Rejected for this study — patient-level is the methodologically standard choice for clinical NLP benchmarks; temporal split could be added as a secondary evaluation.

2026-04-20 — Use MIMIC-IV v3.1 labels joined to MIMIC-IV-Note v2.2 notes

  • What: Labels from hosp/diagnoses_icd.csv.gz (v3.1, Oct 2024); notes from note/discharge.csv.gz (v2.2, Jan 2023).
  • Why: v3.1 is the only version with the corrected diagnoses_icd table. MIMIC-IV-Note has only one public release (v2.2). hadm_id join key is stable across versions.
  • Alternatives: Downgrade labels to v2.2 (loses corrections), upgrade notes (no newer version exists).
  • Risk: Document clearly in reports/data_card.md; a small number of admissions in notes v2.2 may have been removed from v3.1 Hosp; handle as inner join.

2026-04-20 — ICD-10-only cohort

  • What: Filter to admissions where all assigned codes are ICD-10 (icd_version = 10).
  • Why: MIMIC-IV spans ICD-9 (pre-2015) and ICD-10 (post-2015). Mixing breaks the label space. ICD-10 is the operationally relevant system today.
  • Alternatives: Include ICD-9 with code mapping (GEM crosswalk). Adds complexity and mapping loss; reject for now.

2026-04-20 — Bio_ClinicalBERT as primary model family; Clinical-Longformer as fallback

  • What: Primary transformer is Bio_ClinicalBERT. Clinical-Longformer is the fallback, triggered only if Bio_ClinicalBERT misses the Micro F1 target by more than 3 points. This entry scopes the model family choice only. The context-handling mechanism (chunk-and-max-pool vs single-pass) is defined in the Chunk-and-max-pool entry below.
  • Why: Bio_ClinicalBERT is the dominant clinical-pretrained encoder with established benchmarks in clinical NLP (Alsentzer et al. 2019). It is faster per epoch than Longformer and easier to debug. Longformer's 4K context is valuable only if Bio_ClinicalBERT with chunking leaves substantial signal on the floor — an empirical question to answer with data, not a priori.
  • Alternatives: (1) Start with Longformer — rejected, slower and premature without evidence that Bio_ClinicalBERT underperforms. (2) General-domain BERT or RoBERTa — rejected, domain mismatch on clinical text. (3) Other clinical encoders (PubMedBERT, BioLinkBERT, Clinical-T5) — comparable choices; Bio_ClinicalBERT picked for direct lineage to Alsentzer 2019 and prior MIMIC precedent.

2026-04-20 — Chunk-and-max-pool over full notes; Longformer deferred pending empirical result

  • What: Tokenize each note into contiguous 512-BPE-token chunks, run Bio_ClinicalBERT over all chunks, and max-pool logits across chunks per label. Escalate to Clinical-Longformer only if chunked-BERT misses the Micro F1 target (≥ 0.70) by more than 3 points.
  • Why: EDA on MIMIC-IV-Note v2.2 discharge summaries (N = 331,793) — truncation-impact analysis in whitespace tokens:
    cap % truncated median tokens lost per truncated note
    512 98.74% 998
    1024 81.78% 621
    2048 21.11% 372
    4096 0.51% 455
    Exceedance rate collapses between 1K→2K (the "knee"); gains from 2K→4K are marginal. Applying ~1.3× BPE inflation for clinical text: BERT's single-window 512-BPE cap ≈ 400 whitespace tokens (near-universal truncation); Longformer's 4K-BPE cap ≈ 3,150 whitespace tokens (~5–8% still truncate). Discharge summaries front-load Assessment/Plan — chunked BERT over 6 × 512-BPE chunks covers ~3,072 BPE per note, slightly more than Longformer's single-pass 4K cap, at 3–5× lower training cost per epoch.
  • Alternatives: (1) Single-window BERT (first 512 tokens) — rejected, throws away the majority of signal. (2) Start directly with Clinical-Longformer — rejected, slower and no evidence the back half of notes carries decisive signal. (3) Hierarchical transformer (chunk encoder + cross-chunk attention) — rejected for scope; revisit if both chunked-BERT and Longformer miss targets.
  • Evidence: reports/eda_report.md §3 (token length). Even Longformer (4096 BPE) truncates the top ~1% of notes — no transformer at current context lengths sees the full tail.

2026-04-21 — Defer date un-shifting to Silver; EDA chart flagged as diagnostic-only

  • What: Do not un-shift MIMIC-IV de-identified dates in the EDA layer. Compute real_year_approx per admission/patient in Silver using anchor_year + midpoint(anchor_year_group) only when a consumer needs real years (temporal split, drift monitoring, fairness-by-era, data card year coverage). Persist as new columns alongside raw shifted values; never overwrite.
  • Why: Separation of concerns — EDA is diagnostic, not a production transform. Date un-shifting is a Silver-level contract so all downstream consumers get consistent semantics. The ICD-coding-from-text baseline doesn't use dates, so implementing the transform preemptively would be premature.
  • Alternatives: (1) Un-shift in EDA so the §6 transition chart shows the Oct-2015 ICD-9→ICD-10 cutoff cleanly — rejected, pushes a production transform into a diagnostic notebook. (2) Leave dates shifted permanently — rejected, prevents drift monitoring and temporal fairness analysis needed for the model card.
  • Evidence / breadcrumbs: reports/eda_report.md §6 action item + TODO in src/mimic_icd_coder/data/clean.py module docstring.

2026-04-23 — TF-IDF + LR baseline ships on the F1 story; P@k floor lowered to informational

  • What: Accept baseline test-split results as the shipped baseline:
    • Micro F1 = 0.617 (floor ≥ 0.55 ; +0.003 vs. Mullenbach 2018 CAML top-50)
    • Macro F1 = 0.584 (floor ≥ 0.35 ; +0.052 vs. Mullenbach)
    • P@5 = 0.526 (original floor ≥ 0.55 missed by 0.024; −0.083 vs. Mullenbach)
    • P@8 = 0.433 (original floor ≥ 0.50 missed by 0.067; Mullenbach does not report a top-50 baseline)
    • val→test drift <0.01 on every metric, which confirms val-tuned thresholds generalize (small val→test gap does NOT by itself prove no leakage). Leakage is prevented architecturally: train/val/test are disjoint by subject_id, verified in tests/test_smoke.py::test_patient_split_disjoint — no admission-level leakage possible.
    • MLflow run ID 4e577699a67a4027bc27628e9b237ac5 (local file store, data/mlruns/).
  • Why: class_weight="balanced" + per-label F1-optimal thresholds inflates rare-label probabilities to maximize per-label F1, which is why Macro F1 clears the Mullenbach baseline by +0.052. The same calibration distorts global probability ranking, which depresses P@k — a well-known trade-off between F1-optimality and ranking. Not a bug; a consequence of the chosen loss/threshold regime. The baseline's job is to prove pipeline correctness and set an F1 floor every transformer must clear; P@k recovery is the transformer branch's job because Mullenbach CAML's 0.609 P@5 itself came from a custom attention architecture, not from calibration.
  • Alternatives: (1) Re-train with class_weight=null to recover P@k — rejected, likely trades Macro F1 below Mullenbach's 0.532 and weakens the headline result. (2) Switch per-label threshold tuning objective from F1 to ranking-aware (NDCG or P@k) — rejected as scope creep; the transformer branch should own ranking calibration end-to-end. (3) Hold the branch until transformer arrives — rejected, the baseline is its own verifiable deliverable.
  • Consequence for exit criteria: P@5 and P@8 floors for this branch are downgraded from "hard gate" to "informational for baseline, primary gate for transformer." F1 floors remain hard gates.
  • Evidence: logs/train_baseline.log, logs/evaluate_test.log, data/gold/baseline_model.joblib, data/gold/baseline_thresholds.npy, MLflow run above. Reproduce with mic train-baseline --config configs/dev.nancy.yml followed by mic evaluate-test --config configs/dev.nancy.yml on the persisted Silver/Gold artifacts.

2026-04-24 — Transformer fine-tune loop validated locally on T1200

Goal: prove the Bio_ClinicalBERT fine-tune loop runs end-to-end on local hardware before paying for Databricks GPU time.

Setup

  • Hardware: NVIDIA T1200 Laptop GPU, 4 GB VRAM, fp16, gradient_checkpointing=False
  • Subset: 1,500 train docs / 200 val docs (seeded, deterministic)
  • Chunking: max_length=512, stride=128 → avg 9.35 chunks/doc → 14,027 train chunks, 1,949 val chunks
  • Effective batch: 16 (per_device=2 × gradient_accumulation=8)
  • Total optimizer steps for 1 epoch: 877
  • Bio_ClinicalBERT (emilyalsentzer/Bio_ClinicalBERT)

Outcome (killed at step 250, 28.5% through epoch — see "Why partial" below)

Step loss grad_norm lr
25 5.708 8.362 5.5e-06
50 4.949 6.723 1.1e-05
75 3.832 4.368 1.7e-05
100 3.225 3.277 2.0e-05
125 2.791 4.130 2.0e-05
150 2.669 2.579 1.9e-05
250 2.502 2.276 1.6e-05

Loss descended monotonically (5.708 → 2.502 = 56% reduction in 225 steps). Grad norms healthy throughout. Learning-rate warmup completed cleanly at step ~88 then began linear decay. MLflow logged every metric in real time.

Why partial run — T1200 throughput observed at 11.89 sec/iteration, projecting ~3.5 hours to complete one epoch + eval on 1.5K subset. Loop validation is the goal of this branch, not eval F1 on a tiny subset; eval metrics on 1,500 train docs would be portfolio-irrelevant. Real F1 numbers will come from feat/transformer-train-prod on Databricks V100 (next branch).

MLflow run

  • Tracking URI: file:data/mlruns
  • Experiment: transformer_debug_local
  • Run ID: 15933746597445318079621407d817ea
  • Loss curve: reports/figures/transformer_debug_loss_curve.png

Known issues filed for follow-up

  • HF Trainer's report_to=["mlflow"] callback flushes metrics only at logging-strategy boundary. Default logging_strategy="epoch" provides zero observable progress until end of epoch. Resolved in this run by enabling DEBUG-level root logger, which transformer.fine_tune already gates against (is_debug_enabled()) to switch to logging_strategy="steps", logging_steps=25.
  • T1200 is unsuitable for full-scale Bio_ClinicalBERT training. Full 97K-document training set would take ~30+ hours. feat/transformer-train-prod will run on Databricks Standard_NC6s_v3 (V100, 16 GB VRAM, ~10× faster).

Conclusion — loop validated end-to-end. Tokenization, chunking, fp16 training, MLflow logging, and TrainingArguments wiring all confirmed working. Ready to scale on Databricks GPU.

2026-04-26 — Reframing Mullenbach 2018 from benchmark to inspiration

Earlier drafts of the README presented numerical deltas between this work's results on MIMIC-IV/ICD-10 top-50 and Mullenbach et al. 2018's results on MIMIC-III/ICD-9 top-50 (Table 5). On review, the comparison is methodologically invalid: different dataset, different coding system, different cohort, different label space. Numerical proximity ("Micro F1 within 0.003") is confounded by all four factors and does not constitute benchmark equivalence.

Decision: drop all numerical comparisons to Mullenbach 2018 from the README, the metric targets, and the MLflow logged metrics. Retain the methodological inheritance (multi-label framing, patient-level splits, P@k as coder-assist-relevant, top-50 label cardinality) and cite the paper as the intellectual antecedent.

A future apples-to-apples reproduction on MIMIC-III/ICD-9 — using Mullenbach's published cohort logic and label set — would make benchmark comparison meaningful. Tracked as future work; out of scope for the current job-search-window deliverable.

Affected:

  • README.md (headline, §1, §6, §14)
  • src/mimic_icd_coder/evaluate.py (compare_to_mullenbach function deprecated)
  • MLflow logged metrics (delta_vs_caml_* removed from baseline runs going forward)

[YYYY-MM-DD] — [Next decision]

  • What:
  • Why:
  • Alternatives: