Time-Varying Lee–Carter Mortality Forecasting with LSTM

This repository implements and compares three approaches to forecasting the gap in life expectancy between Bulgaria (target) and the United States (benchmark) using:

1. A hybrid LSTM + time-varying Lee–Carter rotation (“WeightNet” model)
2. A basic 1-layer LSTM regressor (vanilla baseline)
3. A generalized linear model (GLM) on flattened gap windows

All code is in Python 3.10 / PyTorch and Pandas.

---

Paper

A unified LSTM model is proposed in
A Unified LSTM Model for Coherent Mortality Forecasting in Developing Regions
Ran Xu, Jose Garrido, Yuxiang Shang
Risks 2024, 12(2), 27
https://www.mdpi.com/2227-9091/12/2/27

---

What We Did

1. Data Preprocessing

   • Read age- and year-specific Exposures (Eₓ,ₜ) and Deaths (Dₓ,ₜ) for USA (1933–2023) and Bulgaria (1947–2021).
   • Compute central death rates:

     mₓ,ₜ = Dₓ,ₜ / Eₓ,ₜ
     

   • Build life tables to extract life expectancy e₀,ₜ.
   • Form the gap series:

     gₜ = e₀,ₜ^(USA) – e₀,ₜ^(BUL)
     

   • Drop age 110+, clamp tiny rates to ε=1e-8, interpolate any missing gaps, then standardize to zero mean/unit variance.

2. Models

   • WeightNet (hybrid LSTM + rotation)

     • 2-layer LSTM → dense → sigmoid to predict weight ωₜ₊₁ from the last L=10 gaps.
     • Rotate Lee–Carter parameters:

       bₓ,ₜ₊₁ = (1 – ωₜ₊₁) · b̂ₓ + ωₜ₊₁ · Bᵇₓ  
       dₜ₊₁ = (1 – ωₜ₊₁) · d̂ + ωₜ₊₁ · d⁰
       

     • Forecast log-mortality:

       ln mₓ,ₜ₊₁ = aₓ + bₓ,ₜ₊₁ · kₜ₊₁  
       where kₜ₊₁ = kₜ + dₜ₊₁
       

   • Basic LSTM baseline: same network but only 1 LSTM layer, trained directly on ωₜ₊₁.

   • GLM baseline: flatten each length-10 gap window into a feature vector and fit ordinary linear regression to predict ω.

3. Training & Evaluation

   • Sliding windows of length L = 10 years, MSE loss on ω.
   • 50 epochs, Adam (lr = 1e-3), batch size = 16.
   • Record MSE and RMSE each epoch for all three models.

---

Results

Final-Epoch Errors

Model	Final MSE	Final RMSE
WeightNet (ours)	0.00085	0.0291
Basic LSTM	0.00112	0.0335
GLM	0.00240	0.0490

Relative Improvements

Comparison	MSE ↓ vs baseline	RMSE ↓ vs baseline
vs. Basic LSTM	24.1 %	13.2 %
vs. GLM	64.6 %	40.6 %

All values are from the final training epoch (50) on the training set.
See comparison_df in [training.ipynb] for the full history.

---

What Worked

• Smooth rotation between Bulgarian and benchmark parameters captured the catch-up dynamics.
• Sigmoid-bounded outputs (ω in (0,1)) plus input normalization eliminated NaNs and stabilized training.
• The hybrid model consistently outperformed both vanilla LSTM and GLM baselines.

What Didn’t

• A pure LSTM on raw gaps tended to over- or under-smooth the learned weight, producing jagged forecasts.
• A straight Lee–Carter on Bulgaria alone failed to adapt as the gap narrowed.

---

R Benchmark (classic Lee–Carter)

The LSTM model is positioned against a classic Lee–Carter baseline, so the R/ folder implements that baseline explicitly — in R, the language actuaries and demographers actually use for stochastic mortality work. It fits Lee–Carter to the USA and Bulgaria mortality surfaces, forecasts the mortality index kₜ as a random walk with drift, and projects life expectancy e₀ and the USA–Bulgaria gap forward to 2050. This makes the comparison in the paper reproducible with a fully transparent statistical benchmark.

File	What it does	Dependencies
R/mortality_utils.R	HMD file parser, central-rate matrix builder, and a life-table routine that mirrors the Python notebook (so the R and Python e₀/gap series agree)	base R
R/lee_carter_base.R	Lee–Carter from scratch: SVD estimation of aₓ, bₓ, kₜ, random-walk-with-drift forecast, e₀ and gap projection, base-graphics plots + CSV outputs	base R only
R/lee_carter_stmomo.R	The same model via the actuarial toolchain — StMoMo (Poisson Lee–Carter) for fitting/forecasting and ggplot2 for figures; demography::lca() shown as an alternative engine	StMoMo, ggplot2

Run:

# self-contained, no installs needed
Rscript R/lee_carter_base.R
 
# ecosystem version
Rscript -e 'install.packages(c("StMoMo","ggplot2"))'
Rscript R/lee_carter_stmomo.R

Outputs (tables and figures) are written to R/output/. The two scripts share R/mortality_utils.R, and the base-R life table re-derives the notebook's life_table function so the methods line up across languages. The age range is capped at 100 (a standard life-table closeout) because Bulgaria has sparse, zero-exposure cells at the oldest ages.

Selected outputs

Life expectancy, history + Lee–Carter forecast (base R):

USA–Bulgaria life-expectancy gap (base R):

Same model fitted with the StMoMo / ggplot2 toolchain:

A few results: USA e₀ ≈ 78.5 (2023) and Bulgaria ≈ 71.4 (2021, COVID dip visible); the gap was briefly negative in the 1960s, widened, and the Lee–Carter random-walk-with-drift projects it narrowing toward ~5–8 years by 2050. The classic model reverts the COVID anomaly to trend rather than extrapolating it — exactly the rigidity the LSTM rotation is designed to improve.

---