diff --git a/NEWS.md b/NEWS.md index bd5f0312..60c24801 100644 --- a/NEWS.md +++ b/NEWS.md @@ -215,17 +215,12 @@ released from this line yet. information about the prior; before, the infinite value was dropped from the maximum and the fit went unmentioned. -* `fit$nuts_diag$warmup_check` reports two new per-chain fields. `energy_tau` is - the integrated autocorrelation time of the energy trend residuals, estimated on - the second half of the run; `slope_t_corrected` is the trend t-statistic with - its standard error widened by `sqrt(energy_tau)`. Both are reported only and do - not change when the warmup flag is raised. - - The flag's trend criterion uses an ordinary least-squares standard error, which - assumes independent draws. Energy traces are autocorrelated, so that standard - error is too small. These fields let you tell a real transient from an - autocorrelation artifact: a corrected `|t|` well below 2.58 alongside an - `energy_tau` near 3 means the apparent trend is an artifact. +* The warmup advisory now uses an energy trend statistic adjusted for + autocorrelation, reducing warnings caused by ordinary sampling variation. + `fit$nuts_diag$warmup_check$slope_significant_corrected` records this + criterion. The uncorrected `slope_significant` remains available for comparison + and is used if the corrected statistic is unavailable. The E-BFMI and + variance-ratio criteria are unchanged. * The NUTS issues message now names the criterion that fired. It previously reported "energy not stationary" even when a low first-half E-BFMI was the diff --git a/R/diagnostics_nuts.R b/R/diagnostics_nuts.R index bf879618..8683c49b 100644 --- a/R/diagnostics_nuts.R +++ b/R/diagnostics_nuts.R @@ -42,9 +42,10 @@ integrated_act = function(x) { # Assess whether warmup is complete using energy stationarity. # # Splits each chain's energy trace at the midpoint and checks three -# criteria: (1) significant linear trend in energy, (2) first-half -# E-BFMI below 0.3, and (3) first-half / second-half variance ratio -# above 2.0. Any triggered criterion flags the chain. +# criteria: (1) a linear trend in energy whose t-statistic, with the standard +# error adjusted for autocorrelation, exceeds 2.58 in absolute +# value, (2) first-half E-BFMI below 0.3, and (3) first-half / second-half +# variance ratio above 2.0. Any triggered criterion flags the chain. # # @param energy_mat Numeric matrix (chains x iterations) of energy # values from the NUTS sampler. @@ -53,23 +54,23 @@ integrated_act = function(x) { # vector of length nchains: # - warmup_incomplete: Logical, TRUE if any criterion triggered. # - energy_slope: Numeric, OLS slope of energy on iteration. -# - slope_significant: Logical, |t| > 2.58 for the slope. +# - slope_significant: Logical, |t| > 2.58 for the slope with the OLS +# standard error, which assumes independent draws. Used as a fallback +# when the corrected statistic is unavailable. +# - slope_significant_corrected: Logical, |slope_t_corrected| > 2.58; this is +# the advisory's trend criterion. If the corrected statistic is not +# finite, it equals slope_significant. # - ebfmi_first_half: Numeric, E-BFMI for the first half. # - ebfmi_second_half: Numeric, E-BFMI for the second half. # - var_ratio: Numeric, var(first_half) / var(second_half). # - energy_tau: Numeric, integrated autocorrelation time of the trend -# residuals, estimated on the SECOND half. A residual transient lives in -# the first half and would inflate tau there, so the second half gives the -# stationary value. -# - slope_t_corrected: Numeric, the slope t-statistic with its standard error -# widened by sqrt(energy_tau). +# residuals, estimated on the second half. This assumes that the second +# half is sufficiently stable to estimate autocorrelation. +# - slope_t_corrected: Numeric, slope / (OLS standard error * sqrt(energy_tau)). # -# energy_tau and slope_t_corrected are reported only; they do not enter -# warmup_incomplete. The flag stays deliberately sensitive because it exists to -# send the user to R-hat and ESS, and nothing else in the package does. They let -# a user who investigates separate a real drift from an autocorrelation artifact: -# a corrected |t| well under 2.58 alongside tau around 3 means the trend is an -# artifact of autocorrelation, not residual warmup. +# The OLS t-statistic is inflated by about sqrt(tau) on an autocorrelated +# trace, so the advisory uses the corrected statistic. The OLS statistic is +# retained for comparison and as a fallback. # ------------------------------------------------------------------------------ check_warmup_complete = function(energy_mat) { nchains = nrow(energy_mat) @@ -80,6 +81,7 @@ check_warmup_complete = function(energy_mat) { warmup_incomplete = rep(FALSE, nchains), energy_slope = rep(NA_real_, nchains), slope_significant = rep(FALSE, nchains), + slope_significant_corrected = rep(FALSE, nchains), ebfmi_first_half = rep(NA_real_, nchains), ebfmi_second_half = rep(NA_real_, nchains), var_ratio = rep(NA_real_, nchains), @@ -93,6 +95,7 @@ check_warmup_complete = function(energy_mat) { warmup_incomplete = FALSE, energy_slope = NA_real_, slope_significant = FALSE, + slope_significant_corrected = FALSE, ebfmi_first_half = NA_real_, ebfmi_second_half = NA_real_, var_ratio = NA_real_, @@ -135,9 +138,9 @@ check_warmup_complete = function(energy_mat) { slope_significant = isTRUE(abs(slope / slope_se) > 2.58) } - # Reported alongside the flag, not part of it: the same statistic with its - # standard error corrected for autocorrelation. tau comes from the second - # half of the residuals, which is stationary even when the first half is not. + # The same statistic with its standard error corrected for autocorrelation. + # Estimate tau from the second half, assuming it is sufficiently stable. + # Use the OLS statistic if the corrected statistic is unavailable. trend_resid = stats::residuals(trend_lm) energy_tau = integrated_act(trend_resid[(mid + 1):n_chain]) slope_t_corrected = if(is.finite(energy_tau)) { @@ -145,6 +148,11 @@ check_warmup_complete = function(energy_mat) { } else { NA_real_ } + slope_significant_corrected = if(is.finite(slope_t_corrected)) { + abs(slope_t_corrected) > 2.58 + } else { + slope_significant + } # E-BFMI per half ebfmi_first = mean(diff(first_half)^2) / stats::var(first_half) @@ -155,7 +163,7 @@ check_warmup_complete = function(energy_mat) { # Flag if any criterion triggered. A constant half makes ebfmi_first and # var_ratio NaN; isTRUE() keeps the flag logical instead of NA. - warmup_incomplete = slope_significant || + warmup_incomplete = slope_significant_corrected || isTRUE(ebfmi_first < 0.3) || isTRUE(var_ratio > 2.0) @@ -163,6 +171,7 @@ check_warmup_complete = function(energy_mat) { warmup_incomplete = warmup_incomplete, energy_slope = slope, slope_significant = slope_significant, + slope_significant_corrected = slope_significant_corrected, ebfmi_first_half = ebfmi_first, ebfmi_second_half = ebfmi_second, var_ratio = var_ratio, @@ -175,6 +184,7 @@ check_warmup_complete = function(energy_mat) { warmup_incomplete = sapply(results, `[[`, "warmup_incomplete"), energy_slope = sapply(results, `[[`, "energy_slope"), slope_significant = sapply(results, `[[`, "slope_significant"), + slope_significant_corrected = sapply(results, `[[`, "slope_significant_corrected"), ebfmi_first_half = sapply(results, `[[`, "ebfmi_first_half"), ebfmi_second_half = sapply(results, `[[`, "ebfmi_second_half"), var_ratio = sapply(results, `[[`, "var_ratio"), diff --git a/tests/testthat/test-warmup-check.R b/tests/testthat/test-warmup-check.R index d560aeeb..e0590e00 100644 --- a/tests/testthat/test-warmup-check.R +++ b/tests/testthat/test-warmup-check.R @@ -11,14 +11,15 @@ test_that("check_warmup_complete returns one value per chain for every field", { expected = c( "warmup_incomplete", "energy_slope", "slope_significant", - "ebfmi_first_half", "ebfmi_second_half", "var_ratio", - "energy_tau", "slope_t_corrected" + "slope_significant_corrected", "ebfmi_first_half", "ebfmi_second_half", + "var_ratio", "energy_tau", "slope_t_corrected" ) expect_true(all(expected %in% names(out))) for(nm in expected) expect_length(out[[nm]], 3) expect_type(out$warmup_incomplete, "logical") expect_type(out$slope_significant, "logical") + expect_type(out$slope_significant_corrected, "logical") expect_type(out$energy_tau, "double") expect_type(out$slope_t_corrected, "double") }) @@ -69,19 +70,6 @@ test_that("a settling transient inflates the naive t and is caught by the flag", }) -test_that("the reported fields do not change when the flag fires", { - # energy_tau and slope_t_corrected are reported only. Removing the trend must - # not alter warmup_incomplete, which is driven by the naive statistic. - set.seed(5) - energy = t(replicate(4, as.numeric(arima.sim(list(ar = 0.5), n = 1000)))) - out = check_warmup_complete(energy) - - naive_flag = abs(out$energy_slope) > 0 & - out$slope_significant | out$ebfmi_first_half < 0.3 | out$var_ratio > 2.0 - expect_equal(out$warmup_incomplete, unname(naive_flag)) -}) - - test_that("integrated_act recovers a known autocorrelation time", { set.seed(6) # tau = (1 + rho) / (1 - rho); rho = 0.8 gives tau = 9. @@ -113,6 +101,7 @@ test_that("a constant half leaves the flag logical rather than NA", { expect_false(anyNA(out$warmup_incomplete)) expect_type(out$slope_significant, "logical") expect_false(anyNA(out$slope_significant)) + expect_false(anyNA(out$slope_significant_corrected)) # The undefined criteria are still reported as the NaN they are; only the # flag derived from them is forced to TRUE/FALSE. @@ -126,4 +115,5 @@ test_that("a constant half leaves the flag logical rather than NA", { expect_equal(flat$warmup_incomplete, c(FALSE, FALSE)) expect_equal(unname(flat$energy_slope), c(0, 0)) expect_equal(flat$slope_significant, c(FALSE, FALSE)) + expect_equal(flat$slope_significant_corrected, c(FALSE, FALSE)) }) diff --git a/vignettes/diagnostics.Rmd b/vignettes/diagnostics.Rmd index ba12b789..63eecc1f 100644 --- a/vignettes/diagnostics.Rmd +++ b/vignettes/diagnostics.Rmd @@ -215,33 +215,21 @@ The returned list contains the following fields (one value per chain): - **warmup_incomplete**: A logical flag that is `TRUE` when any of the indicators below suggest the chain may not have reached stationarity. - **energy_slope**: The slope of a linear regression of energy against iteration number. A slope near zero indicates stable energy; a significant negative slope suggests the chain is still drifting toward higher-probability regions. -- **slope_significant**: `TRUE` when the slope's t-statistic exceeds 2.58 in absolute value. Read this as a screening rule rather than a p-value: the t-statistic uses the ordinary least-squares standard error, which assumes independent draws, while an energy trace is autocorrelated. See "How sensitive is this flag?" below. +- **slope_significant**: `TRUE` when the slope's t-statistic exceeds 2.58 in absolute value with the ordinary least-squares standard error, which assumes independent draws. Reported for comparison and used by the advisory only if the corrected statistic is unavailable. - **ebfmi_first_half** and **ebfmi_second_half**: E-BFMI computed separately for the first and second halves of the post-warmup samples. If the first-half value is much lower (for example, below 0.3) while the second-half value is healthy, the early samples were likely still settling. - **var_ratio**: The ratio of energy variance in the first half to that in the second half. A ratio much greater than 1 (for example, above 2) indicates higher variability early on, consistent with transient behavior. -- **energy_tau**: The integrated autocorrelation time of the trend residuals, estimated on the second half of the run. Typical values for a healthy chain are around 3. A much larger value usually means a transient is still present, since a settling chain leaves strongly autocorrelated residuals behind. -- **slope_t_corrected**: The same t-statistic as `slope_significant` uses, but with its standard error widened by `sqrt(energy_tau)` to account for autocorrelation. This is reported for interpretation only and does not affect the flag. +- **energy_tau**: The integrated autocorrelation time of the trend residuals, estimated on the second half of the run. A large value indicates strong autocorrelation; it does not by itself show that warmup was incomplete. +- **slope_t_corrected**: The energy trend t-statistic with a standard error adjusted for autocorrelation. +- **slope_significant_corrected**: `TRUE` when `slope_t_corrected` exceeds 2.58 in absolute value. The advisory uses this criterion. If the corrected statistic is unavailable, it uses `slope_significant` instead. ## How sensitive is this flag? -The trend criterion compares energy against iteration using an ordinary -least-squares standard error, which assumes independent draws. Energy traces are -autocorrelated, so that standard error is too small and the t-statistic too -large. Across `bgm()` and `bgmCompare()` fits to this package's three example -datasets, the integrated autocorrelation time of the energy trace is around 3, -which understates the standard error by a factor of roughly 1.8. The `|t| > 2.58` -rule therefore behaves like a 15% per-chain screening rule rather than a 1% test, -and because the flag is raised when *any* chain trips it, a well-behaved -four-chain fit raises it a substantial fraction of the time. - -This is deliberate. The flag's purpose is to send you to R-hat and ESS, which are -the diagnostics that decide whether a fit is usable, and nothing else in the -package prompts you to look. A flag that occasionally sends you to check a -healthy fit is less costly than one that stays silent on an unhealthy one. - -`slope_t_corrected` and `energy_tau` are what to read once you are looking. A -corrected `|t|` comfortably below 2.58 alongside an `energy_tau` near 3 means the -apparent trend is an artifact of autocorrelation rather than residual warmup. A -large corrected `|t|`, or an `energy_tau` far above 3, points to a real transient. +Successive energy draws are correlated. The trend check adjusts its standard +error for this dependence, reducing warnings caused by ordinary sampling +variation. If the correction is unavailable, it uses the uncorrected statistic. +The E-BFMI and variance-ratio checks are unchanged. + +Check R-hat and ESS alongside the other sampling diagnostics. ## What to do about it