Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 6 additions & 11 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,17 +215,12 @@ released from this line yet.
information about the prior; before, the infinite value was dropped from the
maximum and the fit went unmentioned.

* `fit$nuts_diag$warmup_check` reports two new per-chain fields. `energy_tau` is
the integrated autocorrelation time of the energy trend residuals, estimated on
the second half of the run; `slope_t_corrected` is the trend t-statistic with
its standard error widened by `sqrt(energy_tau)`. Both are reported only and do
not change when the warmup flag is raised.

The flag's trend criterion uses an ordinary least-squares standard error, which
assumes independent draws. Energy traces are autocorrelated, so that standard
error is too small. These fields let you tell a real transient from an
autocorrelation artifact: a corrected `|t|` well below 2.58 alongside an
`energy_tau` near 3 means the apparent trend is an artifact.
* The warmup advisory now uses an energy trend statistic adjusted for
autocorrelation, reducing warnings caused by ordinary sampling variation.
`fit$nuts_diag$warmup_check$slope_significant_corrected` records this
criterion. The uncorrected `slope_significant` remains available for comparison
and is used if the corrected statistic is unavailable. The E-BFMI and
variance-ratio criteria are unchanged.

* The NUTS issues message now names the criterion that fired. It previously
reported "energy not stationary" even when a low first-half E-BFMI was the
Expand Down
48 changes: 29 additions & 19 deletions R/diagnostics_nuts.R
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@ integrated_act = function(x) {
# Assess whether warmup is complete using energy stationarity.
#
# Splits each chain's energy trace at the midpoint and checks three
# criteria: (1) significant linear trend in energy, (2) first-half
# E-BFMI below 0.3, and (3) first-half / second-half variance ratio
# above 2.0. Any triggered criterion flags the chain.
# criteria: (1) a linear trend in energy whose t-statistic, with the standard
# error adjusted for autocorrelation, exceeds 2.58 in absolute
# value, (2) first-half E-BFMI below 0.3, and (3) first-half / second-half
# variance ratio above 2.0. Any triggered criterion flags the chain.
#
# @param energy_mat Numeric matrix (chains x iterations) of energy
# values from the NUTS sampler.
Expand All @@ -53,23 +54,23 @@ integrated_act = function(x) {
# vector of length nchains:
# - warmup_incomplete: Logical, TRUE if any criterion triggered.
# - energy_slope: Numeric, OLS slope of energy on iteration.
# - slope_significant: Logical, |t| > 2.58 for the slope.
# - slope_significant: Logical, |t| > 2.58 for the slope with the OLS
# standard error, which assumes independent draws. Used as a fallback
# when the corrected statistic is unavailable.
# - slope_significant_corrected: Logical, |slope_t_corrected| > 2.58; this is
# the advisory's trend criterion. If the corrected statistic is not
# finite, it equals slope_significant.
# - ebfmi_first_half: Numeric, E-BFMI for the first half.
# - ebfmi_second_half: Numeric, E-BFMI for the second half.
# - var_ratio: Numeric, var(first_half) / var(second_half).
# - energy_tau: Numeric, integrated autocorrelation time of the trend
# residuals, estimated on the SECOND half. A residual transient lives in
# the first half and would inflate tau there, so the second half gives the
# stationary value.
# - slope_t_corrected: Numeric, the slope t-statistic with its standard error
# widened by sqrt(energy_tau).
# residuals, estimated on the second half. This assumes that the second
# half is sufficiently stable to estimate autocorrelation.
# - slope_t_corrected: Numeric, slope / (OLS standard error * sqrt(energy_tau)).
#
# energy_tau and slope_t_corrected are reported only; they do not enter
# warmup_incomplete. The flag stays deliberately sensitive because it exists to
# send the user to R-hat and ESS, and nothing else in the package does. They let
# a user who investigates separate a real drift from an autocorrelation artifact:
# a corrected |t| well under 2.58 alongside tau around 3 means the trend is an
# artifact of autocorrelation, not residual warmup.
# The OLS t-statistic is inflated by about sqrt(tau) on an autocorrelated
# trace, so the advisory uses the corrected statistic. The OLS statistic is
# retained for comparison and as a fallback.
# ------------------------------------------------------------------------------
check_warmup_complete = function(energy_mat) {
nchains = nrow(energy_mat)
Expand All @@ -80,6 +81,7 @@ check_warmup_complete = function(energy_mat) {
warmup_incomplete = rep(FALSE, nchains),
energy_slope = rep(NA_real_, nchains),
slope_significant = rep(FALSE, nchains),
slope_significant_corrected = rep(FALSE, nchains),
ebfmi_first_half = rep(NA_real_, nchains),
ebfmi_second_half = rep(NA_real_, nchains),
var_ratio = rep(NA_real_, nchains),
Expand All @@ -93,6 +95,7 @@ check_warmup_complete = function(energy_mat) {
warmup_incomplete = FALSE,
energy_slope = NA_real_,
slope_significant = FALSE,
slope_significant_corrected = FALSE,
ebfmi_first_half = NA_real_,
ebfmi_second_half = NA_real_,
var_ratio = NA_real_,
Expand Down Expand Up @@ -135,16 +138,21 @@ check_warmup_complete = function(energy_mat) {
slope_significant = isTRUE(abs(slope / slope_se) > 2.58)
}

# Reported alongside the flag, not part of it: the same statistic with its
# standard error corrected for autocorrelation. tau comes from the second
# half of the residuals, which is stationary even when the first half is not.
# The same statistic with its standard error corrected for autocorrelation.
# Estimate tau from the second half, assuming it is sufficiently stable.
# Use the OLS statistic if the corrected statistic is unavailable.
trend_resid = stats::residuals(trend_lm)
energy_tau = integrated_act(trend_resid[(mid + 1):n_chain])
slope_t_corrected = if(is.finite(energy_tau)) {
unname(slope / (slope_se * sqrt(energy_tau)))
} else {
NA_real_
}
slope_significant_corrected = if(is.finite(slope_t_corrected)) {
abs(slope_t_corrected) > 2.58
} else {
slope_significant
}

# E-BFMI per half
ebfmi_first = mean(diff(first_half)^2) / stats::var(first_half)
Expand All @@ -155,14 +163,15 @@ check_warmup_complete = function(energy_mat) {

# Flag if any criterion triggered. A constant half makes ebfmi_first and
# var_ratio NaN; isTRUE() keeps the flag logical instead of NA.
warmup_incomplete = slope_significant ||
warmup_incomplete = slope_significant_corrected ||
isTRUE(ebfmi_first < 0.3) ||
isTRUE(var_ratio > 2.0)

list(
warmup_incomplete = warmup_incomplete,
energy_slope = slope,
slope_significant = slope_significant,
slope_significant_corrected = slope_significant_corrected,
ebfmi_first_half = ebfmi_first,
ebfmi_second_half = ebfmi_second,
var_ratio = var_ratio,
Expand All @@ -175,6 +184,7 @@ check_warmup_complete = function(energy_mat) {
warmup_incomplete = sapply(results, `[[`, "warmup_incomplete"),
energy_slope = sapply(results, `[[`, "energy_slope"),
slope_significant = sapply(results, `[[`, "slope_significant"),
slope_significant_corrected = sapply(results, `[[`, "slope_significant_corrected"),
ebfmi_first_half = sapply(results, `[[`, "ebfmi_first_half"),
ebfmi_second_half = sapply(results, `[[`, "ebfmi_second_half"),
var_ratio = sapply(results, `[[`, "var_ratio"),
Expand Down
20 changes: 5 additions & 15 deletions tests/testthat/test-warmup-check.R
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,15 @@ test_that("check_warmup_complete returns one value per chain for every field", {

expected = c(
"warmup_incomplete", "energy_slope", "slope_significant",
"ebfmi_first_half", "ebfmi_second_half", "var_ratio",
"energy_tau", "slope_t_corrected"
"slope_significant_corrected", "ebfmi_first_half", "ebfmi_second_half",
"var_ratio", "energy_tau", "slope_t_corrected"
)
expect_true(all(expected %in% names(out)))
for(nm in expected) expect_length(out[[nm]], 3)

expect_type(out$warmup_incomplete, "logical")
expect_type(out$slope_significant, "logical")
expect_type(out$slope_significant_corrected, "logical")
expect_type(out$energy_tau, "double")
expect_type(out$slope_t_corrected, "double")
})
Expand Down Expand Up @@ -69,19 +70,6 @@ test_that("a settling transient inflates the naive t and is caught by the flag",
})


test_that("the reported fields do not change when the flag fires", {
# energy_tau and slope_t_corrected are reported only. Removing the trend must
# not alter warmup_incomplete, which is driven by the naive statistic.
set.seed(5)
energy = t(replicate(4, as.numeric(arima.sim(list(ar = 0.5), n = 1000))))
out = check_warmup_complete(energy)

naive_flag = abs(out$energy_slope) > 0 &
out$slope_significant | out$ebfmi_first_half < 0.3 | out$var_ratio > 2.0
expect_equal(out$warmup_incomplete, unname(naive_flag))
})


test_that("integrated_act recovers a known autocorrelation time", {
set.seed(6)
# tau = (1 + rho) / (1 - rho); rho = 0.8 gives tau = 9.
Expand Down Expand Up @@ -113,6 +101,7 @@ test_that("a constant half leaves the flag logical rather than NA", {
expect_false(anyNA(out$warmup_incomplete))
expect_type(out$slope_significant, "logical")
expect_false(anyNA(out$slope_significant))
expect_false(anyNA(out$slope_significant_corrected))

# The undefined criteria are still reported as the NaN they are; only the
# flag derived from them is forced to TRUE/FALSE.
Expand All @@ -126,4 +115,5 @@ test_that("a constant half leaves the flag logical rather than NA", {
expect_equal(flat$warmup_incomplete, c(FALSE, FALSE))
expect_equal(unname(flat$energy_slope), c(0, 0))
expect_equal(flat$slope_significant, c(FALSE, FALSE))
expect_equal(flat$slope_significant_corrected, c(FALSE, FALSE))
})
32 changes: 10 additions & 22 deletions vignettes/diagnostics.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -215,33 +215,21 @@ The returned list contains the following fields (one value per chain):

- **warmup_incomplete**: A logical flag that is `TRUE` when any of the indicators below suggest the chain may not have reached stationarity.
- **energy_slope**: The slope of a linear regression of energy against iteration number. A slope near zero indicates stable energy; a significant negative slope suggests the chain is still drifting toward higher-probability regions.
- **slope_significant**: `TRUE` when the slope's t-statistic exceeds 2.58 in absolute value. Read this as a screening rule rather than a p-value: the t-statistic uses the ordinary least-squares standard error, which assumes independent draws, while an energy trace is autocorrelated. See "How sensitive is this flag?" below.
- **slope_significant**: `TRUE` when the slope's t-statistic exceeds 2.58 in absolute value with the ordinary least-squares standard error, which assumes independent draws. Reported for comparison and used by the advisory only if the corrected statistic is unavailable.
- **ebfmi_first_half** and **ebfmi_second_half**: E-BFMI computed separately for the first and second halves of the post-warmup samples. If the first-half value is much lower (for example, below 0.3) while the second-half value is healthy, the early samples were likely still settling.
- **var_ratio**: The ratio of energy variance in the first half to that in the second half. A ratio much greater than 1 (for example, above 2) indicates higher variability early on, consistent with transient behavior.
- **energy_tau**: The integrated autocorrelation time of the trend residuals, estimated on the second half of the run. Typical values for a healthy chain are around 3. A much larger value usually means a transient is still present, since a settling chain leaves strongly autocorrelated residuals behind.
- **slope_t_corrected**: The same t-statistic as `slope_significant` uses, but with its standard error widened by `sqrt(energy_tau)` to account for autocorrelation. This is reported for interpretation only and does not affect the flag.
- **energy_tau**: The integrated autocorrelation time of the trend residuals, estimated on the second half of the run. A large value indicates strong autocorrelation; it does not by itself show that warmup was incomplete.
- **slope_t_corrected**: The energy trend t-statistic with a standard error adjusted for autocorrelation.
- **slope_significant_corrected**: `TRUE` when `slope_t_corrected` exceeds 2.58 in absolute value. The advisory uses this criterion. If the corrected statistic is unavailable, it uses `slope_significant` instead.

## How sensitive is this flag?

The trend criterion compares energy against iteration using an ordinary
least-squares standard error, which assumes independent draws. Energy traces are
autocorrelated, so that standard error is too small and the t-statistic too
large. Across `bgm()` and `bgmCompare()` fits to this package's three example
datasets, the integrated autocorrelation time of the energy trace is around 3,
which understates the standard error by a factor of roughly 1.8. The `|t| > 2.58`
rule therefore behaves like a 15% per-chain screening rule rather than a 1% test,
and because the flag is raised when *any* chain trips it, a well-behaved
four-chain fit raises it a substantial fraction of the time.

This is deliberate. The flag's purpose is to send you to R-hat and ESS, which are
the diagnostics that decide whether a fit is usable, and nothing else in the
package prompts you to look. A flag that occasionally sends you to check a
healthy fit is less costly than one that stays silent on an unhealthy one.

`slope_t_corrected` and `energy_tau` are what to read once you are looking. A
corrected `|t|` comfortably below 2.58 alongside an `energy_tau` near 3 means the
apparent trend is an artifact of autocorrelation rather than residual warmup. A
large corrected `|t|`, or an `energy_tau` far above 3, points to a real transient.
Successive energy draws are correlated. The trend check adjusts its standard
error for this dependence, reducing warnings caused by ordinary sampling
variation. If the correction is unavailable, it uses the uncorrected statistic.
The E-BFMI and variance-ratio checks are unchanged.

Check R-hat and ESS alongside the other sampling diagnostics.

## What to do about it

Expand Down
Loading