From a2dd9f31b16bf21fd2cee5fc930e12f68bca9820 Mon Sep 17 00:00:00 2001 From: Maarten Marsman Date: Fri, 6 Nov 2026 23:56:06 +0100 Subject: [PATCH 1/3] fix(sensitivity): count verdicts()' flag as the check's fourth count; one saturation rule for the summary cells and the check prior_sensitivity_check() takes its chosen-scale flag from verdicts() on the original fit, so the two reports count the same edges. The between-chain band and the per-chain verdict test are gone, as are the $edges columns insufficient, insufficient_noisy and insufficient_disagree; the flag is the fragile column. The printed count labels are "verdict held at every scale", "changed within Monte Carlo error", "changed beyond Monte Carlo error" and "could change on a rerun"; the sentence under them names the flagged edges and states that their log Bayes factors lie within two standard errors of an evidence threshold. The split-R-hat kernel applies the ESS kernel's variance floor: a sub-chain with biased variance below 1e-15 counts as constant, so a Rao-Blackwellized inclusion series constant to numerical precision has NA for mcse, n_eff and Rhat together. The check's saturated column is the same criterion (the chosen-scale MCSE is NA). The note under the summary tables and in the check says "0 or 1 in every draw of every chain, or constant to numerical precision"; the Details of summary.bgms, summary.bgmCompare and the diagnostics vignette explain why the series is constant. The sentence for a fit with an infinite split-R-hat says the sampler did not move and that the fit's verdicts do not describe the posterior distribution at that scale; the help page states that rhat_max covers the continuous parameters only and what such a refit needs. The Refits line prints NUTS. ?verdicts states that an indicator whose standard errors are undefined is not flagged. The vignette's saved check object is regenerated. --- NEWS.md | 32 ++++ R/methods_bgmcompare.R | 29 +++- R/methods_bgms.R | 27 +++- R/prior_sensitivity.R | 187 +++++++++++------------ R/refit_engine.R | 28 ++-- R/verdicts.R | 3 +- man/print.bgms_prior_sensitivity.Rd | 11 +- man/prior_sensitivity_check.Rd | 50 +++--- man/summary.bgmCompare.Rd | 25 +++ man/summary.bgms.Rd | 24 +++ man/verdicts.Rd | 3 +- src/mcmc_diagnostics.cpp | 30 ++-- tests/testthat/_snaps/summary-marking.md | 4 +- tests/testthat/test-mcmc-diagnostics.R | 16 ++ tests/testthat/test-prior-sensitivity.R | 95 +++++++----- vignettes/diagnostics.Rmd | 31 ++-- vignettes/prior-sensitivity-ps.rds | Bin 2075 -> 2053 bytes vignettes/prior-sensitivity.Rmd | 20 +-- 18 files changed, 392 insertions(+), 223 deletions(-) diff --git a/NEWS.md b/NEWS.md index 3b43d3fd..ec99ce37 100644 --- a/NEWS.md +++ b/NEWS.md @@ -165,6 +165,38 @@ released from this line yet. ## Diagnostics +* The fourth count of the printed `prior_sensitivity_check()` report, "could + change on a rerun", is `verdicts()`' near-threshold flag on the original fit, + computed by the same function, so the two reports count the same edges. The + between-chain band and the per-chain verdict test that the check applied on + top of it are gone, as are the `$edges` columns `insufficient`, + `insufficient_noisy` and `insufficient_disagree`; the flag is the `fragile` + column. The other three count labels lose their commas. The sentence under + the counts names the flagged edges and states that their log Bayes factors + lie within two standard errors of an evidence threshold. + +* The split-R-hat kernel applies the ESS kernel's variance floor: a sub-chain + whose variance is below 1e-15 counts as constant, so a Rao-Blackwellized + inclusion series that is constant to numerical precision has `NA` for + `mcse`, `n_eff` and `Rhat` together, where before `Rhat` was computed from + the residual variation. The `saturated` column of `prior_sensitivity_check()` + follows the same criterion, so it marks exactly the edges whose chosen-scale + Monte Carlo standard error is `NA`. The one-sentence note under the summary + tables and in the check now says "0 or 1 in every draw of every chain, or + constant to numerical precision", and the Details of `?summary.bgms`, + `?summary.bgmCompare` and `vignette("diagnostics")` explain why: the + recorded inclusion probability is one minus the acceptance probability of + removing the edge, which rounds to 1 once that probability is below about + `1e-16`, or varies below the kernels' floor when it is around `1e-12`. + +* The sentence `prior_sensitivity_check()` prints for a fit with an infinite + split-R-hat says that the sampler did not move, in which fit, and that the + fit's verdicts do not describe the posterior distribution at that scale. The + `$grid` documentation states that `rhat_max` covers the continuous + parameters only, and the Details say what such a refit needs. The `Refits:` + line prints NUTS in capitals. `?verdicts` states that an indicator whose + standard errors are undefined is not flagged. + * The pairwise summary table for a `bgm()` fit and the difference summary tables for a `bgmCompare()` fit no longer include `share_incl`, the proportion of Monte Carlo variance attributed to uncertainty in inclusion. The reported diff --git a/R/methods_bgmcompare.R b/R/methods_bgmcompare.R index 3c4d3313..a1b84e4b 100644 --- a/R/methods_bgmcompare.R +++ b/R/methods_bgmcompare.R @@ -239,6 +239,31 @@ print.bgmCompare = function(x, ...) { #' are exempt from the union recode and are never marked). Fits made before #' that field existed print unmarked. #' +#' The inclusion table (`$indicator`) reports, per difference indicator, the Rao-Blackwellized +#' inclusion probability with its Monte Carlo standard error, effective sample +#' size and split-R-hat. At every update of a difference indicator the sampler +#' proposes to flip it and records the probability that the difference is in the +#' model after the move: the acceptance probability \eqn{\alpha} of adding it +#' when it is out, \eqn{1 - \alpha} of removing it when it is in. The +#' inclusion probability is the average of those recorded values, and `mcse`, +#' `n_eff` and `Rhat` are computed from their series. +#' +#' Those three cells are `NA`, and print blank, when the series has no +#' variation the Monte Carlo machinery can work on. For a difference the data +#' support overwhelmingly, the probability of accepting its removal is below +#' about `1e-16` at every update, so \eqn{1 - \alpha} rounds to exactly 1 in +#' double precision and the series is constant at 1 in every draw of every +#' chain; a difference the data rule out as strongly gives a series constant at 0. +#' A difference whose removal probability is around `1e-12` gives a series that +#' varies by that much, below the variance floor of `1e-15` at which the +#' autocorrelation kernels stop estimating. In both cases there is nothing to +#' estimate: the inclusion probability is at 1 or 0 to numerical precision, +#' its Monte Carlo error is smaller than any printed digit, and the verdict +#' lies beyond any evidence threshold. A difference only slightly less certain, with +#' a removal probability of `1e-8` say, has a varying series and shows numbers. +#' Main-effect difference rows are `NA` throughout when +#' `main_difference_selection = FALSE`: their indicators were never sampled. +#' #' @param object An object of class `bgmCompare`. #' @param ... Currently ignored. #' @@ -352,8 +377,8 @@ print.summary.bgmCompare = function(x, digits = 3, ...) { } if(ind_has_na) { cat("Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion\n") - cat("probability was 0 or 1 in every draw; there is no variation to estimate them\n") - cat("from.\n") + cat("probability was 0 or 1 in every draw of every chain, or constant to numerical\n") + cat("precision; there is no variation to estimate them from.\n") if(isFALSE(x$arguments$main_difference_selection)) { cat("Main-effect difference rows are blank when main_difference_selection = FALSE\n") cat("left their indicators unsampled.\n") diff --git a/R/methods_bgms.R b/R/methods_bgms.R index 9d4bc681..c39a712a 100644 --- a/R/methods_bgms.R +++ b/R/methods_bgms.R @@ -81,6 +81,30 @@ print.bgms = function(x, ...) { #' #' @description Returns posterior summaries and diagnostics for a fitted `bgms` model. #' +#' @details +#' The inclusion table (`$indicator`) reports, per edge, the Rao-Blackwellized +#' inclusion probability with its Monte Carlo standard error, effective sample +#' size and split-R-hat. At every update of an edge indicator the sampler +#' proposes to flip it and records the probability that the edge is in the +#' model after the move: the acceptance probability \eqn{\alpha} of adding it +#' when it is out, \eqn{1 - \alpha} of removing it when it is in. The +#' inclusion probability is the average of those recorded values, and `mcse`, +#' `n_eff` and `Rhat` are computed from their series. +#' +#' Those three cells are `NA`, and print blank, when the series has no +#' variation the Monte Carlo machinery can work on. For an edge the data +#' support overwhelmingly, the probability of accepting its removal is below +#' about `1e-16` at every update, so \eqn{1 - \alpha} rounds to exactly 1 in +#' double precision and the series is constant at 1 in every draw of every +#' chain; an edge the data rule out as strongly gives a series constant at 0. +#' An edge whose removal probability is around `1e-12` gives a series that +#' varies by that much, below the variance floor of `1e-15` at which the +#' autocorrelation kernels stop estimating. In both cases there is nothing to +#' estimate: the inclusion probability is at 1 or 0 to numerical precision, +#' its Monte Carlo error is smaller than any printed digit, and the verdict +#' lies beyond any evidence threshold. An edge only slightly less certain, with +#' a removal probability of `1e-8` say, has a varying series and shows numbers. +#' #' @param object An object of class `bgms`. #' @param ... Currently ignored. #' @@ -187,7 +211,8 @@ print.summary.bgms = function(x, digits = 3, ...) { if(nrow(x$indicator) > .summary_preview_rows) cat("... (use `summary(fit)$indicator` to see full output)\n") if(ind_has_na) { cat("Note: blank mcse, n_eff and Rhat cells mark edges whose inclusion probability\n") - cat("was 0 or 1 in every draw; there is no variation to estimate them from.\n") + cat("was 0 or 1 in every draw of every chain, or constant to numerical precision;\n") + cat("there is no variation to estimate them from.\n") } cat("\n") } diff --git a/R/prior_sensitivity.R b/R/prior_sensitivity.R index b16e3396..77f98a8a 100644 --- a/R/prior_sensitivity.R +++ b/R/prior_sensitivity.R @@ -68,22 +68,27 @@ verdict_from_bf = function(bf, threshold) { #' smallest and largest multiplier in the contiguous run of fits around #' \code{1x} that hold the reference verdict. #' -#' \strong{Sufficiency at the chosen scale.} An edge whose per-chain verdicts -#' disagree in the original fit, or whose log Bayes factor lies within -#' \code{max(between-chain band, 2 * MCSE)} of a verdict threshold there, is -#' marked \code{insufficient}: the fit cannot certify the verdict the check -#' starts from, and the printed report lists the edge as not certifiable -#' rather than in a mover category. +#' \strong{The chosen-scale flag.} An edge is \code{fragile} when +#' \code{\link{verdicts}()} flags it in the original fit: its log Bayes factor +#' lies within two standard errors of a verdict threshold, by either of the two +#' standard errors \code{verdicts()} computes. The verdict the check starts +#' from could then change on a rerun with another seed, so the printed report +#' counts the edge among the verdicts that could change on a rerun rather than +#' in a mover category. An edge whose standard errors are undefined is not +#' flagged, as in \code{verdicts()}. #' #' \strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the -#' continuous parameters, smallest effective sample size over the continuous -#' parameters and over the indicators, and for NUTS the smallest E-BFMI and -#' the largest energy variance ratio, are in \code{$grid}. The classification -#' uses every fit; the printed report names a fit whose largest -#' split-\eqn{\hat R} exceeds 1.01, and states when a fit did not sample: a -#' parameter constant in every chain at values that differ across chains has -#' an infinite split-\eqn{\hat R}, and that fit's verdicts say nothing about -#' the prior. +#' continuous parameters (the main-effect and pairwise blocks of the fit +#' summary, not the indicators), smallest effective sample size over the +#' continuous parameters and over the indicators, and for NUTS the smallest +#' E-BFMI and the largest energy variance ratio, are in \code{$grid}. The +#' classification uses every fit; the printed report names a fit whose largest +#' split-\eqn{\hat R} exceeds 1.01, and states when a fit's sampler did not +#' move: a parameter that kept a single value throughout every chain, at values +#' that differ between chains, has an infinite split-\eqn{\hat R}, and the +#' verdicts of that fit do not describe the posterior distribution at that +#' scale. Such a refit needs a longer warmup or a cold start; an ordinal refit +#' starts from the original fit's final state, so \code{warmup} is the lever. #' #' \strong{Models with a precision diagonal.} A continuous or mixed fit has a #' Gamma or exponential prior on the precision diagonal with a standardized @@ -155,16 +160,21 @@ verdict_from_bf = function(bf, threshold) { #' \code{stability_upper}; \code{mover} (\code{stable}, #' \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or #' \code{NA} for an indicator without a verdict) and \code{moved_at}; -#' \code{insufficient} with its two subcauses \code{insufficient_noisy} -#' (log Bayes factor within Monte Carlo error of a threshold) and -#' \code{insufficient_disagree} (chains disagree on the verdict); -#' \code{saturated} (the original fit's inclusion probability is 0 or 1 in -#' every draw of every chain); +#' \code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the +#' original fit); \code{saturated} (the original fit's Rao-Blackwellized +#' inclusion draws have no variation the ESS kernel can use, because the +#' acceptance probability of flipping the indicator was below double +#' precision at every update: the inclusion probability is 0 or 1 in every +#' draw of every chain, or constant to numerical precision; +#' \code{chosen_scale_mcse} is \code{NA} there; see the Details of +#' \code{\link{summary.bgms}}); #' then one \code{verdict_x}, \code{log_bf_x} and \code{mcse_x} #' column per multiplier \code{m}, from that fit's own statistics.} #' \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale}, #' \code{original_fit}, \code{iter}, \code{warmup}, \code{seed}, the -#' diagnostics \code{rhat_max}, \code{ess_min}, \code{inclusion_ess_min}, +#' diagnostics \code{rhat_max} (largest split-\eqn{\hat R} over the +#' continuous parameters only), \code{ess_min} (over the continuous +#' parameters), \code{inclusion_ess_min} (over the indicators), #' \code{ebfmi_min}, \code{energy_variance_ratio_max} and #' \code{warmup_incomplete}, \code{sampler_notes}, and \code{seconds}.} #' \item{\code{verdict_by_scale}}{Fit-by-indicator matrix of verdicts.} @@ -481,6 +491,10 @@ prior_sensitivity_engine = function(bgms_object, for(k in seq_along(refit_idx)) fits[[refit_idx[k]]] = refits[[k]] fit_stats = lapply(fits, refit_edge_stats, evidence_threshold = evidence_threshold) + # The chosen-scale flag is verdicts()' near-threshold flag on the original + # fit, so the two reports count the same edges. + v0 = verdicts(bgms_object, evidence_threshold = evidence_threshold) + fit_stats[[s0_idx]]$fragile = v0$fragile[match(fit_stats[[s0_idx]]$edge, v0$parameter)] diag_rows = lapply(fits, refit_diagnostics_row) cls = sensitivity_classification(fit_stats, multipliers, s0_idx, lthr) @@ -570,23 +584,11 @@ sensitivity_classification = function(fit_stats, multipliers, s0_idx, lthr) { mcse_mat = t(stat_mat("mcse_lbf", "numeric")) verdict_mat = t(stat_mat("verdict", "character")) - # --- Sufficiency at the chosen scale, from the original fit ----------------- - # Could a rerun of the original fit with another seed have given this edge a - # different verdict, with no change of prior? Two Monte Carlo error measures - # of the log Bayes factor are available: twice its within-chain standard - # error (mcse_lbf) and its between-chain spread (band_half). The larger of - # the two is the half-width of an error band around the log Bayes factor. If - # one measure is NA (a single-chain fit has no between-chain spread), the - # other sets the half-width; if both are NA, the half-width is NA. The verdict - # is insufficient when that band contains a verdict threshold, +/- log of the - # evidence threshold, or when the chains' individual verdicts disagree. - hw = pmax(s0$band_half, 2 * s0$mcse_lbf, na.rm = TRUE) - straddle = ((s0$lbf - hw < lthr) & (s0$lbf + hw > lthr)) | - ((s0$lbf - hw < -lthr) & (s0$lbf + hw > -lthr)) - disagree = !s0$unanimous - insufficient = disagree | straddle - insufficient_noisy = straddle & insufficient - insufficient_disagree = disagree & insufficient + # --- The chosen-scale flag ------------------------------------------------- + # verdicts()' near-threshold flag on the original fit: could a rerun with + # another seed have given this edge a different verdict, with no change of + # prior? An edge whose standard errors are undefined is not flagged. + fragile = s0$fragile %in% TRUE # --- Mover category, where the verdict first differs, stability bracket ----- ref_v = s0$verdict @@ -634,10 +636,8 @@ sensitivity_classification = function(fit_stats, multipliers, s0_idx, lthr) { stability_upper = stability_upper, mover = mover, moved_at = moved_at, - insufficient = insufficient, - insufficient_noisy = insufficient_noisy, - insufficient_disagree = insufficient_disagree, - saturated = s0$zeroflip, + fragile = fragile, + saturated = s0$saturated, stringsAsFactors = FALSE, row.names = NULL ) vcols = as.data.frame(t(verdict_mat), stringsAsFactors = FALSE) @@ -680,11 +680,12 @@ order_upper_tri_rowmajor = function(p) { #' @description #' Prints the verdict counts at each scale, the number of edges whose verdict #' held at every scale, changed within Monte Carlo error, changed beyond it, -#' or could not be certified at the chosen scale, the edges that changed -#' beyond Monte Carlo error by name, the fits that did not sample (a -#' parameter constant in every chain at different values, infinite split -#' R-hat), the fits whose largest split R-hat exceeds 1.01, and how the chosen -#' scale compares with the scale the estimated interactions imply. +#' or could change on a rerun of the original fit, the edges that changed +#' beyond Monte Carlo error by name, the fits whose sampler did not move (a +#' parameter constant throughout every chain at values that differ between +#' chains, infinite split R-hat), the fits whose largest split R-hat exceeds +#' 1.01, and how the chosen scale compares with the scale the estimated +#' interactions imply. #' #' @param x A \code{bgms_prior_sensitivity} object. #' @param max_rows Integer. Maximum edges to name in the table of edges that @@ -760,19 +761,20 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { cat(paste0(" ", out, collapse = "\n"), "\n", sep = "") cat("\n") - # One category per edge: not certifiable at the chosen scale first, then the - # mover category. - uncert = edges$insufficient %in% TRUE | is.na(edges$insufficient) | - is.na(edges$mover) + # One category per edge: the verdicts that could change on a rerun first + # (verdicts()' flag on the original fit, and edges whose verdict change could + # not be compared with an error), then the mover category. + flagged = edges$fragile %in% TRUE + uncert = flagged | is.na(edges$mover) held = !uncert & edges$mover %in% "stable" within = !uncert & edges$mover %in% "within-monte-carlo-error" beyond = !uncert & edges$mover %in% "beyond-monte-carlo-error" counts = c(sum(held), sum(within), sum(beyond), sum(uncert)) labels = c( "verdict held at every scale", - "changed, within Monte Carlo error", - "changed, beyond Monte Carlo error", - "not certifiable at the chosen scale" + "changed within Monte Carlo error", + "changed beyond Monte Carlo error", + "could change on a rerun" ) cat(sprintf(" %-38s %4d\n", labels, counts), sep = "") cat("\n") @@ -797,45 +799,28 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { cat("\n") } - # The edges the original fit cannot certify, and why. + # The verdicts that could change on a rerun, by name; then the edges whose + # verdict change could not be compared with an error. if(sum(uncert) > 0) { - nm = edges$edge[uncert] - shown = utils::head(nm, max_rows) - tail_txt = if(length(nm) > length(shown)) { - sprintf(", and %d more (see $edges)", length(nm) - length(shown)) - } else { - "" - } - one = length(nm) == 1L - noisy = any(edges$insufficient_noisy[uncert], na.rm = TRUE) - disagree = any(edges$insufficient_disagree[uncert], na.rm = TRUE) - cause = if(noisy && disagree) { - "the Bayes factor lies within Monte Carlo error of a threshold or the chains disagree on the verdict" - } else if(disagree) { - "the chains disagree on the verdict" - } else { - "the Bayes factor lies within Monte Carlo error of a threshold" - } - explanation = if(noisy || disagree) { - sprintf( - "In the original fit %s, so a rerun with another seed could change %s verdict without any change of prior.", - cause, if(one) "its" else "the" - ) - } else { - character(0) + if(any(flagged)) { + nm = edges$edge[flagged] + shown = utils::head(nm, max_rows) + tail_txt = if(length(nm) > length(shown)) { + sprintf(", and %d more (see $edges)", length(nm) - length(shown)) + } else { + "" + } + one = length(nm) == 1L + wrap(sprintf( + "%d %s could change on a rerun without any change to the prior distribution: %s%s. %s within two standard errors of an evidence threshold.", + length(nm), if(one) "verdict" else "verdicts", + paste(shown, collapse = ", "), tail_txt, + if(one) "Its log Bayes factor lies" else "Their log Bayes factors lie" + )) } - if(anyNA(edges$insufficient[uncert]) || anyNA(edges$mover[uncert])) { - explanation = c( - explanation, - "Monte Carlo uncertainty could not be assessed for every verdict or verdict change; see $edges." - ) + if(any(uncert & !flagged)) { + wrap("Monte Carlo uncertainty could not be assessed for every verdict or verdict change; see $edges.") } - explanation = paste(explanation, collapse = " ") - wrap(sprintf( - "%d %s %s not certifiable at the chosen scale: %s%s. %s", - length(nm), if(one) unit$noun else unit$nouns, if(one) "is" else "are", - paste(shown, collapse = ", "), tail_txt, explanation - )) cat("\n") } @@ -844,14 +829,15 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { if(length(mcse_cols) > 0 && anyNA(edges[, mcse_cols])) { wrap(paste0( "Monte Carlo standard errors in $edges are NA where a fit's inclusion ", - "probability was 0 or 1 in every draw; there is no variation to estimate ", - "them from." + "probability was 0 or 1 in every draw of every chain, or constant to ", + "numerical precision; there is no variation to estimate them from." )) cat("\n") } - # Fits with a parameter constant at different values across chains, and - # fits whose largest split R-hat exceeds 1.01. + # Fits whose sampler did not move (a parameter constant throughout every + # chain at values that differ between chains), and fits whose largest split + # R-hat exceeds 1.01. g = x$grid if(!is.null(g$rhat_max)) { fit_label = function(i) { @@ -862,11 +848,13 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { one = length(stuck) == 1L wrap(sprintf( paste0( - "The %s %s at least one parameter with infinite split R-hat: ", - "its sub-chains are constant but differ in value. %s verdicts should ", - "be interpreted only after checking convergence." + "In the %s %s the sampler did not move: at least one parameter kept ", + "a single value throughout every chain, and those values differ ", + "between chains (split R-hat infinite). The verdicts of %s do not ", + "describe the posterior distribution at %s." ), - list_of(fit_label(stuck)), if(one) "fit includes" else "fits include", if(one) "Its" else "Their" + list_of(fit_label(stuck)), if(one) "fit" else "fits", + if(one) "that fit" else "those fits", if(one) "that scale" else "those scales" )) cat("\n") } @@ -908,9 +896,10 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "cold starts" it = g$iter[!g$original_fit][1] wu = g$warmup[!g$original_fit][1] + sampler_label = if(identical(x$refit_sampler, "nuts")) "NUTS" else x$refit_sampler cat(sprintf( - "Refits: %d %s refit%s (%d warmup, %d sampling iterations each), %s, %s total.\n", - n_refit, x$refit_sampler, if(n_refit == 1L) "" else "s", wu, it, start_txt, secs + "Refits: %d %s refit%s (%d warmup, %d sampling iterations each), %s, %s total.\n", + n_refit, sampler_label, if(n_refit == 1L) "" else "s", wu, it, start_txt, secs )) cat("See ?prior_sensitivity_check for the construction.\n") invisible(x) diff --git a/R/refit_engine.R b/R/refit_engine.R index 827a3237..6d8f6f33 100644 --- a/R/refit_engine.R +++ b/R/refit_engine.R @@ -256,10 +256,10 @@ verdict_from_lbf = function(lbf, lthr) { # ------------------------------------------------------------------ # Per-edge quantities from one refit: the Rao-Blackwellized inclusion # probability (canonical), its natural log inclusion Bayes factor and verdict, -# the within-chain MCSE of the log BF (RB machinery, PR #182), the between-chain -# half-band (2 * SEM of the grand mean from chain spread, propagated to the log -# BF), per-chain verdict unanimity, and the zero-flip mask. Edge order is the -# native row-major upper triangle. +# the MCSE of the log BF from the summary table's inclusion MCSE by the delta +# method, and whether the RB draws had no variation the ESS kernel could use +# (saturated: the summary MCSE is NA). Edge order is the native row-major +# upper triangle. # ------------------------------------------------------------------ refit_edge_stats = function(fit, evidence_threshold) { raw = get_raw_samples(fit) @@ -267,9 +267,8 @@ refit_edge_stats = function(fit, evidence_threshold) { # exactly. pnm = raw$parameter_names enm = pnm[["indicator"]] %||% pnm[["indicators"]] %||% pnm[["pairwise"]] - M = length(raw$rb_inclusion) - pc = sapply(raw$rb_inclusion, colMeans) # n_edges x M - if(is.null(dim(pc))) pc = matrix(pc, ncol = M) + pc = sapply(raw$rb_inclusion, colMeans) # n_edges x chains + if(is.null(dim(pc))) pc = matrix(pc, ncol = length(raw$rb_inclusion)) pbar = rowMeans(pc) prior_q = if(inherits(fit, "bgmCompare")) { @@ -287,23 +286,14 @@ refit_edge_stats = function(fit, evidence_threshold) { mcse_pip = ind[, "mcse"] lthr = log(evidence_threshold) - lbf = function(p) log((p / (1 - p)) / prior_odds) + lbf_bar = log((pbar / (1 - pbar)) / prior_odds) dfac = 1 / pmax(pbar * (1 - pbar), 1e-6) # d log BF / d p - lbf_bar = lbf(pbar) mcse_lbf = mcse_pip * dfac - se_between = apply(pc, 1, stats::sd) / sqrt(M) - band_half = 2 * se_between * dfac - - verdict = verdict_from_lbf(lbf_bar, lthr) - vc = apply(pc, 2, function(p) verdict_from_lbf(lbf(p), lthr)) # n_edges x M - if(is.null(dim(vc))) vc = matrix(vc, ncol = M) - unanimous = apply(vc, 1, function(v) length(unique(v[!is.na(v)])) <= 1L) - zeroflip = apply(pc, 1, function(p) all(p <= 0) || all(p >= 1)) list( edge = enm, pip = pbar, prior_odds = prior_odds, lbf = lbf_bar, - verdict = verdict, mcse_lbf = mcse_lbf, band_half = band_half, - unanimous = unanimous, zeroflip = zeroflip + verdict = verdict_from_lbf(lbf_bar, lthr), mcse_lbf = mcse_lbf, + saturated = is.na(mcse_pip) ) } diff --git a/R/verdicts.R b/R/verdicts.R index 098c3c7c..1a8e3df9 100644 --- a/R/verdicts.R +++ b/R/verdicts.R @@ -276,7 +276,8 @@ build_verdicts = function(parameter, log_bf, pip, mcse, draws, evidence_threshol #' \item{distance_two_state, distance_rb}{Distance from `log_bf` to the #' nearer verdict boundary, in units of each standard error.} #' \item{fragile}{`TRUE` when either distance is below 2: the -#' near-threshold flag.} +#' near-threshold flag. An indicator whose standard errors are both +#' undefined is not flagged.} #' } #' The evidence threshold is attached as the `evidence_threshold` attribute, #' and whether the fragility flag's operating point covers this kind of diff --git a/man/print.bgms_prior_sensitivity.Rd b/man/print.bgms_prior_sensitivity.Rd index 261d4c4b..c6fefb94 100644 --- a/man/print.bgms_prior_sensitivity.Rd +++ b/man/print.bgms_prior_sensitivity.Rd @@ -21,11 +21,12 @@ changed beyond Monte Carlo error; the rest are counted and left to \description{ Prints the verdict counts at each scale, the number of edges whose verdict held at every scale, changed within Monte Carlo error, changed beyond it, -or could not be certified at the chosen scale, the edges that changed -beyond Monte Carlo error by name, the fits that did not sample (a -parameter constant in every chain at different values, infinite split -R-hat), the fits whose largest split R-hat exceeds 1.01, and how the chosen -scale compares with the scale the estimated interactions imply. +or could change on a rerun of the original fit, the edges that changed +beyond Monte Carlo error by name, the fits whose sampler did not move (a +parameter constant throughout every chain at values that differ between +chains, infinite split R-hat), the fits whose largest split R-hat exceeds +1.01, and how the chosen scale compares with the scale the estimated +interactions imply. } \seealso{ \code{\link{prior_sensitivity_check}()} diff --git a/man/prior_sensitivity_check.Rd b/man/prior_sensitivity_check.Rd index 665fc890..af27fe50 100644 --- a/man/prior_sensitivity_check.Rd +++ b/man/prior_sensitivity_check.Rd @@ -69,16 +69,21 @@ factor) and \code{chosen_scale_verdict}; \code{stability_lower} and \code{stability_upper}; \code{mover} (\code{stable}, \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or \code{NA} for an indicator without a verdict) and \code{moved_at}; -\code{insufficient} with its two subcauses \code{insufficient_noisy} -(log Bayes factor within Monte Carlo error of a threshold) and -\code{insufficient_disagree} (chains disagree on the verdict); -\code{saturated} (the original fit's inclusion probability is 0 or 1 in -every draw of every chain); +\code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the +original fit); \code{saturated} (the original fit's Rao-Blackwellized +inclusion draws have no variation the ESS kernel can use, because the +acceptance probability of flipping the indicator was below double +precision at every update: the inclusion probability is 0 or 1 in every +draw of every chain, or constant to numerical precision; +\code{chosen_scale_mcse} is \code{NA} there; see the Details of +\code{\link{summary.bgms}}); then one \code{verdict_x}, \code{log_bf_x} and \code{mcse_x} column per multiplier \code{m}, from that fit's own statistics.} \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale}, \code{original_fit}, \code{iter}, \code{warmup}, \code{seed}, the -diagnostics \code{rhat_max}, \code{ess_min}, \code{inclusion_ess_min}, +diagnostics \code{rhat_max} (largest split-\eqn{\hat R} over the +continuous parameters only), \code{ess_min} (over the continuous +parameters), \code{inclusion_ess_min} (over the indicators), \code{ebfmi_min}, \code{energy_variance_ratio_max} and \code{warmup_incomplete}, \code{sampler_notes}, and \code{seconds}.} \item{\code{verdict_by_scale}}{Fit-by-indicator matrix of verdicts.} @@ -135,22 +140,27 @@ verdict differs. \code{stability_lower} and \code{stability_upper} are the smallest and largest multiplier in the contiguous run of fits around \code{1x} that hold the reference verdict. -\strong{Sufficiency at the chosen scale.} An edge whose per-chain verdicts -disagree in the original fit, or whose log Bayes factor lies within -\code{max(between-chain band, 2 * MCSE)} of a verdict threshold there, is -marked \code{insufficient}: the fit cannot certify the verdict the check -starts from, and the printed report lists the edge as not certifiable -rather than in a mover category. +\strong{The chosen-scale flag.} An edge is \code{fragile} when +\code{\link{verdicts}()} flags it in the original fit: its log Bayes factor +lies within two standard errors of a verdict threshold, by either of the two +standard errors \code{verdicts()} computes. The verdict the check starts +from could then change on a rerun with another seed, so the printed report +counts the edge among the verdicts that could change on a rerun rather than +in a mover category. An edge whose standard errors are undefined is not +flagged, as in \code{verdicts()}. \strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the -continuous parameters, smallest effective sample size over the continuous -parameters and over the indicators, and for NUTS the smallest E-BFMI and -the largest energy variance ratio, are in \code{$grid}. The classification -uses every fit; the printed report names a fit whose largest -split-\eqn{\hat R} exceeds 1.01, and states when a fit did not sample: a -parameter constant in every chain at values that differ across chains has -an infinite split-\eqn{\hat R}, and that fit's verdicts say nothing about -the prior. +continuous parameters (the main-effect and pairwise blocks of the fit +summary, not the indicators), smallest effective sample size over the +continuous parameters and over the indicators, and for NUTS the smallest +E-BFMI and the largest energy variance ratio, are in \code{$grid}. The +classification uses every fit; the printed report names a fit whose largest +split-\eqn{\hat R} exceeds 1.01, and states when a fit's sampler did not +move: a parameter that kept a single value throughout every chain, at values +that differ between chains, has an infinite split-\eqn{\hat R}, and the +verdicts of that fit do not describe the posterior distribution at that +scale. Such a refit needs a longer warmup or a cold start; an ordinal refit +starts from the original fit's final state, so \code{warmup} is the lever. \strong{Models with a precision diagonal.} A continuous or mixed fit has a Gamma or exponential prior on the precision diagonal with a standardized diff --git a/man/summary.bgmCompare.Rd b/man/summary.bgmCompare.Rd index 49a71cd9..f23587c0 100644 --- a/man/summary.bgmCompare.Rd +++ b/man/summary.bgmCompare.Rd @@ -39,6 +39,31 @@ The per-group counts behind the mark are in observation counts per variable (\code{NULL} for Blume-Capel variables, which are exempt from the union recode and are never marked). Fits made before that field existed print unmarked. + +The inclusion table (\verb{$indicator}) reports, per difference indicator, the Rao-Blackwellized +inclusion probability with its Monte Carlo standard error, effective sample +size and split-R-hat. At every update of a difference indicator the sampler +proposes to flip it and records the probability that the difference is in the +model after the move: the acceptance probability \eqn{\alpha} of adding it +when it is out, \eqn{1 - \alpha} of removing it when it is in. The +inclusion probability is the average of those recorded values, and \code{mcse}, +\code{n_eff} and \code{Rhat} are computed from their series. + +Those three cells are \code{NA}, and print blank, when the series has no +variation the Monte Carlo machinery can work on. For a difference the data +support overwhelmingly, the probability of accepting its removal is below +about \code{1e-16} at every update, so \eqn{1 - \alpha} rounds to exactly 1 in +double precision and the series is constant at 1 in every draw of every +chain; a difference the data rule out as strongly gives a series constant at 0. +A difference whose removal probability is around \code{1e-12} gives a series that +varies by that much, below the variance floor of \code{1e-15} at which the +autocorrelation kernels stop estimating. In both cases there is nothing to +estimate: the inclusion probability is at 1 or 0 to numerical precision, +its Monte Carlo error is smaller than any printed digit, and the verdict +lies beyond any evidence threshold. A difference only slightly less certain, with +a removal probability of \code{1e-8} say, has a varying series and shows numbers. +Main-effect difference rows are \code{NA} throughout when +\code{main_difference_selection = FALSE}: their indicators were never sampled. } \examples{ \donttest{ diff --git a/man/summary.bgms.Rd b/man/summary.bgms.Rd index 99b58486..0525c3f8 100644 --- a/man/summary.bgms.Rd +++ b/man/summary.bgms.Rd @@ -17,6 +17,30 @@ An object of class \code{summary.bgms} with posterior summaries. \description{ Returns posterior summaries and diagnostics for a fitted \code{bgms} model. } +\details{ +The inclusion table (\verb{$indicator}) reports, per edge, the Rao-Blackwellized +inclusion probability with its Monte Carlo standard error, effective sample +size and split-R-hat. At every update of an edge indicator the sampler +proposes to flip it and records the probability that the edge is in the +model after the move: the acceptance probability \eqn{\alpha} of adding it +when it is out, \eqn{1 - \alpha} of removing it when it is in. The +inclusion probability is the average of those recorded values, and \code{mcse}, +\code{n_eff} and \code{Rhat} are computed from their series. + +Those three cells are \code{NA}, and print blank, when the series has no +variation the Monte Carlo machinery can work on. For an edge the data +support overwhelmingly, the probability of accepting its removal is below +about \code{1e-16} at every update, so \eqn{1 - \alpha} rounds to exactly 1 in +double precision and the series is constant at 1 in every draw of every +chain; an edge the data rule out as strongly gives a series constant at 0. +An edge whose removal probability is around \code{1e-12} gives a series that +varies by that much, below the variance floor of \code{1e-15} at which the +autocorrelation kernels stop estimating. In both cases there is nothing to +estimate: the inclusion probability is at 1 or 0 to numerical precision, +its Monte Carlo error is smaller than any printed digit, and the verdict +lies beyond any evidence threshold. An edge only slightly less certain, with +a removal probability of \code{1e-8} say, has a varying series and shows numbers. +} \examples{ \donttest{ fit = bgm(x = Wenchuan[, 1:3]) diff --git a/man/verdicts.Rd b/man/verdicts.Rd index 27f2df9d..441cf510 100644 --- a/man/verdicts.Rd +++ b/man/verdicts.Rd @@ -37,7 +37,8 @@ where the Rao-Blackwellized draws are constant to double precision.} \item{distance_two_state, distance_rb}{Distance from \code{log_bf} to the nearer verdict boundary, in units of each standard error.} \item{fragile}{\code{TRUE} when either distance is below 2: the -near-threshold flag.} +near-threshold flag. An indicator whose standard errors are both +undefined is not flagged.} } The evidence threshold is attached as the \code{evidence_threshold} attribute, and whether the fragility flag's operating point covers this kind of diff --git a/src/mcmc_diagnostics.cpp b/src/mcmc_diagnostics.cpp index cd562dbd..543f0fb8 100644 --- a/src/mcmc_diagnostics.cpp +++ b/src/mcmc_diagnostics.cpp @@ -223,20 +223,32 @@ struct RhatWorker : public RcppParallel::Worker { } B *= n / (m - 1.0); + // A sub-chain counts as constant when its variance is below the same + // floor the ESS kernel applies to a chain's lag-0 autocovariance + // (biased variance below 1e-15), so a series the ESS kernel finds + // constant to numerical precision has no R-hat either. + bool all_constant = true; + for(int c = 0; c < m; c++) { + if(chain_var[c] * (n - 1.0) / n >= 1e-15) { + all_constant = false; + break; + } + } + // --- Classic split-Rhat (Gelman et al. 2013 / Stan) --- // var_plus = (n-1)/n * W + B/n; Rhat = sqrt(var_plus / W). - if(W > 0.0) { + if(!all_constant) { double var_plus = (n - 1.0) / n * W + B / n; rhat[j] = std::sqrt(var_plus / W); } else { - // W == 0: every sub-chain is individually constant. - // B == 0 -> all sub-chains hold the SAME constant; Rhat is undefined - // (0/0), report NA. - // B > 0 -> sub-chains are constant at DIFFERENT values (chains stuck - // in different states); the between/within ratio diverges, so - // report +Inf as a loud non-convergence alarm rather than the - // silent NA that would hide genuinely stuck chains. - rhat[j] = (B > 0.0) ? R_PosInf : NA_REAL; + // Every sub-chain is individually constant. + // The chain means agree to the same floor -> all sub-chains hold the + // SAME constant; Rhat is undefined (0/0), report NA. + // They differ -> sub-chains are constant at DIFFERENT values (chains + // stuck in different states); the between/within ratio + // diverges, so report +Inf as a loud non-convergence alarm + // rather than the silent NA that would hide stuck chains. + rhat[j] = (B / n >= 1e-15) ? R_PosInf : NA_REAL; } } } diff --git a/tests/testthat/_snaps/summary-marking.md b/tests/testthat/_snaps/summary-marking.md index d994f4cf..505a0305 100644 --- a/tests/testthat/_snaps/summary-marking.md +++ b/tests/testthat/_snaps/summary-marking.md @@ -32,8 +32,8 @@ B-C (pairwise) C (main) Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion - probability was 0 or 1 in every draw; there is no variation to estimate them - from. + probability was 0 or 1 in every draw of every chain, or constant to numerical + precision; there is no variation to estimate them from. Main-effect difference rows are blank when main_difference_selection = FALSE left their indicators unsampled. diff --git a/tests/testthat/test-mcmc-diagnostics.R b/tests/testthat/test-mcmc-diagnostics.R index ca924256..0ee2b729 100644 --- a/tests/testthat/test-mcmc-diagnostics.R +++ b/tests/testthat/test-mcmc-diagnostics.R @@ -375,6 +375,22 @@ test_that("all-constant-and-equal sub-chains give NA", { expect_true(is.na(rhat)) }) +test_that("sub-chains constant to numerical precision at one value give NA, at different values Inf", { + # The R-hat kernel applies the ESS kernel's variance floor (1e-15), so a + # series the ESS kernel finds constant has no R-hat either, and a Rao- + # Blackwellized inclusion series at 1 - 1e-12 gets NA in all three cells. + set.seed(11) + near_one = array(1 - abs(rnorm(400, sd = 1e-12)), dim = c(100, 4, 1)) + expect_true(is.na(bgms:::.compute_ess_cpp(near_one))) + expect_true(is.na(bgms:::.compute_rhat_cpp(bgms:::split_chains(near_one)))) + apart = near_one + apart[, 3:4, 1] = abs(rnorm(200, sd = 1e-12)) + expect_identical(bgms:::.compute_rhat_cpp(bgms:::split_chains(apart)), Inf) + # A variance at the floor is not constant: the classic formula applies. + varying = array(1 + rnorm(400, sd = 1e-6), dim = c(100, 4, 1)) + expect_true(is.finite(bgms:::.compute_rhat_cpp(bgms:::split_chains(varying)))) +}) + test_that("constant-but-unequal sub-chains give +Inf, not NA", { # Two chains stuck at 0, two stuck at 1: W == 0 but B > 0. Silence (NA) here # is the worst failure mode, so the estimator must raise a loud +Inf alarm. diff --git a/tests/testthat/test-prior-sensitivity.R b/tests/testthat/test-prior-sensitivity.R index c6e31aeb..3bd16649 100644 --- a/tests/testthat/test-prior-sensitivity.R +++ b/tests/testthat/test-prior-sensitivity.R @@ -22,15 +22,14 @@ skip_unless_slow = function() { # Per-fit statistics in the shape refit_edge_stats() returns, for n edges. fake_stats = function(lbf, mcse, prior_q = 0.5, lthr = log(10), - unanimous = TRUE, zeroflip = FALSE) { + fragile = FALSE, saturated = FALSE) { n = length(lbf) prior_odds = rep(prior_q / (1 - prior_q), n) pip = plogis(lbf + log(prior_odds)) list( edge = paste0("e", seq_len(n)), pip = pip, prior_odds = prior_odds, lbf = lbf, verdict = verdict_from_lbf(lbf, lthr), mcse_lbf = mcse, - band_half = rep(0, n), unanimous = rep_len(unanimous, n), - zeroflip = rep_len(zeroflip, n) + fragile = rep_len(fragile, n), saturated = rep_len(saturated, n) ) } @@ -92,23 +91,37 @@ test_that("the classification compares each verdict change with the two fits' Mo }) -test_that("sufficiency at the chosen scale is the only not-certifiable criterion", { +test_that("the chosen-scale flag is the original fit's verdicts() flag", { lthr = log(10) multipliers = c(0.5, 1, 2) - # Edge 1 sits within 2 * MCSE of the presence threshold at 1x; edge 2 is - # within error at 2x only; edge 3 has disagreeing chains at 1x. - s0 = fake_stats(c(2.2, 1.0, 1.5), mcse = c(0.1, 0.1, 0.1), unanimous = c(TRUE, TRUE, FALSE)) + # Edge 1 is flagged by verdicts() in the original fit; edge 2 is near a + # threshold at 2x only, which is not a flag. + s0 = fake_stats(c(2.2, 1.0, 1.5), mcse = c(0.1, 0.1, 0.1), fragile = c(TRUE, FALSE, FALSE)) s_lo = fake_stats(c(2.2, 1.0, 1.5), mcse = c(0.1, 0.1, 0.1)) s_hi = fake_stats(c(2.2, 2.5, 1.5), mcse = c(0.1, 0.1, 0.1)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) - expect_equal(cls$edges$insufficient, c(TRUE, FALSE, TRUE)) - expect_equal(cls$edges$insufficient_noisy, c(TRUE, FALSE, FALSE)) - expect_equal(cls$edges$insufficient_disagree, c(FALSE, FALSE, TRUE)) + expect_equal(cls$edges$fragile, c(TRUE, FALSE, FALSE)) + expect_false(any(c("insufficient", "insufficient_noisy", "insufficient_disagree") %in% names(cls$edges))) # Edge 2 is classified on its own change, not on its nearness to a threshold # at the other scale. expect_equal(cls$edges$mover[2], "beyond-monte-carlo-error") }) +test_that("the check's flag on a fit is verdicts()' flag on that fit", { + skip_on_cran() + fit = bgm(Wenchuan[1:150, 1:5], + iter = 300, warmup = 300, chains = 2, seed = 3, + display_progress = "none" + ) + ps = prior_sensitivity_check(fit, scale_multipliers = 2, seed = 3) + v = verdicts(fit) + expect_identical(ps$edges$fragile, v$fragile[match(ps$edges$edge, v$parameter)]) + # saturated follows the kernel: exactly the edges whose chosen-scale MCSE is NA. + expect_identical(ps$edges$saturated, is.na(ps$edges$chosen_scale_mcse)) + ind = summary(fit)$indicator + expect_identical(unname(is.na(ind$mcse)), unname(is.na(ind$Rhat))) +}) + test_that("saturated and verdict-less indicators are handled", { lthr = log(10) @@ -116,14 +129,14 @@ test_that("saturated and verdict-less indicators are handled", { # Edge 1 is at inclusion probability 1 in every draw of the original fit # (infinite log BF, no MCSE) and becomes finite presence at 2x; edge 2 has # no verdict. - s0 = fake_stats(c(Inf, NA), mcse = c(NA, NA), zeroflip = c(TRUE, FALSE)) + s0 = fake_stats(c(Inf, NA), mcse = c(NA, NA), saturated = c(TRUE, FALSE)) s0$verdict[2] = NA_character_ - s_lo = fake_stats(c(Inf, NA), mcse = c(NA, NA), zeroflip = c(TRUE, FALSE)) + s_lo = fake_stats(c(Inf, NA), mcse = c(NA, NA), saturated = c(TRUE, FALSE)) s_hi = fake_stats(c(4, NA), mcse = c(0.2, NA)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) expect_equal(cls$edges$mover, c("stable", NA)) expect_true(cls$edges$saturated[1]) - expect_false(cls$edges$insufficient[1]) + expect_false(cls$edges$fragile[1]) expect_true(is.na(cls$edges$stability_lower[2])) # A verdict change away from a saturated indicator cannot be compared with # an error, since the saturated fit has none. @@ -138,30 +151,29 @@ test_that("an unknown Monte Carlo error is not assessed", { multipliers = c(0.5, 1, 2) # Edge 1: the refit whose verdict differs has no MCSE, so the change cannot # be compared with an error and the mover category is NA. Edge 2: the - # original fit has no MCSE and no band, so its sufficiency is NA. + # original fit has no MCSE; verdicts() does not flag an undefined standard + # error, so the edge is not flagged. s0 = fake_stats(c(1.0, 1.0), mcse = c(0.1, NA)) - s0$band_half[2] = NA_real_ s_lo = fake_stats(c(1.0, 1.0), mcse = c(0.1, 0.1)) s_hi = fake_stats(c(2.5, 1.0), mcse = c(NA, 0.1)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) expect_true(is.na(cls$edges$mover[1])) expect_equal(cls$edges$moved_at[1], 2) - expect_false(cls$edges$insufficient[1]) - expect_true(is.na(cls$edges$insufficient[2])) + expect_false(cls$edges$fragile[1]) + expect_false(cls$edges$fragile[2]) expect_equal(cls$edges$mover[2], "stable") - # A single-chain fit has no between-chain band; the MCSE alone sets the - # half-width. - s1 = fake_stats(c(2.2, 1.0), mcse = c(0.1, 0.1)) - s1$band_half[] = NA_real_ - cls1 = sensitivity_classification(list(s1, s1, s1), multipliers, 2L, lthr) - expect_equal(cls1$edges$insufficient, c(TRUE, FALSE)) }) test_that("print() reports the counts, the movers and high R-hat fits", { lthr = log(10) multipliers = c(0.5, 1, 2) - s0 = fake_stats(c(3.0, 1.0, 2.0, -3.0, 2.6), mcse = c(0.2, 0.1, 0.2, 0.2, 0.3)) + # Edges 3 and 5 are flagged by verdicts() in the original fit, so they count + # as verdicts that could change on a rerun rather than as movers. + s0 = fake_stats(c(3.0, 1.0, 2.0, -3.0, 2.6), + mcse = c(0.2, 0.1, 0.2, 0.2, 0.3), + fragile = c(FALSE, FALSE, TRUE, FALSE, TRUE) + ) s_lo = fake_stats(c(3.2, 2.5, 2.1, -2.9, 2.8), mcse = c(0.2, 0.1, 0.2, 0.2, 0.3)) s_hi = fake_stats(c(2.9, 1.1, 2.35, -3.1, 2.2), mcse = c(0.2, 0.1, 0.2, 0.2, 0.3)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) @@ -169,20 +181,19 @@ test_that("print() reports the counts, the movers and high R-hat fits", { ps = fake_sensitivity(cls, multipliers, rhat_max = c(1.03, 1.005, 1.02), s_hat = 0.2) out = gsub("\\s+", " ", paste(utils::capture.output(print(ps)), collapse = " ")) expect_match(out, "fits at 0.5x, 1x and 2x the chosen scale") - # Edges 3 and 5 lie within 2 * MCSE of the presence threshold in the - # original fit, so they are not certifiable rather than movers. expect_match(out, "verdict held at every scale 2") - expect_match(out, "changed, within Monte Carlo error 0") - expect_match(out, "changed, beyond Monte Carlo error 1") - expect_match(out, "not certifiable at the chosen scale 2") - expect_match(out, "2 edges are not certifiable at the chosen scale: e3, e5") - expect_match(out, "within Monte Carlo error of a threshold, so a rerun") + expect_match(out, "changed within Monte Carlo error 0") + expect_match(out, "changed beyond Monte Carlo error 1") + expect_match(out, "could change on a rerun 2") + expect_match(out, "2 verdicts could change on a rerun without any change to the prior distribution: e3, e5") + expect_match(out, "Their log Bayes factors lie within two standard errors of an evidence threshold") + expect_false(grepl("not certifiable", out, fixed = TRUE)) expect_match(out, "1 edge changed verdict beyond Monte Carlo error") expect_match(out, "e2 presence undecided undecided 0.5x") expect_match(out, "Largest split R-hat above 1.01: 0.5x fit 1.03 and 2x fit 1.02") expect_match(out, "the slab is 5 times that") expect_match(out, "toward absence") - expect_match(out, "2 nuts refits") + expect_match(out, "Refits: 2 NUTS refits") # A changed verdict with an unavailable MCSE belongs in the fourth count. unknown = fake_stats(2.5, mcse = NA_real_) @@ -192,7 +203,7 @@ test_that("print() reports the counts, the movers and high R-hat fits", { ) out_unknown = utils::capture.output(print(fake_sensitivity(cls_unknown, multipliers))) count_lines = grep( - "^ (verdict held|changed,|not certifiable)", out_unknown, + "^ (verdict held|changed |could change)", out_unknown, value = TRUE ) counts = as.integer(sub(".* +([0-9]+)$", "\\1", count_lines)) @@ -206,11 +217,11 @@ test_that("print() reports the counts, the movers and high R-hat fits", { utils::capture.output(print(fake_sensitivity(cls_mixed, multipliers))), collapse = " " )) - expect_match(out_mixed, "not certifiable at the chosen scale 3") - expect_match(out_mixed, "within Monte Carlo error of a threshold, so a rerun") + expect_match(out_mixed, "could change on a rerun 3") + expect_match(out_mixed, "2 verdicts could change on a rerun without any change to the prior distribution: e3, e5") expect_match(out_mixed, "Monte Carlo uncertainty could not be assessed for every verdict or verdict change") - # An NA-only group has no known threshold or chain-disagreement cause. - expect_false(any(grepl("rerun with another seed", out_unknown, fixed = TRUE))) + # An NA-only group names no flagged verdict. + expect_false(any(grepl("without any change to the prior distribution", out_unknown, fixed = TRUE))) # Every fit at or below 1.01, and a scale within a factor two: neither # sentence appears, and the verdict table has every scale. @@ -225,8 +236,8 @@ test_that("print() reports the counts, the movers and high R-hat fits", { # out of the numeric R-hat line. ps3 = fake_sensitivity(cls, multipliers, rhat_max = c(1.03, 1.005, Inf), s_hat = 0.8) out3 = gsub("\\s+", " ", paste(utils::capture.output(print(ps3)), collapse = " ")) - expect_match(out3, "The 2x fit includes at least one parameter with infinite split R-hat") - expect_match(out3, "Its verdicts should be interpreted only after checking convergence") + expect_match(out3, "In the 2x fit the sampler did not move: at least one parameter kept a single value throughout every chain, and those values differ between chains \\(split R-hat infinite\\)") + expect_match(out3, "The verdicts of that fit do not describe the posterior distribution at that scale") expect_match(out3, "Largest split R-hat above 1.01: 0.5x fit 1.03\\.") expect_false(grepl("Monte Carlo standard errors in $edges", out3, fixed = TRUE)) # An NA Monte Carlo error in some fit is explained in one sentence. @@ -234,11 +245,11 @@ test_that("print() reports the counts, the movers and high R-hat fits", { cls_na$edges$mcse_x2[1] = NA_real_ ps5 = fake_sensitivity(cls_na, multipliers, s_hat = 0.8) out5 = gsub("\\s+", " ", paste(utils::capture.output(print(ps5)), collapse = " ")) - expect_match(out5, "Monte Carlo standard errors in \\$edges are NA where a fit's inclusion probability was 0 or 1 in every draw") + expect_match(out5, "Monte Carlo standard errors in \\$edges are NA where a fit's inclusion probability was 0 or 1 in every draw of every chain, or constant to numerical precision") ps4 = fake_sensitivity(cls, multipliers, rhat_max = c(Inf, 1.005, Inf), s_hat = 0.8) out4 = gsub("\\s+", " ", paste(utils::capture.output(print(ps4)), collapse = " ")) - expect_match(out4, "The 0.5x and 2x fits include at least one parameter with infinite split R-hat") - expect_match(out4, "Their verdicts should be interpreted only after checking convergence") + expect_match(out4, "In the 0.5x and 2x fits the sampler did not move") + expect_match(out4, "The verdicts of those fits do not describe the posterior distribution at those scales") expect_false(grepl("Largest split R-hat", out4, fixed = TRUE)) }) diff --git a/vignettes/diagnostics.Rmd b/vignettes/diagnostics.Rmd index 3648661d..7a80918f 100644 --- a/vignettes/diagnostics.Rmd +++ b/vignettes/diagnostics.Rmd @@ -85,19 +85,24 @@ summary(fit)$indicator ``` - **mean**, **mcse**, **sd**, **n_eff**, and **Rhat** describe the - Rao-Blackwellized inclusion probability. The sampler records a one-step - inclusion draw at every indicator update, and that draw is continuous, so the - standard Monte Carlo machinery applies to it. They are `NA` when that draw - carries too little variability for the machinery to work on, which happens by - two routes. The first is an indicator whose Rao-Blackwellized draws are - constant to double precision, where the inclusion probability sits at its - numerical bound; `sd` is then exactly `0`. The second is a draw whose - variance is non-zero but falls below the numerical floor of the - autocovariance kernel: `mcse` and `n_eff` come back `NA` while `sd` and - `Rhat` are still reported. In the table above, `intrusion-flash` takes the - second route — an `sd` of about `2e-11` with `NA` `mcse` and `n_eff` and a - finite `Rhat`. Either way the reading is the same: the inclusion probability - is pinned, not poorly estimated. + Rao-Blackwellized inclusion probability. At every update of an edge + indicator the sampler proposes to flip it and records the probability that + the edge is in the model after the move: the acceptance probability of + adding it when it is out, one minus the acceptance probability of removing + it when it is in. That recorded value is continuous, so the standard Monte + Carlo machinery applies to its series. All three of `mcse`, `n_eff` and + `Rhat` are `NA` when the series has no variation the machinery can work on. + For an edge the data support overwhelmingly, the probability of accepting + its removal is below about 1e-16 at every update, so one minus it rounds to + exactly 1 and the series is constant at 1 in every draw of every chain; an + edge the data rule out as strongly is constant at 0. An edge whose removal + probability is around 1e-12 gives a series that varies by that much, below + the variance floor of 1e-15 at which the ESS and R-hat kernels stop + estimating, so `sd` prints as a number like `2e-11` while the three cells + are blank. In both cases there is nothing to estimate: the inclusion + probability is at 1 or 0 to numerical precision and the verdict lies beyond + any evidence threshold. An edge only slightly less certain, with a removal + probability of 1e-8 say, has a varying series and shows numbers. ## Traceplots diff --git a/vignettes/prior-sensitivity-ps.rds b/vignettes/prior-sensitivity-ps.rds index 4345e222bdedaae304380bf3d09ac22952b307e2..048d091b1e55ac5073d691df3b8c5424e3d51cc6 100644 GIT binary patch literal 2053 zcmV+g2>SOQiwFP!000001Lar?a1`Yo-#qS;J94=Td7(v23AQm3AaRsPCEqfDhKB($ zB08p0z!l! z4;$0n+x@<~y@SyB2+nk8cE9iYegDVz`aiy1V8t*j7K=A4-#Dcjm#Td1dYP`gj$k_; zv#52NlSRKo^IV==q)1lMaeD~&Fr?&*hW&y>$vT`8f(2i-Qi{JsVi(eZQr z{ehMDdcdhSiqBO}c>$d3xcJ!BNjYGx`PcgkCyooz9cbIM=iXgl)5e#C#|fb zvuZDR{iC8q1kt7KZdm=nhUP$g>8Ed%dq=cfp5%IY;`)~ZN$ESD z-1FoE`1r3Ms;P;&gy$^y&DCH2t^xAFvgQ0klh>Da4M+Mft7yVnuru=y=U3MhgFRak zMnACjDe%w26J;e+kAV*|@%fW>?n88)zukG>lYJ5F*eFcdGPwe5f7W)UtmRkW!*h+7 zsmgNDoKbYNWX<#7XkEd~&DM7yUr0~m+}@^9DVWis5(SZxD9%M)r)}|bZpuZwDfe{< z5=iGj{5K$v5q>m_}eq3)1b%`1G-7LwZlc9dV)(9t6&2#QNmgi(D z&qaz1pGS!z@>W0O2n;r^*uWr5xM$y%U-?*rYd!E2bAx^;<63kB@y3MpJFuH)>rwKC z)Y@<6%x|Wb`+uFcF{&4|HssbjcH!8{W%u1br4vtiw7dBi>+cMhz8IDEbZAzW71(chANEo}OZUp$(@!pP(%@grdaJn+R(;To1^-Jek|AXj~+4eaxhaLN#XYUJOwb>^RzHlnw_($10Rcro+-*sy2o6j9B#P4W$W%u|+{Zab+4&T{sTV04n zW;L~+zn)op@b};^%rbcfa}XRVT4wS7y%%iCs}I~aGaJzz8FS>q^!Fx!&41$8+}64Q zY+hWHGP<)6Y@BsZUCtIx$>vPMKHoMH9DaLN^ZoXDQCUsnt{Ej=Cti%wKlE9ZQU5FJ z`vRqOD`((65*uA3t_J#Ao>BFq!-y_Sx*}JbaT_p}j>2G`DnTbm_@E8^!i>Y2;GYk_)vXrDd zgLEB?r`%p>IX@9ZPlhhmB|&v*hK>qKbki<)>9MOZJE&L@R!ND`uwM`mY-0He)JRktZi0lO5#6dUcneptO&(zm~p1QaH zy0U8?;$J-H`E{*dPQY1t<*hxhK7yBDx%#3h_bb%Du%0oqW9Q?p8jCr5`853T$}#ot zz2rpl&%Cy~PkQ4qaP~x_FmqiBIGeg_cy`m-kbdXL!lVmbO4+@m?BV60r~Tq><&G^8 zbHCPl80LZBTAH~(48(8gJ>m$~)QyYj$F!|YhFi<}8JQLsI&{Q?RsxSKFVbF`BN@U& z!($##%ZS)qNs6o=+Ge~&g+jULBV~dmtKfN~B#B^YF7&Sh+P<;FLODGw9R}P&ag^w- zCMrmgCY9hzC?-YoVUkm{F{s{zX9b3m;oxl&skTLe<7Eo|t4WY3mvS$VwDpLE>v7$q z9@mc=(yB)DZ%N)uwi2~N4SJ!fuzMsT8WmL zXN)p!1GSds7%`KIMF!RJq)27SJ}OJ?%+iCtl%*s>eo4-f`Cy;Kkj|{2T7YRzU%y=& zW^IOvf|lTTXo^8OM|kQ3hsg1M=v_@}9s0&ZA4R$unnUM6ITs#E6oVHW=<{O2elzG> zaqWfR{IX;mw6ULOMGumZJEmF)iU@8&oKA~HmxhVQL0L+g1H(aVKB^T|>Er$;g3oT; z7+r{0q~Wrq=s6?`j?w_@>3|uVv)X{$(efFW-_M~~n&hTQFR#W|N+xw|I8~t-mKy0~ zpJJCj9&M&gkw?pf9@d~ZHTOKx^%2~zM;QPBg1IO8 literal 2075 zcmV+$2;}!4iwFP!000001Lar?a1=!vo;-GwEP0X_cOXWCbr=#b@r1}pdMHH0!w|iQ zoMoKM?)L6>F^{d8*}RH7E|zdv6+9m5iElhK8q4F*;}Jr#V1&zsLc}|QP<4k12UNrm zA;`mvxtX2m*`3`07tuh~ZPjc~fBiqY|L*_Mv&cdaL?jVymj9#V*QiwaFG*`N5 zKRr0;TRqhQPQF@vx@zXL;B@D?M=njD2-cf_y}xkU+OO`Ea}Mt4YZpows29OEsLFaw2m^w*`W; zv)9b)D#{?!XT3k(`uswlWywR$%XTi)>C$#LuKnPh7GHGPr*Bj^$FyFUZhK)`!wbIn z^c|1wdF(!N%C`^H)<&EsCocNUrN{r!i1~2adgg%{4Q1VvYiorB-oktVJx!(k&EsXyGI&(1TYw_uZ16JuI1gusHe`qP z%MirS)+_N}g*;6BFvSgH`qz5y`-Vv$(qjFj)*0!RFz&lqlxQmhf5hGxE3Dx;dmhVk z63nyF0>kG)LC~Ysk2wQ_kIOzV%ogw2_vM%WEPh*G+wVq`5i!KrP+0I79#1B&U(5-U zH=M%(|_bsMXTg0{vSZ}KjZSiR!zduDc#DG&Fw{G#D@pXsadX;1Wb=yVCS zjpr{;-_|;C{}Yz^rQ2J3ww}DD_%q})*{XB;9z8(bR$|>>{K;Z+Oh&&O-Ps;iU3^j4 znUQ|xmtp#^S9OEs<{xtBuB#sd{&lRsvvcz_G8KGTmEL9cCAq!JOD+G==@QwF1rdkt z_l<&2xA61#`G~sg69=C?=}Z1c`CHZN{zgtZnfuz)M+?c@8eiN!WyxTe{(-}{cOiZTNt^jG>!Uv+c7grLn>AH^$K<;-%%WrYVMqbK zZmJdvW{bm6ekg#M;uPBjv&5)CzOh>ny6rak%8IuggBCk2(UTp5d?G@~;{?8HGCnB@ z@)P-6`EPJ=@N2pGPWu#)s6A1k1vyFRzpITag-n$8jti3n6^q#)c!tFMvdJ~`Rhesw zwYJo#yDzisCF-CZ^v!wd&!@a~ps)VTzpn0nNatU=;F%3=U;mtBrPV*}dkK;i7cc$Z zl=F=~zSzr{*<2fGtBo^fuPh~hRW-r;-t!f@{8KOQ9uQxB1oR$na?jn60(w(-jm~cF z4d}OyD~vzeEw?>8${$<_`Z~_tRFS+TWbH%Q!>|r=Ea=@B1d?NOh1dOW%Emg!|;*i1>{5=%}@@6-60xDIu3=M$^gMdOB732|ECijfpM8#(USKARV~+EHea=D z8)lt$w@~cSV`}Hw(sEsC+=<2`+IX@_kQw5C+?FI-hMx=_KSHTVa=omG0LUq^hk!sZ!Uxb zmy$0AXk2c>O4+fLOeeWyzjX2XW+uq;kVI)|4U1FKpPtQmJ7g%S#;wmm z@Ng1h<+H1$J#BreBt<*NixRTwMv&pl<;D!FlNBko?3ZmIA_-M0#lCFc{{j@0gCGkU F005a_95ny{ diff --git a/vignettes/prior-sensitivity.Rmd b/vignettes/prior-sensitivity.Rmd index f7e64f94..ca613216 100644 --- a/vignettes/prior-sensitivity.Rmd +++ b/vignettes/prior-sensitivity.Rmd @@ -82,17 +82,19 @@ The four counts below that table classify every edge against the verdict of the original fit, the `1x` column: - *verdict held at every scale*: the edge has the same verdict in every fit. -- *changed, within Monte Carlo error*: the verdict differs at some scale, but +- *changed within Monte Carlo error*: the verdict differs at some scale, but the change in the log Bayes factor from the original fit to that fit is smaller than twice the combined Monte Carlo standard error of the two fits. Two runs of the same analysis could have produced this difference. -- *changed, beyond Monte Carlo error*: the change is larger than that. The +- *changed beyond Monte Carlo error*: the change is larger than that. The verdict depends on the scale. These edges are named, with their verdict at each scale and the scale nearest the chosen one at which it differs. -- *not certifiable at the chosen scale*: in the original fit the log Bayes - factor lies within Monte Carlo error of a verdict threshold, or the chains - disagree on the verdict. The verdict the check starts from is not settled, - so the edge is not classified; running the original fit longer settles it. +- *could change on a rerun*: `verdicts()` flags the edge in the original fit, + because its log Bayes factor lies within two standard errors of an evidence + threshold. The verdict the check starts from could change on a rerun with + another seed and no change to the prior distribution, so the edge is not + classified by scale. The same edges carry `fragile = TRUE` in + `verdicts(fit)`. A line naming fits whose largest split R-hat exceeds 1.01 appears when there is one. The check uses every fit; that line, and the per-fit diagnostics in @@ -109,9 +111,9 @@ moves near-threshold verdicts. Bayes factor, Monte Carlo standard error and verdict; the `mover` category and `moved_at`; `stability_lower` and `stability_upper`, the smallest and largest multiplier in the contiguous run of fits around `1x` that hold the -original verdict; the `insufficient` flag with its two subcauses; and one -verdict, log Bayes factor and Monte Carlo standard error column per scale, -each from that fit's own statistics. +original verdict; the `fragile` flag of `verdicts()` on the original fit and +the `saturated` flag; and one verdict, log Bayes factor and Monte Carlo +standard error column per scale, each from that fit's own statistics. ```{r, eval = FALSE} head(ps$edges) From 06ee4af867d4528ce6d000a34b660b7c49348631 Mon Sep 17 00:00:00 2001 From: Maarten Marsman Date: Sat, 7 Nov 2026 09:37:52 +0100 Subject: [PATCH 2/3] fix(sensitivity): leave the split-R-hat computation as it was; correct the blank-cell and infinite-R-hat wording The split-R-hat kernel is unchanged from develop: a sub-chain counts as constant only when its draws are exactly equal, so the Rhat of an inclusion series that is constant to numerical precision is computed, as before. The saturated column of prior_sensitivity_check() stays the edges whose chosen-scale Monte Carlo standard error is NA. The Details of summary.bgms and summary.bgmCompare and the diagnostics vignette say that mcse and n_eff are NA when the series has no usable variation and that Rhat is NA only for exactly constant chains; the recorded value for an included edge is the probability of retaining it; the cutoff is a variation of about 3e-8, not a removal probability of 1e-8. The infinite-R-hat sentence states what the diagnostic shows: a parameter constant within every split chain at values that differ between split chains, which the sampler did not explore. The help page says such a refit may need a longer warmup or a cold start. NEWS states that the fourth count is the verdicts() flag plus the edges whose verdict change could not be compared with an error. --- NEWS.md | 20 ++++++++-------- R/methods_bgmcompare.R | 25 +++++++++++--------- R/methods_bgms.R | 25 +++++++++++--------- R/prior_sensitivity.R | 25 ++++++++++---------- man/print.bgms_prior_sensitivity.Rd | 6 ++--- man/prior_sensitivity_check.Rd | 13 ++++++----- man/summary.bgmCompare.Rd | 25 +++++++++++--------- man/summary.bgms.Rd | 25 +++++++++++--------- src/mcmc_diagnostics.cpp | 30 +++++++----------------- tests/testthat/test-mcmc-diagnostics.R | 16 ------------- tests/testthat/test-prior-sensitivity.R | 8 +++---- vignettes/diagnostics.Rmd | 31 +++++++++++++------------ 12 files changed, 117 insertions(+), 132 deletions(-) diff --git a/NEWS.md b/NEWS.md index ec99ce37..18906f58 100644 --- a/NEWS.md +++ b/NEWS.md @@ -167,7 +167,9 @@ released from this line yet. * The fourth count of the printed `prior_sensitivity_check()` report, "could change on a rerun", is `verdicts()`' near-threshold flag on the original fit, - computed by the same function, so the two reports count the same edges. The + computed by the same function, plus the edges whose verdict change could not + be compared with a Monte Carlo error (a refit with an `NA` error); each part + has its own sentence, and the counts agree whenever no such edge exists. The between-chain band and the per-chain verdict test that the check applied on top of it are gone, as are the `$edges` columns `insufficient`, `insufficient_noisy` and `insufficient_disagree`; the flag is the `fragile` @@ -175,15 +177,13 @@ released from this line yet. the counts names the flagged edges and states that their log Bayes factors lie within two standard errors of an evidence threshold. -* The split-R-hat kernel applies the ESS kernel's variance floor: a sub-chain - whose variance is below 1e-15 counts as constant, so a Rao-Blackwellized - inclusion series that is constant to numerical precision has `NA` for - `mcse`, `n_eff` and `Rhat` together, where before `Rhat` was computed from - the residual variation. The `saturated` column of `prior_sensitivity_check()` - follows the same criterion, so it marks exactly the edges whose chosen-scale - Monte Carlo standard error is `NA`. The one-sentence note under the summary - tables and in the check now says "0 or 1 in every draw of every chain, or - constant to numerical precision", and the Details of `?summary.bgms`, +* The `saturated` column of `prior_sensitivity_check()` marks the edges whose + chosen-scale Monte Carlo standard error is `NA`: the original fit's + Rao-Blackwellized inclusion series had no variation the ESS kernel could use, + whether exactly constant or constant to numerical precision. Before, it + marked only the edges at exactly 0 or 1 in every chain. The one-sentence note + under the summary tables and in the check now says "0 or 1 in every draw of + every chain, or constant to numerical precision", and the Details of `?summary.bgms`, `?summary.bgmCompare` and `vignette("diagnostics")` explain why: the recorded inclusion probability is one minus the acceptance probability of removing the edge, which rounds to 1 once that probability is below about diff --git a/R/methods_bgmcompare.R b/R/methods_bgmcompare.R index a1b84e4b..f04376b5 100644 --- a/R/methods_bgmcompare.R +++ b/R/methods_bgmcompare.R @@ -244,23 +244,26 @@ print.bgmCompare = function(x, ...) { #' size and split-R-hat. At every update of a difference indicator the sampler #' proposes to flip it and records the probability that the difference is in the #' model after the move: the acceptance probability \eqn{\alpha} of adding it -#' when it is out, \eqn{1 - \alpha} of removing it when it is in. The +#' when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it +#' is in. The #' inclusion probability is the average of those recorded values, and `mcse`, #' `n_eff` and `Rhat` are computed from their series. #' -#' Those three cells are `NA`, and print blank, when the series has no -#' variation the Monte Carlo machinery can work on. For a difference the data +#' `mcse` and `n_eff` are `NA`, and print blank, when the series has no +#' variation the Monte Carlo machinery can work on; `Rhat` is `NA` when every +#' chain is exactly constant, and otherwise computed. For a difference the data #' support overwhelmingly, the probability of accepting its removal is below #' about `1e-16` at every update, so \eqn{1 - \alpha} rounds to exactly 1 in #' double precision and the series is constant at 1 in every draw of every -#' chain; a difference the data rule out as strongly gives a series constant at 0. -#' A difference whose removal probability is around `1e-12` gives a series that -#' varies by that much, below the variance floor of `1e-15` at which the -#' autocorrelation kernels stop estimating. In both cases there is nothing to -#' estimate: the inclusion probability is at 1 or 0 to numerical precision, -#' its Monte Carlo error is smaller than any printed digit, and the verdict -#' lies beyond any evidence threshold. A difference only slightly less certain, with -#' a removal probability of `1e-8` say, has a varying series and shows numbers. +#' chain; a difference the data rule out as strongly gives a series constant at 0. A +#' series that varies by less than about `3e-8` (a variance below `1e-15`) is +#' below the floor at which the ESS kernel stops estimating and is treated the +#' same way. In both cases the inclusion probability stayed within `3e-8` of 1 or 0 +#' throughout the run, so there is no Monte Carlo error at any printed digit +#' and the verdict lies far beyond any evidence threshold; a difference whose series +#' varies by more than that shows numbers. A series constant at a value away +#' from 0 and 1 would give the same blanks, but only a sampler that never moved +#' the indicator in any chain produces it. #' Main-effect difference rows are `NA` throughout when #' `main_difference_selection = FALSE`: their indicators were never sampled. #' diff --git a/R/methods_bgms.R b/R/methods_bgms.R index c39a712a..d18e0bd4 100644 --- a/R/methods_bgms.R +++ b/R/methods_bgms.R @@ -87,23 +87,26 @@ print.bgms = function(x, ...) { #' size and split-R-hat. At every update of an edge indicator the sampler #' proposes to flip it and records the probability that the edge is in the #' model after the move: the acceptance probability \eqn{\alpha} of adding it -#' when it is out, \eqn{1 - \alpha} of removing it when it is in. The +#' when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it +#' is in. The #' inclusion probability is the average of those recorded values, and `mcse`, #' `n_eff` and `Rhat` are computed from their series. #' -#' Those three cells are `NA`, and print blank, when the series has no -#' variation the Monte Carlo machinery can work on. For an edge the data +#' `mcse` and `n_eff` are `NA`, and print blank, when the series has no +#' variation the Monte Carlo machinery can work on; `Rhat` is `NA` when every +#' chain is exactly constant, and otherwise computed. For an edge the data #' support overwhelmingly, the probability of accepting its removal is below #' about `1e-16` at every update, so \eqn{1 - \alpha} rounds to exactly 1 in #' double precision and the series is constant at 1 in every draw of every -#' chain; an edge the data rule out as strongly gives a series constant at 0. -#' An edge whose removal probability is around `1e-12` gives a series that -#' varies by that much, below the variance floor of `1e-15` at which the -#' autocorrelation kernels stop estimating. In both cases there is nothing to -#' estimate: the inclusion probability is at 1 or 0 to numerical precision, -#' its Monte Carlo error is smaller than any printed digit, and the verdict -#' lies beyond any evidence threshold. An edge only slightly less certain, with -#' a removal probability of `1e-8` say, has a varying series and shows numbers. +#' chain; an edge the data rule out as strongly gives a series constant at 0. A +#' series that varies by less than about `3e-8` (a variance below `1e-15`) is +#' below the floor at which the ESS kernel stops estimating and is treated the +#' same way. In both cases the inclusion probability stayed within `3e-8` of 1 or 0 +#' throughout the run, so there is no Monte Carlo error at any printed digit +#' and the verdict lies far beyond any evidence threshold; an edge whose series +#' varies by more than that shows numbers. A series constant at a value away +#' from 0 and 1 would give the same blanks, but only a sampler that never moved +#' the indicator in any chain produces it. #' #' @param object An object of class `bgms`. #' @param ... Currently ignored. diff --git a/R/prior_sensitivity.R b/R/prior_sensitivity.R index 77f98a8a..2eeef221 100644 --- a/R/prior_sensitivity.R +++ b/R/prior_sensitivity.R @@ -83,12 +83,13 @@ verdict_from_bf = function(bf, threshold) { #' continuous parameters and over the indicators, and for NUTS the smallest #' E-BFMI and the largest energy variance ratio, are in \code{$grid}. The #' classification uses every fit; the printed report names a fit whose largest -#' split-\eqn{\hat R} exceeds 1.01, and states when a fit's sampler did not -#' move: a parameter that kept a single value throughout every chain, at values -#' that differ between chains, has an infinite split-\eqn{\hat R}, and the -#' verdicts of that fit do not describe the posterior distribution at that -#' scale. Such a refit needs a longer warmup or a cold start; an ordinal refit -#' starts from the original fit's final state, so \code{warmup} is the lever. +#' split-\eqn{\hat R} exceeds 1.01, and states when a fit has a parameter +#' with an infinite split-\eqn{\hat R}: constant within every split chain, at +#' values that differ between split chains, so the sampler did not explore it, +#' and the verdicts of that fit do not describe the posterior distribution at +#' that scale. Such a refit may need a longer warmup or a cold start; an +#' ordinal refit starts from the original fit's final state, so \code{warmup} +#' is the lever to try first. #' #' \strong{Models with a precision diagonal.} A continuous or mixed fit has a #' Gamma or exponential prior on the precision diagonal with a standardized @@ -681,9 +682,9 @@ order_upper_tri_rowmajor = function(p) { #' Prints the verdict counts at each scale, the number of edges whose verdict #' held at every scale, changed within Monte Carlo error, changed beyond it, #' or could change on a rerun of the original fit, the edges that changed -#' beyond Monte Carlo error by name, the fits whose sampler did not move (a -#' parameter constant throughout every chain at values that differ between -#' chains, infinite split R-hat), the fits whose largest split R-hat exceeds +#' beyond Monte Carlo error by name, the fits with a parameter the sampler did +#' not explore (constant within every split chain at values that differ +#' between split chains, infinite split R-hat), the fits whose largest split R-hat exceeds #' 1.01, and how the chosen scale compares with the scale the estimated #' interactions imply. #' @@ -848,9 +849,9 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { one = length(stuck) == 1L wrap(sprintf( paste0( - "In the %s %s the sampler did not move: at least one parameter kept ", - "a single value throughout every chain, and those values differ ", - "between chains (split R-hat infinite). The verdicts of %s do not ", + "In the %s %s at least one parameter was constant within every split ", + "chain, at values that differ between split chains (split R-hat ", + "infinite): the sampler did not explore it. The verdicts of %s do not ", "describe the posterior distribution at %s." ), list_of(fit_label(stuck)), if(one) "fit" else "fits", diff --git a/man/print.bgms_prior_sensitivity.Rd b/man/print.bgms_prior_sensitivity.Rd index c6fefb94..1744e4ec 100644 --- a/man/print.bgms_prior_sensitivity.Rd +++ b/man/print.bgms_prior_sensitivity.Rd @@ -22,9 +22,9 @@ changed beyond Monte Carlo error; the rest are counted and left to Prints the verdict counts at each scale, the number of edges whose verdict held at every scale, changed within Monte Carlo error, changed beyond it, or could change on a rerun of the original fit, the edges that changed -beyond Monte Carlo error by name, the fits whose sampler did not move (a -parameter constant throughout every chain at values that differ between -chains, infinite split R-hat), the fits whose largest split R-hat exceeds +beyond Monte Carlo error by name, the fits with a parameter the sampler did +not explore (constant within every split chain at values that differ +between split chains, infinite split R-hat), the fits whose largest split R-hat exceeds 1.01, and how the chosen scale compares with the scale the estimated interactions imply. } diff --git a/man/prior_sensitivity_check.Rd b/man/prior_sensitivity_check.Rd index af27fe50..cf63f74d 100644 --- a/man/prior_sensitivity_check.Rd +++ b/man/prior_sensitivity_check.Rd @@ -155,12 +155,13 @@ summary, not the indicators), smallest effective sample size over the continuous parameters and over the indicators, and for NUTS the smallest E-BFMI and the largest energy variance ratio, are in \code{$grid}. The classification uses every fit; the printed report names a fit whose largest -split-\eqn{\hat R} exceeds 1.01, and states when a fit's sampler did not -move: a parameter that kept a single value throughout every chain, at values -that differ between chains, has an infinite split-\eqn{\hat R}, and the -verdicts of that fit do not describe the posterior distribution at that -scale. Such a refit needs a longer warmup or a cold start; an ordinal refit -starts from the original fit's final state, so \code{warmup} is the lever. +split-\eqn{\hat R} exceeds 1.01, and states when a fit has a parameter +with an infinite split-\eqn{\hat R}: constant within every split chain, at +values that differ between split chains, so the sampler did not explore it, +and the verdicts of that fit do not describe the posterior distribution at +that scale. Such a refit may need a longer warmup or a cold start; an +ordinal refit starts from the original fit's final state, so \code{warmup} +is the lever to try first. \strong{Models with a precision diagonal.} A continuous or mixed fit has a Gamma or exponential prior on the precision diagonal with a standardized diff --git a/man/summary.bgmCompare.Rd b/man/summary.bgmCompare.Rd index f23587c0..46fbb3ab 100644 --- a/man/summary.bgmCompare.Rd +++ b/man/summary.bgmCompare.Rd @@ -45,23 +45,26 @@ inclusion probability with its Monte Carlo standard error, effective sample size and split-R-hat. At every update of a difference indicator the sampler proposes to flip it and records the probability that the difference is in the model after the move: the acceptance probability \eqn{\alpha} of adding it -when it is out, \eqn{1 - \alpha} of removing it when it is in. The +when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it +is in. The inclusion probability is the average of those recorded values, and \code{mcse}, \code{n_eff} and \code{Rhat} are computed from their series. -Those three cells are \code{NA}, and print blank, when the series has no -variation the Monte Carlo machinery can work on. For a difference the data +\code{mcse} and \code{n_eff} are \code{NA}, and print blank, when the series has no +variation the Monte Carlo machinery can work on; \code{Rhat} is \code{NA} when every +chain is exactly constant, and otherwise computed. For a difference the data support overwhelmingly, the probability of accepting its removal is below about \code{1e-16} at every update, so \eqn{1 - \alpha} rounds to exactly 1 in double precision and the series is constant at 1 in every draw of every -chain; a difference the data rule out as strongly gives a series constant at 0. -A difference whose removal probability is around \code{1e-12} gives a series that -varies by that much, below the variance floor of \code{1e-15} at which the -autocorrelation kernels stop estimating. In both cases there is nothing to -estimate: the inclusion probability is at 1 or 0 to numerical precision, -its Monte Carlo error is smaller than any printed digit, and the verdict -lies beyond any evidence threshold. A difference only slightly less certain, with -a removal probability of \code{1e-8} say, has a varying series and shows numbers. +chain; a difference the data rule out as strongly gives a series constant at 0. A +series that varies by less than about \code{3e-8} (a variance below \code{1e-15}) is +below the floor at which the ESS kernel stops estimating and is treated the +same way. In both cases the inclusion probability stayed within \code{3e-8} of 1 or 0 +throughout the run, so there is no Monte Carlo error at any printed digit +and the verdict lies far beyond any evidence threshold; a difference whose series +varies by more than that shows numbers. A series constant at a value away +from 0 and 1 would give the same blanks, but only a sampler that never moved +the indicator in any chain produces it. Main-effect difference rows are \code{NA} throughout when \code{main_difference_selection = FALSE}: their indicators were never sampled. } diff --git a/man/summary.bgms.Rd b/man/summary.bgms.Rd index 0525c3f8..7bd851f8 100644 --- a/man/summary.bgms.Rd +++ b/man/summary.bgms.Rd @@ -23,23 +23,26 @@ inclusion probability with its Monte Carlo standard error, effective sample size and split-R-hat. At every update of an edge indicator the sampler proposes to flip it and records the probability that the edge is in the model after the move: the acceptance probability \eqn{\alpha} of adding it -when it is out, \eqn{1 - \alpha} of removing it when it is in. The +when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it +is in. The inclusion probability is the average of those recorded values, and \code{mcse}, \code{n_eff} and \code{Rhat} are computed from their series. -Those three cells are \code{NA}, and print blank, when the series has no -variation the Monte Carlo machinery can work on. For an edge the data +\code{mcse} and \code{n_eff} are \code{NA}, and print blank, when the series has no +variation the Monte Carlo machinery can work on; \code{Rhat} is \code{NA} when every +chain is exactly constant, and otherwise computed. For an edge the data support overwhelmingly, the probability of accepting its removal is below about \code{1e-16} at every update, so \eqn{1 - \alpha} rounds to exactly 1 in double precision and the series is constant at 1 in every draw of every -chain; an edge the data rule out as strongly gives a series constant at 0. -An edge whose removal probability is around \code{1e-12} gives a series that -varies by that much, below the variance floor of \code{1e-15} at which the -autocorrelation kernels stop estimating. In both cases there is nothing to -estimate: the inclusion probability is at 1 or 0 to numerical precision, -its Monte Carlo error is smaller than any printed digit, and the verdict -lies beyond any evidence threshold. An edge only slightly less certain, with -a removal probability of \code{1e-8} say, has a varying series and shows numbers. +chain; an edge the data rule out as strongly gives a series constant at 0. A +series that varies by less than about \code{3e-8} (a variance below \code{1e-15}) is +below the floor at which the ESS kernel stops estimating and is treated the +same way. In both cases the inclusion probability stayed within \code{3e-8} of 1 or 0 +throughout the run, so there is no Monte Carlo error at any printed digit +and the verdict lies far beyond any evidence threshold; an edge whose series +varies by more than that shows numbers. A series constant at a value away +from 0 and 1 would give the same blanks, but only a sampler that never moved +the indicator in any chain produces it. } \examples{ \donttest{ diff --git a/src/mcmc_diagnostics.cpp b/src/mcmc_diagnostics.cpp index 543f0fb8..cd562dbd 100644 --- a/src/mcmc_diagnostics.cpp +++ b/src/mcmc_diagnostics.cpp @@ -223,32 +223,20 @@ struct RhatWorker : public RcppParallel::Worker { } B *= n / (m - 1.0); - // A sub-chain counts as constant when its variance is below the same - // floor the ESS kernel applies to a chain's lag-0 autocovariance - // (biased variance below 1e-15), so a series the ESS kernel finds - // constant to numerical precision has no R-hat either. - bool all_constant = true; - for(int c = 0; c < m; c++) { - if(chain_var[c] * (n - 1.0) / n >= 1e-15) { - all_constant = false; - break; - } - } - // --- Classic split-Rhat (Gelman et al. 2013 / Stan) --- // var_plus = (n-1)/n * W + B/n; Rhat = sqrt(var_plus / W). - if(!all_constant) { + if(W > 0.0) { double var_plus = (n - 1.0) / n * W + B / n; rhat[j] = std::sqrt(var_plus / W); } else { - // Every sub-chain is individually constant. - // The chain means agree to the same floor -> all sub-chains hold the - // SAME constant; Rhat is undefined (0/0), report NA. - // They differ -> sub-chains are constant at DIFFERENT values (chains - // stuck in different states); the between/within ratio - // diverges, so report +Inf as a loud non-convergence alarm - // rather than the silent NA that would hide stuck chains. - rhat[j] = (B / n >= 1e-15) ? R_PosInf : NA_REAL; + // W == 0: every sub-chain is individually constant. + // B == 0 -> all sub-chains hold the SAME constant; Rhat is undefined + // (0/0), report NA. + // B > 0 -> sub-chains are constant at DIFFERENT values (chains stuck + // in different states); the between/within ratio diverges, so + // report +Inf as a loud non-convergence alarm rather than the + // silent NA that would hide genuinely stuck chains. + rhat[j] = (B > 0.0) ? R_PosInf : NA_REAL; } } } diff --git a/tests/testthat/test-mcmc-diagnostics.R b/tests/testthat/test-mcmc-diagnostics.R index 0ee2b729..ca924256 100644 --- a/tests/testthat/test-mcmc-diagnostics.R +++ b/tests/testthat/test-mcmc-diagnostics.R @@ -375,22 +375,6 @@ test_that("all-constant-and-equal sub-chains give NA", { expect_true(is.na(rhat)) }) -test_that("sub-chains constant to numerical precision at one value give NA, at different values Inf", { - # The R-hat kernel applies the ESS kernel's variance floor (1e-15), so a - # series the ESS kernel finds constant has no R-hat either, and a Rao- - # Blackwellized inclusion series at 1 - 1e-12 gets NA in all three cells. - set.seed(11) - near_one = array(1 - abs(rnorm(400, sd = 1e-12)), dim = c(100, 4, 1)) - expect_true(is.na(bgms:::.compute_ess_cpp(near_one))) - expect_true(is.na(bgms:::.compute_rhat_cpp(bgms:::split_chains(near_one)))) - apart = near_one - apart[, 3:4, 1] = abs(rnorm(200, sd = 1e-12)) - expect_identical(bgms:::.compute_rhat_cpp(bgms:::split_chains(apart)), Inf) - # A variance at the floor is not constant: the classic formula applies. - varying = array(1 + rnorm(400, sd = 1e-6), dim = c(100, 4, 1)) - expect_true(is.finite(bgms:::.compute_rhat_cpp(bgms:::split_chains(varying)))) -}) - test_that("constant-but-unequal sub-chains give +Inf, not NA", { # Two chains stuck at 0, two stuck at 1: W == 0 but B > 0. Silence (NA) here # is the worst failure mode, so the estimator must raise a loud +Inf alarm. diff --git a/tests/testthat/test-prior-sensitivity.R b/tests/testthat/test-prior-sensitivity.R index 3bd16649..ad166428 100644 --- a/tests/testthat/test-prior-sensitivity.R +++ b/tests/testthat/test-prior-sensitivity.R @@ -116,10 +116,8 @@ test_that("the check's flag on a fit is verdicts()' flag on that fit", { ps = prior_sensitivity_check(fit, scale_multipliers = 2, seed = 3) v = verdicts(fit) expect_identical(ps$edges$fragile, v$fragile[match(ps$edges$edge, v$parameter)]) - # saturated follows the kernel: exactly the edges whose chosen-scale MCSE is NA. + # saturated is the edges whose chosen-scale MCSE is NA. expect_identical(ps$edges$saturated, is.na(ps$edges$chosen_scale_mcse)) - ind = summary(fit)$indicator - expect_identical(unname(is.na(ind$mcse)), unname(is.na(ind$Rhat))) }) @@ -236,7 +234,7 @@ test_that("print() reports the counts, the movers and high R-hat fits", { # out of the numeric R-hat line. ps3 = fake_sensitivity(cls, multipliers, rhat_max = c(1.03, 1.005, Inf), s_hat = 0.8) out3 = gsub("\\s+", " ", paste(utils::capture.output(print(ps3)), collapse = " ")) - expect_match(out3, "In the 2x fit the sampler did not move: at least one parameter kept a single value throughout every chain, and those values differ between chains \\(split R-hat infinite\\)") + expect_match(out3, "In the 2x fit at least one parameter was constant within every split chain, at values that differ between split chains \\(split R-hat infinite\\): the sampler did not explore it") expect_match(out3, "The verdicts of that fit do not describe the posterior distribution at that scale") expect_match(out3, "Largest split R-hat above 1.01: 0.5x fit 1.03\\.") expect_false(grepl("Monte Carlo standard errors in $edges", out3, fixed = TRUE)) @@ -248,7 +246,7 @@ test_that("print() reports the counts, the movers and high R-hat fits", { expect_match(out5, "Monte Carlo standard errors in \\$edges are NA where a fit's inclusion probability was 0 or 1 in every draw of every chain, or constant to numerical precision") ps4 = fake_sensitivity(cls, multipliers, rhat_max = c(Inf, 1.005, Inf), s_hat = 0.8) out4 = gsub("\\s+", " ", paste(utils::capture.output(print(ps4)), collapse = " ")) - expect_match(out4, "In the 0.5x and 2x fits the sampler did not move") + expect_match(out4, "In the 0.5x and 2x fits at least one parameter was constant within every split chain") expect_match(out4, "The verdicts of those fits do not describe the posterior distribution at those scales") expect_false(grepl("Largest split R-hat", out4, fixed = TRUE)) }) diff --git a/vignettes/diagnostics.Rmd b/vignettes/diagnostics.Rmd index 7a80918f..b7e58664 100644 --- a/vignettes/diagnostics.Rmd +++ b/vignettes/diagnostics.Rmd @@ -88,21 +88,22 @@ summary(fit)$indicator Rao-Blackwellized inclusion probability. At every update of an edge indicator the sampler proposes to flip it and records the probability that the edge is in the model after the move: the acceptance probability of - adding it when it is out, one minus the acceptance probability of removing - it when it is in. That recorded value is continuous, so the standard Monte - Carlo machinery applies to its series. All three of `mcse`, `n_eff` and - `Rhat` are `NA` when the series has no variation the machinery can work on. - For an edge the data support overwhelmingly, the probability of accepting - its removal is below about 1e-16 at every update, so one minus it rounds to - exactly 1 and the series is constant at 1 in every draw of every chain; an - edge the data rule out as strongly is constant at 0. An edge whose removal - probability is around 1e-12 gives a series that varies by that much, below - the variance floor of 1e-15 at which the ESS and R-hat kernels stop - estimating, so `sd` prints as a number like `2e-11` while the three cells - are blank. In both cases there is nothing to estimate: the inclusion - probability is at 1 or 0 to numerical precision and the verdict lies beyond - any evidence threshold. An edge only slightly less certain, with a removal - probability of 1e-8 say, has a varying series and shows numbers. + adding it when it is out, the probability of retaining it (one minus the + acceptance probability of removing it) when it is in. That recorded value + is continuous, so the standard Monte Carlo machinery applies to its series. + `mcse` and `n_eff` are `NA` when the series has no variation the machinery + can work on; `Rhat` is `NA` when every chain is exactly constant, and + otherwise computed. For an edge the data support overwhelmingly, the + probability of accepting its removal is below about 1e-16 at every update, + so one minus it rounds to exactly 1 and the series is constant at 1 in + every draw of every chain; an edge the data rule out as strongly is + constant at 0. A series that varies by less than about 3e-8 (a variance + below 1e-15) is below the floor at which the ESS kernel stops estimating + and is treated the same way; `sd` then prints as a number like `2e-11` + while `mcse` and `n_eff` are blank. In both cases the inclusion probability + stayed within 3e-8 of 1 or 0 throughout the run, so there is no Monte Carlo + error at any printed digit and the verdict lies far beyond any evidence + threshold; an edge whose series varies by more than that shows numbers. ## Traceplots From 3e349a135280ceef2680c08d85af993d006fd752 Mon Sep 17 00:00:00 2001 From: Maarten Marsman Date: Sat, 7 Nov 2026 09:52:26 +0100 Subject: [PATCH 3/3] fix(diagnostics): clarify sensitivity counts and convergence guidance --- NEWS.md | 45 ++++-------- R/methods_bgmcompare.R | 44 ++++------- R/methods_bgms.R | 39 +++------- R/prior_sensitivity.R | 94 ++++++++++-------------- R/refit_engine.R | 5 +- man/print.bgms_prior_sensitivity.Rd | 12 +-- man/prior_sensitivity_check.Rd | 54 ++++++-------- man/summary.bgmCompare.Rd | 39 ++++------ man/summary.bgms.Rd | 34 +++------ tests/testthat/_snaps/summary-marking.md | 5 +- tests/testthat/test-mcmc-diagnostics.R | 18 +++++ tests/testthat/test-prior-sensitivity.R | 19 +++-- vignettes/diagnostics.Rmd | 30 +++----- vignettes/prior-sensitivity.Rmd | 2 + 14 files changed, 176 insertions(+), 264 deletions(-) diff --git a/NEWS.md b/NEWS.md index 18906f58..bd5f0312 100644 --- a/NEWS.md +++ b/NEWS.md @@ -165,37 +165,20 @@ released from this line yet. ## Diagnostics -* The fourth count of the printed `prior_sensitivity_check()` report, "could - change on a rerun", is `verdicts()`' near-threshold flag on the original fit, - computed by the same function, plus the edges whose verdict change could not - be compared with a Monte Carlo error (a refit with an `NA` error); each part - has its own sentence, and the counts agree whenever no such edge exists. The - between-chain band and the per-chain verdict test that the check applied on - top of it are gone, as are the `$edges` columns `insufficient`, - `insufficient_noisy` and `insufficient_disagree`; the flag is the `fragile` - column. The other three count labels lose their commas. The sentence under - the counts names the flagged edges and states that their log Bayes factors - lie within two standard errors of an evidence threshold. - -* The `saturated` column of `prior_sensitivity_check()` marks the edges whose - chosen-scale Monte Carlo standard error is `NA`: the original fit's - Rao-Blackwellized inclusion series had no variation the ESS kernel could use, - whether exactly constant or constant to numerical precision. Before, it - marked only the edges at exactly 0 or 1 in every chain. The one-sentence note - under the summary tables and in the check now says "0 or 1 in every draw of - every chain, or constant to numerical precision", and the Details of `?summary.bgms`, - `?summary.bgmCompare` and `vignette("diagnostics")` explain why: the - recorded inclusion probability is one minus the acceptance probability of - removing the edge, which rounds to 1 once that probability is below about - `1e-16`, or varies below the kernels' floor when it is around `1e-12`. - -* The sentence `prior_sensitivity_check()` prints for a fit with an infinite - split-R-hat says that the sampler did not move, in which fit, and that the - fit's verdicts do not describe the posterior distribution at that scale. The - `$grid` documentation states that `rhat_max` covers the continuous - parameters only, and the Details say what such a refit needs. The `Refits:` - line prints NUTS in capitals. `?verdicts` states that an indicator whose - standard errors are undefined is not flagged. +* `prior_sensitivity_check()` now uses the original fit's `verdicts()` flag + in its `fragile` column, replacing `insufficient` and its two subcause + columns. The fourth printed count, "could change on a rerun", includes + flagged edges and changes that could not be assessed because a Monte Carlo + error was unavailable. The report gives the number of these additional edges. + +* The sensitivity check's `saturated` column now indicates an unavailable + chosen-scale Monte Carlo standard error. It no longer requires inclusion + probabilities of exactly 0 or 1. The summary notes explain that blank + diagnostics could not be estimated; they do not imply zero Monte Carlo error. + +* Sensitivity reports identify fits with infinite split-R-hat and advise + checking convergence before interpreting their verdicts. The R-hat + calculation is unchanged. The `Refits:` line now capitalizes NUTS. * The pairwise summary table for a `bgm()` fit and the difference summary tables for a `bgmCompare()` fit no longer include `share_incl`, the proportion diff --git a/R/methods_bgmcompare.R b/R/methods_bgmcompare.R index f04376b5..77f108f9 100644 --- a/R/methods_bgmcompare.R +++ b/R/methods_bgmcompare.R @@ -239,33 +239,20 @@ print.bgmCompare = function(x, ...) { #' are exempt from the union recode and are never marked). Fits made before #' that field existed print unmarked. #' -#' The inclusion table (`$indicator`) reports, per difference indicator, the Rao-Blackwellized -#' inclusion probability with its Monte Carlo standard error, effective sample -#' size and split-R-hat. At every update of a difference indicator the sampler -#' proposes to flip it and records the probability that the difference is in the -#' model after the move: the acceptance probability \eqn{\alpha} of adding it -#' when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it -#' is in. The -#' inclusion probability is the average of those recorded values, and `mcse`, -#' `n_eff` and `Rhat` are computed from their series. +#' The inclusion table (`$indicator`) reports the Rao-Blackwellized inclusion +#' probability and its diagnostics for each difference. At each update the sampler +#' records the probability of adding the difference if absent, or retaining it if +#' present. The table summarizes these recorded probabilities. #' -#' `mcse` and `n_eff` are `NA`, and print blank, when the series has no -#' variation the Monte Carlo machinery can work on; `Rhat` is `NA` when every -#' chain is exactly constant, and otherwise computed. For a difference the data -#' support overwhelmingly, the probability of accepting its removal is below -#' about `1e-16` at every update, so \eqn{1 - \alpha} rounds to exactly 1 in -#' double precision and the series is constant at 1 in every draw of every -#' chain; a difference the data rule out as strongly gives a series constant at 0. A -#' series that varies by less than about `3e-8` (a variance below `1e-15`) is -#' below the floor at which the ESS kernel stops estimating and is treated the -#' same way. In both cases the inclusion probability stayed within `3e-8` of 1 or 0 -#' throughout the run, so there is no Monte Carlo error at any printed digit -#' and the verdict lies far beyond any evidence threshold; a difference whose series -#' varies by more than that shows numbers. A series constant at a value away -#' from 0 and 1 would give the same blanks, but only a sampler that never moved -#' the indicator in any chain produces it. -#' Main-effect difference rows are `NA` throughout when -#' `main_difference_selection = FALSE`: their indicators were never sampled. +#' Blank diagnostic cells contain `NA`: the diagnostic could not be estimated. +#' `mcse` and `n_eff` can be unavailable when the recorded probabilities vary +#' too little, often near 0 or 1. This does not establish that Monte Carlo +#' error is zero. Split-R-hat uses a different calculation: identical constant +#' split chains give `NA`, while split chains constant at different values give +#' `Inf`. It can remain finite when `mcse` and `n_eff` are unavailable. +#' +#' Main-effect difference rows are blank when `main_difference_selection = FALSE` +#' because those indicators were not sampled. #' #' @param object An object of class `bgmCompare`. #' @param ... Currently ignored. @@ -379,9 +366,8 @@ print.summary.bgmCompare = function(x, digits = 3, ...) { cat("... (use `summary(fit)$indicator` to see full output)\n") } if(ind_has_na) { - cat("Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion\n") - cat("probability was 0 or 1 in every draw of every chain, or constant to numerical\n") - cat("precision; there is no variation to estimate them from.\n") + cat("Note: blank diagnostic cells mean these values could not be estimated\n") + cat("from the draws.\n") if(isFALSE(x$arguments$main_difference_selection)) { cat("Main-effect difference rows are blank when main_difference_selection = FALSE\n") cat("left their indicators unsampled.\n") diff --git a/R/methods_bgms.R b/R/methods_bgms.R index d18e0bd4..28ef45de 100644 --- a/R/methods_bgms.R +++ b/R/methods_bgms.R @@ -82,31 +82,17 @@ print.bgms = function(x, ...) { #' @description Returns posterior summaries and diagnostics for a fitted `bgms` model. #' #' @details -#' The inclusion table (`$indicator`) reports, per edge, the Rao-Blackwellized -#' inclusion probability with its Monte Carlo standard error, effective sample -#' size and split-R-hat. At every update of an edge indicator the sampler -#' proposes to flip it and records the probability that the edge is in the -#' model after the move: the acceptance probability \eqn{\alpha} of adding it -#' when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it -#' is in. The -#' inclusion probability is the average of those recorded values, and `mcse`, -#' `n_eff` and `Rhat` are computed from their series. +#' The inclusion table (`$indicator`) reports the Rao-Blackwellized inclusion +#' probability and its diagnostics for each edge. At each update the sampler +#' records the probability of adding the edge if absent, or retaining it if +#' present. The table summarizes these recorded probabilities. #' -#' `mcse` and `n_eff` are `NA`, and print blank, when the series has no -#' variation the Monte Carlo machinery can work on; `Rhat` is `NA` when every -#' chain is exactly constant, and otherwise computed. For an edge the data -#' support overwhelmingly, the probability of accepting its removal is below -#' about `1e-16` at every update, so \eqn{1 - \alpha} rounds to exactly 1 in -#' double precision and the series is constant at 1 in every draw of every -#' chain; an edge the data rule out as strongly gives a series constant at 0. A -#' series that varies by less than about `3e-8` (a variance below `1e-15`) is -#' below the floor at which the ESS kernel stops estimating and is treated the -#' same way. In both cases the inclusion probability stayed within `3e-8` of 1 or 0 -#' throughout the run, so there is no Monte Carlo error at any printed digit -#' and the verdict lies far beyond any evidence threshold; an edge whose series -#' varies by more than that shows numbers. A series constant at a value away -#' from 0 and 1 would give the same blanks, but only a sampler that never moved -#' the indicator in any chain produces it. +#' Blank diagnostic cells contain `NA`: the diagnostic could not be estimated. +#' `mcse` and `n_eff` can be unavailable when the recorded probabilities vary +#' too little, often near 0 or 1. This does not establish that Monte Carlo +#' error is zero. Split-R-hat uses a different calculation: identical constant +#' split chains give `NA`, while split chains constant at different values give +#' `Inf`. It can remain finite when `mcse` and `n_eff` are unavailable. #' #' @param object An object of class `bgms`. #' @param ... Currently ignored. @@ -213,9 +199,8 @@ print.summary.bgms = function(x, digits = 3, ...) { print(ind) if(nrow(x$indicator) > .summary_preview_rows) cat("... (use `summary(fit)$indicator` to see full output)\n") if(ind_has_na) { - cat("Note: blank mcse, n_eff and Rhat cells mark edges whose inclusion probability\n") - cat("was 0 or 1 in every draw of every chain, or constant to numerical precision;\n") - cat("there is no variation to estimate them from.\n") + cat("Note: blank diagnostic cells mean these values could not be estimated\n") + cat("from the draws.\n") } cat("\n") } diff --git a/R/prior_sensitivity.R b/R/prior_sensitivity.R index 2eeef221..6e413c58 100644 --- a/R/prior_sensitivity.R +++ b/R/prior_sensitivity.R @@ -50,7 +50,8 @@ verdict_from_bf = function(bf, threshold) { #' are given. The original fit is the \code{1x} fit and is not refit. An #' ordinal (omrf) refit starts each chain from the original fit's final #' per-chain state, and a NUTS refit also from its adapted step size and mass -#' matrix; continuous (GGM) and mixed-MRF refits start cold. Every quantity +#' matrix; continuous (GGM) and mixed-MRF refits use default initial values. +#' Every quantity #' the check reports for a fit is that fit's own Rao-Blackwellized inclusion #' probability, its natural log inclusion Bayes factor and verdict, and the #' Monte Carlo standard error of the log Bayes factor from the fit's own @@ -68,28 +69,22 @@ verdict_from_bf = function(bf, threshold) { #' smallest and largest multiplier in the contiguous run of fits around #' \code{1x} that hold the reference verdict. #' -#' \strong{The chosen-scale flag.} An edge is \code{fragile} when -#' \code{\link{verdicts}()} flags it in the original fit: its log Bayes factor -#' lies within two standard errors of a verdict threshold, by either of the two -#' standard errors \code{verdicts()} computes. The verdict the check starts -#' from could then change on a rerun with another seed, so the printed report -#' counts the edge among the verdicts that could change on a rerun rather than -#' in a mover category. An edge whose standard errors are undefined is not -#' flagged, as in \code{verdicts()}. +#' \strong{The chosen-scale flag.} The \code{fragile} column is the original +#' fit's flag from \code{\link{verdicts}()}: a log Bayes factor lies within +#' two standard errors of an evidence threshold. The fourth printed count, +#' "could change on a rerun", includes these edges and any other edge whose +#' verdict changed but whose Monte Carlo error was unavailable. The report +#' states how many of those additional edges there are. An edge whose standard +#' errors are both undefined is not flagged by \code{verdicts()}. #' -#' \strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the -#' continuous parameters (the main-effect and pairwise blocks of the fit -#' summary, not the indicators), smallest effective sample size over the -#' continuous parameters and over the indicators, and for NUTS the smallest -#' E-BFMI and the largest energy variance ratio, are in \code{$grid}. The -#' classification uses every fit; the printed report names a fit whose largest -#' split-\eqn{\hat R} exceeds 1.01, and states when a fit has a parameter -#' with an infinite split-\eqn{\hat R}: constant within every split chain, at -#' values that differ between split chains, so the sampler did not explore it, -#' and the verdicts of that fit do not describe the posterior distribution at -#' that scale. Such a refit may need a longer warmup or a cold start; an -#' ordinal refit starts from the original fit's final state, so \code{warmup} -#' is the lever to try first. +#' \strong{Convergence.} The \code{grid} table contains each fit's diagnostics. +#' Its \code{rhat_max} covers the main-effect and pairwise parameter blocks, +#' excluding indicators. All fits enter the classification, including those +#' with convergence problems. The printed report identifies fits with +#' split-\eqn{\hat R} above 1.01 or infinite. Check convergence before +#' interpreting their verdicts. For a refit, you can try increasing +#' \code{warmup} in \code{prior_sensitivity_check()}; for the original fit, +#' rerun \code{bgm()} or \code{bgmCompare()} with more warmup iterations. #' #' \strong{Models with a precision diagonal.} A continuous or mixed fit has a #' Gamma or exponential prior on the precision diagonal with a standardized @@ -160,15 +155,12 @@ verdict_from_bf = function(bf, threshold) { #' factor) and \code{chosen_scale_verdict}; \code{stability_lower} and #' \code{stability_upper}; \code{mover} (\code{stable}, #' \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or -#' \code{NA} for an indicator without a verdict) and \code{moved_at}; +#' \code{NA} when a verdict or the error needed to assess its change is +#' unavailable) and \code{moved_at}; #' \code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the -#' original fit); \code{saturated} (the original fit's Rao-Blackwellized -#' inclusion draws have no variation the ESS kernel can use, because the -#' acceptance probability of flipping the indicator was below double -#' precision at every update: the inclusion probability is 0 or 1 in every -#' draw of every chain, or constant to numerical precision; -#' \code{chosen_scale_mcse} is \code{NA} there; see the Details of -#' \code{\link{summary.bgms}}); +#' original fit); \code{saturated} (\code{chosen_scale_mcse} is +#' \code{NA}; this indicates an unavailable error estimate, not necessarily +#' an inclusion probability of exactly 0 or 1); #' then one \code{verdict_x}, \code{log_bf_x} and \code{mcse_x} #' column per multiplier \code{m}, from that fit's own statistics.} #' \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale}, @@ -493,7 +485,7 @@ prior_sensitivity_engine = function(bgms_object, fit_stats = lapply(fits, refit_edge_stats, evidence_threshold = evidence_threshold) # The chosen-scale flag is verdicts()' near-threshold flag on the original - # fit, so the two reports count the same edges. + # fit, so both reports flag the same edges. v0 = verdicts(bgms_object, evidence_threshold = evidence_threshold) fit_stats[[s0_idx]]$fragile = v0$fragile[match(fit_stats[[s0_idx]]$edge, v0$parameter)] diag_rows = lapply(fits, refit_diagnostics_row) @@ -679,14 +671,10 @@ order_upper_tri_rowmajor = function(p) { #' @title Print a Prior Sensitivity Check #' #' @description -#' Prints the verdict counts at each scale, the number of edges whose verdict -#' held at every scale, changed within Monte Carlo error, changed beyond it, -#' or could change on a rerun of the original fit, the edges that changed -#' beyond Monte Carlo error by name, the fits with a parameter the sampler did -#' not explore (constant within every split chain at values that differ -#' between split chains, infinite split R-hat), the fits whose largest split R-hat exceeds -#' 1.01, and how the chosen scale compares with the scale the estimated -#' interactions imply. +#' Prints verdict counts by scale, the four sensitivity counts, and the edges +#' whose verdicts changed beyond Monte Carlo error or could change on a rerun. +#' Also reports fits with high or infinite split-R-hat and compares the chosen +#' slab scale with the estimated interaction sizes. #' #' @param x A \code{bgms_prior_sensitivity} object. #' @param max_rows Integer. Maximum edges to name in the table of edges that @@ -820,7 +808,11 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { )) } if(any(uncert & !flagged)) { - wrap("Monte Carlo uncertainty could not be assessed for every verdict or verdict change; see $edges.") + n_unknown = sum(uncert & !flagged) + wrap(sprintf( + "The fourth count also includes %d %s whose verdict changed but whose Monte Carlo error was unavailable; see $edges.", + n_unknown, if(n_unknown == 1L) unit$noun else unit$nouns + )) } cat("\n") } @@ -828,17 +820,11 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { # Edges whose Monte Carlo error is NA in some fit. mcse_cols = grep("^mcse_x", names(edges), value = TRUE) if(length(mcse_cols) > 0 && anyNA(edges[, mcse_cols])) { - wrap(paste0( - "Monte Carlo standard errors in $edges are NA where a fit's inclusion ", - "probability was 0 or 1 in every draw of every chain, or constant to ", - "numerical precision; there is no variation to estimate them from." - )) + wrap("NA Monte Carlo standard errors in $edges could not be estimated from the draws.") cat("\n") } - # Fits whose sampler did not move (a parameter constant throughout every - # chain at values that differ between chains), and fits whose largest split - # R-hat exceeds 1.01. + # Report infinite split R-hat separately from finite values above 1.01. g = x$grid if(!is.null(g$rhat_max)) { fit_label = function(i) { @@ -849,13 +835,11 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { one = length(stuck) == 1L wrap(sprintf( paste0( - "In the %s %s at least one parameter was constant within every split ", - "chain, at values that differ between split chains (split R-hat ", - "infinite): the sampler did not explore it. The verdicts of %s do not ", - "describe the posterior distribution at %s." + "The %s %s at least one parameter with infinite split R-hat. ", + "Check convergence before interpreting %s verdicts." ), - list_of(fit_label(stuck)), if(one) "fit" else "fits", - if(one) "that fit" else "those fits", if(one) "that scale" else "those scales" + list_of(fit_label(stuck)), if(one) "fit has" else "fits have", + if(one) "its" else "their" )) cat("\n") } @@ -894,7 +878,7 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { sprintf("%.0f s", x$runtime_seconds) } n_refit = sum(!g$original_fit) - start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "cold starts" + start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "started from default initial values" it = g$iter[!g$original_fit][1] wu = g$warmup[!g$original_fit][1] sampler_label = if(identical(x$refit_sampler, "nuts")) "NUTS" else x$refit_sampler diff --git a/R/refit_engine.R b/R/refit_engine.R index 6d8f6f33..aca9dc8b 100644 --- a/R/refit_engine.R +++ b/R/refit_engine.R @@ -257,9 +257,8 @@ verdict_from_lbf = function(lbf, lthr) { # Per-edge quantities from one refit: the Rao-Blackwellized inclusion # probability (canonical), its natural log inclusion Bayes factor and verdict, # the MCSE of the log BF from the summary table's inclusion MCSE by the delta -# method, and whether the RB draws had no variation the ESS kernel could use -# (saturated: the summary MCSE is NA). Edge order is the native row-major -# upper triangle. +# method, and whether the inclusion MCSE is unavailable (the saturated column). +# Edge order is the native row-major upper triangle. # ------------------------------------------------------------------ refit_edge_stats = function(fit, evidence_threshold) { raw = get_raw_samples(fit) diff --git a/man/print.bgms_prior_sensitivity.Rd b/man/print.bgms_prior_sensitivity.Rd index 1744e4ec..3089b8b6 100644 --- a/man/print.bgms_prior_sensitivity.Rd +++ b/man/print.bgms_prior_sensitivity.Rd @@ -19,14 +19,10 @@ changed beyond Monte Carlo error; the rest are counted and left to \code{x}, invisibly. } \description{ -Prints the verdict counts at each scale, the number of edges whose verdict -held at every scale, changed within Monte Carlo error, changed beyond it, -or could change on a rerun of the original fit, the edges that changed -beyond Monte Carlo error by name, the fits with a parameter the sampler did -not explore (constant within every split chain at values that differ -between split chains, infinite split R-hat), the fits whose largest split R-hat exceeds -1.01, and how the chosen scale compares with the scale the estimated -interactions imply. +Prints verdict counts by scale, the four sensitivity counts, and the edges +whose verdicts changed beyond Monte Carlo error or could change on a rerun. +Also reports fits with high or infinite split-R-hat and compares the chosen +slab scale with the estimated interaction sizes. } \seealso{ \code{\link{prior_sensitivity_check}()} diff --git a/man/prior_sensitivity_check.Rd b/man/prior_sensitivity_check.Rd index cf63f74d..7a8629ca 100644 --- a/man/prior_sensitivity_check.Rd +++ b/man/prior_sensitivity_check.Rd @@ -68,15 +68,12 @@ inclusion probability, the original fit's \code{chosen_scale_pip}, factor) and \code{chosen_scale_verdict}; \code{stability_lower} and \code{stability_upper}; \code{mover} (\code{stable}, \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or -\code{NA} for an indicator without a verdict) and \code{moved_at}; +\code{NA} when a verdict or the error needed to assess its change is +unavailable) and \code{moved_at}; \code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the -original fit); \code{saturated} (the original fit's Rao-Blackwellized -inclusion draws have no variation the ESS kernel can use, because the -acceptance probability of flipping the indicator was below double -precision at every update: the inclusion probability is 0 or 1 in every -draw of every chain, or constant to numerical precision; -\code{chosen_scale_mcse} is \code{NA} there; see the Details of -\code{\link{summary.bgms}}); +original fit); \code{saturated} (\code{chosen_scale_mcse} is +\code{NA}; this indicates an unavailable error estimate, not necessarily +an inclusion probability of exactly 0 or 1); then one \code{verdict_x}, \code{log_bf_x} and \code{mcse_x} column per multiplier \code{m}, from that fit's own statistics.} \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale}, @@ -122,7 +119,8 @@ fit's \code{iter} and \code{warmup} unless \code{iter} and \code{warmup} are given. The original fit is the \code{1x} fit and is not refit. An ordinal (omrf) refit starts each chain from the original fit's final per-chain state, and a NUTS refit also from its adapted step size and mass -matrix; continuous (GGM) and mixed-MRF refits start cold. Every quantity +matrix; continuous (GGM) and mixed-MRF refits use default initial values. +Every quantity the check reports for a fit is that fit's own Rao-Blackwellized inclusion probability, its natural log inclusion Bayes factor and verdict, and the Monte Carlo standard error of the log Bayes factor from the fit's own @@ -140,28 +138,22 @@ verdict differs. \code{stability_lower} and \code{stability_upper} are the smallest and largest multiplier in the contiguous run of fits around \code{1x} that hold the reference verdict. -\strong{The chosen-scale flag.} An edge is \code{fragile} when -\code{\link{verdicts}()} flags it in the original fit: its log Bayes factor -lies within two standard errors of a verdict threshold, by either of the two -standard errors \code{verdicts()} computes. The verdict the check starts -from could then change on a rerun with another seed, so the printed report -counts the edge among the verdicts that could change on a rerun rather than -in a mover category. An edge whose standard errors are undefined is not -flagged, as in \code{verdicts()}. - -\strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the -continuous parameters (the main-effect and pairwise blocks of the fit -summary, not the indicators), smallest effective sample size over the -continuous parameters and over the indicators, and for NUTS the smallest -E-BFMI and the largest energy variance ratio, are in \code{$grid}. The -classification uses every fit; the printed report names a fit whose largest -split-\eqn{\hat R} exceeds 1.01, and states when a fit has a parameter -with an infinite split-\eqn{\hat R}: constant within every split chain, at -values that differ between split chains, so the sampler did not explore it, -and the verdicts of that fit do not describe the posterior distribution at -that scale. Such a refit may need a longer warmup or a cold start; an -ordinal refit starts from the original fit's final state, so \code{warmup} -is the lever to try first. +\strong{The chosen-scale flag.} The \code{fragile} column is the original +fit's flag from \code{\link{verdicts}()}: a log Bayes factor lies within +two standard errors of an evidence threshold. The fourth printed count, +"could change on a rerun", includes these edges and any other edge whose +verdict changed but whose Monte Carlo error was unavailable. The report +states how many of those additional edges there are. An edge whose standard +errors are both undefined is not flagged by \code{verdicts()}. + +\strong{Convergence.} The \code{grid} table contains each fit's diagnostics. +Its \code{rhat_max} covers the main-effect and pairwise parameter blocks, +excluding indicators. All fits enter the classification, including those +with convergence problems. The printed report identifies fits with +split-\eqn{\hat R} above 1.01 or infinite. Check convergence before +interpreting their verdicts. For a refit, you can try increasing +\code{warmup} in \code{prior_sensitivity_check()}; for the original fit, +rerun \code{bgm()} or \code{bgmCompare()} with more warmup iterations. \strong{Models with a precision diagonal.} A continuous or mixed fit has a Gamma or exponential prior on the precision diagonal with a standardized diff --git a/man/summary.bgmCompare.Rd b/man/summary.bgmCompare.Rd index 46fbb3ab..60bfb85f 100644 --- a/man/summary.bgmCompare.Rd +++ b/man/summary.bgmCompare.Rd @@ -40,33 +40,20 @@ observation counts per variable (\code{NULL} for Blume-Capel variables, which are exempt from the union recode and are never marked). Fits made before that field existed print unmarked. -The inclusion table (\verb{$indicator}) reports, per difference indicator, the Rao-Blackwellized -inclusion probability with its Monte Carlo standard error, effective sample -size and split-R-hat. At every update of a difference indicator the sampler -proposes to flip it and records the probability that the difference is in the -model after the move: the acceptance probability \eqn{\alpha} of adding it -when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it -is in. The -inclusion probability is the average of those recorded values, and \code{mcse}, -\code{n_eff} and \code{Rhat} are computed from their series. +The inclusion table (\verb{$indicator}) reports the Rao-Blackwellized inclusion +probability and its diagnostics for each difference. At each update the sampler +records the probability of adding the difference if absent, or retaining it if +present. The table summarizes these recorded probabilities. -\code{mcse} and \code{n_eff} are \code{NA}, and print blank, when the series has no -variation the Monte Carlo machinery can work on; \code{Rhat} is \code{NA} when every -chain is exactly constant, and otherwise computed. For a difference the data -support overwhelmingly, the probability of accepting its removal is below -about \code{1e-16} at every update, so \eqn{1 - \alpha} rounds to exactly 1 in -double precision and the series is constant at 1 in every draw of every -chain; a difference the data rule out as strongly gives a series constant at 0. A -series that varies by less than about \code{3e-8} (a variance below \code{1e-15}) is -below the floor at which the ESS kernel stops estimating and is treated the -same way. In both cases the inclusion probability stayed within \code{3e-8} of 1 or 0 -throughout the run, so there is no Monte Carlo error at any printed digit -and the verdict lies far beyond any evidence threshold; a difference whose series -varies by more than that shows numbers. A series constant at a value away -from 0 and 1 would give the same blanks, but only a sampler that never moved -the indicator in any chain produces it. -Main-effect difference rows are \code{NA} throughout when -\code{main_difference_selection = FALSE}: their indicators were never sampled. +Blank diagnostic cells contain \code{NA}: the diagnostic could not be estimated. +\code{mcse} and \code{n_eff} can be unavailable when the recorded probabilities vary +too little, often near 0 or 1. This does not establish that Monte Carlo +error is zero. Split-R-hat uses a different calculation: identical constant +split chains give \code{NA}, while split chains constant at different values give +\code{Inf}. It can remain finite when \code{mcse} and \code{n_eff} are unavailable. + +Main-effect difference rows are blank when \code{main_difference_selection = FALSE} +because those indicators were not sampled. } \examples{ \donttest{ diff --git a/man/summary.bgms.Rd b/man/summary.bgms.Rd index 7bd851f8..7e013ad2 100644 --- a/man/summary.bgms.Rd +++ b/man/summary.bgms.Rd @@ -18,31 +18,17 @@ An object of class \code{summary.bgms} with posterior summaries. Returns posterior summaries and diagnostics for a fitted \code{bgms} model. } \details{ -The inclusion table (\verb{$indicator}) reports, per edge, the Rao-Blackwellized -inclusion probability with its Monte Carlo standard error, effective sample -size and split-R-hat. At every update of an edge indicator the sampler -proposes to flip it and records the probability that the edge is in the -model after the move: the acceptance probability \eqn{\alpha} of adding it -when it is out, \eqn{1 - \alpha}, the probability of retaining it, when it -is in. The -inclusion probability is the average of those recorded values, and \code{mcse}, -\code{n_eff} and \code{Rhat} are computed from their series. +The inclusion table (\verb{$indicator}) reports the Rao-Blackwellized inclusion +probability and its diagnostics for each edge. At each update the sampler +records the probability of adding the edge if absent, or retaining it if +present. The table summarizes these recorded probabilities. -\code{mcse} and \code{n_eff} are \code{NA}, and print blank, when the series has no -variation the Monte Carlo machinery can work on; \code{Rhat} is \code{NA} when every -chain is exactly constant, and otherwise computed. For an edge the data -support overwhelmingly, the probability of accepting its removal is below -about \code{1e-16} at every update, so \eqn{1 - \alpha} rounds to exactly 1 in -double precision and the series is constant at 1 in every draw of every -chain; an edge the data rule out as strongly gives a series constant at 0. A -series that varies by less than about \code{3e-8} (a variance below \code{1e-15}) is -below the floor at which the ESS kernel stops estimating and is treated the -same way. In both cases the inclusion probability stayed within \code{3e-8} of 1 or 0 -throughout the run, so there is no Monte Carlo error at any printed digit -and the verdict lies far beyond any evidence threshold; an edge whose series -varies by more than that shows numbers. A series constant at a value away -from 0 and 1 would give the same blanks, but only a sampler that never moved -the indicator in any chain produces it. +Blank diagnostic cells contain \code{NA}: the diagnostic could not be estimated. +\code{mcse} and \code{n_eff} can be unavailable when the recorded probabilities vary +too little, often near 0 or 1. This does not establish that Monte Carlo +error is zero. Split-R-hat uses a different calculation: identical constant +split chains give \code{NA}, while split chains constant at different values give +\code{Inf}. It can remain finite when \code{mcse} and \code{n_eff} are unavailable. } \examples{ \donttest{ diff --git a/tests/testthat/_snaps/summary-marking.md b/tests/testthat/_snaps/summary-marking.md index 505a0305..fcc44b9a 100644 --- a/tests/testthat/_snaps/summary-marking.md +++ b/tests/testthat/_snaps/summary-marking.md @@ -31,9 +31,8 @@ B (main) B-C (pairwise) C (main) - Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion - probability was 0 or 1 in every draw of every chain, or constant to numerical - precision; there is no variation to estimate them from. + Note: blank diagnostic cells mean these values could not be estimated + from the draws. Main-effect difference rows are blank when main_difference_selection = FALSE left their indicators unsampled. diff --git a/tests/testthat/test-mcmc-diagnostics.R b/tests/testthat/test-mcmc-diagnostics.R index ca924256..f34f8397 100644 --- a/tests/testthat/test-mcmc-diagnostics.R +++ b/tests/testthat/test-mcmc-diagnostics.R @@ -369,6 +369,24 @@ test_that("a single brief excursion does not inflate Rhat", { expect_lt(rhat, 1.01) }) +test_that("split Rhat preserves disagreement when parameter units change", { + set.seed(227) + x = array(rnorm(800), c(200, 4, 1)) + x[, 3:4, 1] = x[, 3:4, 1] + 4 + rhat = function(draws) bgms:::.compute_rhat_cpp(bgms:::split_chains(draws)) + expect_gt(rhat(x), 2) + expect_equal(rhat(x * 1e-9), rhat(x), tolerance = 1e-12) + + # Distinct, exactly representable constants must retain the Inf alarm. + stuck = array(0, c(100, 2, 1)) + stuck[, 2, 1] = 2^-30 + expect_identical(rhat(stuck), Inf) + + # Infinite split Rhat does not imply that a whole chain never moved. + moved = array(c(rep(0, 50), rep(1, 50)), c(100, 1, 1)) + expect_identical(rhat(moved), Inf) +}) + test_that("all-constant-and-equal sub-chains give NA", { x = array(1, dim = c(1000, 4, 1)) rhat = bgms:::.compute_rhat_cpp(bgms:::split_chains(x)) diff --git a/tests/testthat/test-prior-sensitivity.R b/tests/testthat/test-prior-sensitivity.R index ad166428..e63e858d 100644 --- a/tests/testthat/test-prior-sensitivity.R +++ b/tests/testthat/test-prior-sensitivity.R @@ -206,6 +206,11 @@ test_that("print() reports the counts, the movers and high R-hat fits", { ) counts = as.integer(sub(".* +([0-9]+)$", "\\1", count_lines)) expect_equal(c(fourth = counts[4], total = sum(counts)), c(fourth = 1L, total = 1L)) + expect_false(any(cls_unknown$edges$fragile)) + expect_match( + paste(out_unknown, collapse = " "), + "The fourth count also includes 1 edge whose verdict changed" + ) # Missing uncertainty must not hide the explanation for ordinary noisy edges. s_lo_mixed = s_lo @@ -217,7 +222,7 @@ test_that("print() reports the counts, the movers and high R-hat fits", { )) expect_match(out_mixed, "could change on a rerun 3") expect_match(out_mixed, "2 verdicts could change on a rerun without any change to the prior distribution: e3, e5") - expect_match(out_mixed, "Monte Carlo uncertainty could not be assessed for every verdict or verdict change") + expect_match(out_mixed, "The fourth count also includes 1 edge whose verdict changed but whose Monte Carlo error was unavailable") # An NA-only group names no flagged verdict. expect_false(any(grepl("without any change to the prior distribution", out_unknown, fixed = TRUE))) @@ -234,20 +239,20 @@ test_that("print() reports the counts, the movers and high R-hat fits", { # out of the numeric R-hat line. ps3 = fake_sensitivity(cls, multipliers, rhat_max = c(1.03, 1.005, Inf), s_hat = 0.8) out3 = gsub("\\s+", " ", paste(utils::capture.output(print(ps3)), collapse = " ")) - expect_match(out3, "In the 2x fit at least one parameter was constant within every split chain, at values that differ between split chains \\(split R-hat infinite\\): the sampler did not explore it") - expect_match(out3, "The verdicts of that fit do not describe the posterior distribution at that scale") + expect_match(out3, "The 2x fit has at least one parameter with infinite split R-hat") + expect_match(out3, "Check convergence before interpreting its verdicts") expect_match(out3, "Largest split R-hat above 1.01: 0.5x fit 1.03\\.") - expect_false(grepl("Monte Carlo standard errors in $edges", out3, fixed = TRUE)) + expect_false(grepl("NA Monte Carlo standard errors in $edges", out3, fixed = TRUE)) # An NA Monte Carlo error in some fit is explained in one sentence. cls_na = cls cls_na$edges$mcse_x2[1] = NA_real_ ps5 = fake_sensitivity(cls_na, multipliers, s_hat = 0.8) out5 = gsub("\\s+", " ", paste(utils::capture.output(print(ps5)), collapse = " ")) - expect_match(out5, "Monte Carlo standard errors in \\$edges are NA where a fit's inclusion probability was 0 or 1 in every draw of every chain, or constant to numerical precision") + expect_match(out5, "NA Monte Carlo standard errors in \\$edges could not be estimated from the draws") ps4 = fake_sensitivity(cls, multipliers, rhat_max = c(Inf, 1.005, Inf), s_hat = 0.8) out4 = gsub("\\s+", " ", paste(utils::capture.output(print(ps4)), collapse = " ")) - expect_match(out4, "In the 0.5x and 2x fits at least one parameter was constant within every split chain") - expect_match(out4, "The verdicts of those fits do not describe the posterior distribution at those scales") + expect_match(out4, "The 0.5x and 2x fits have at least one parameter with infinite split R-hat") + expect_match(out4, "Check convergence before interpreting their verdicts") expect_false(grepl("Largest split R-hat", out4, fixed = TRUE)) }) diff --git a/vignettes/diagnostics.Rmd b/vignettes/diagnostics.Rmd index b7e58664..ba12b789 100644 --- a/vignettes/diagnostics.Rmd +++ b/vignettes/diagnostics.Rmd @@ -84,26 +84,16 @@ The inclusion table carries its own diagnostics: summary(fit)$indicator ``` -- **mean**, **mcse**, **sd**, **n_eff**, and **Rhat** describe the - Rao-Blackwellized inclusion probability. At every update of an edge - indicator the sampler proposes to flip it and records the probability that - the edge is in the model after the move: the acceptance probability of - adding it when it is out, the probability of retaining it (one minus the - acceptance probability of removing it) when it is in. That recorded value - is continuous, so the standard Monte Carlo machinery applies to its series. - `mcse` and `n_eff` are `NA` when the series has no variation the machinery - can work on; `Rhat` is `NA` when every chain is exactly constant, and - otherwise computed. For an edge the data support overwhelmingly, the - probability of accepting its removal is below about 1e-16 at every update, - so one minus it rounds to exactly 1 and the series is constant at 1 in - every draw of every chain; an edge the data rule out as strongly is - constant at 0. A series that varies by less than about 3e-8 (a variance - below 1e-15) is below the floor at which the ESS kernel stops estimating - and is treated the same way; `sd` then prints as a number like `2e-11` - while `mcse` and `n_eff` are blank. In both cases the inclusion probability - stayed within 3e-8 of 1 or 0 throughout the run, so there is no Monte Carlo - error at any printed digit and the verdict lies far beyond any evidence - threshold; an edge whose series varies by more than that shows numbers. +The inclusion table summarizes the Rao-Blackwellized inclusion probabilities +recorded at each update: the probability of adding an absent edge or retaining +a present one. It reports their mean, Monte Carlo standard error (`mcse`), +standard deviation (`sd`), effective sample size (`n_eff`), and split-R-hat. + +Blank diagnostic cells contain `NA`. The recorded probabilities may vary too +little to estimate `mcse` and `n_eff`, often when they are close to 0 or 1. +This does not establish that Monte Carlo error is zero. Split-R-hat is computed +separately and can remain finite. Identical constant split chains give `NA`; +split chains constant at different values give `Inf`. ## Traceplots diff --git a/vignettes/prior-sensitivity.Rmd b/vignettes/prior-sensitivity.Rmd index ca613216..8d5ce373 100644 --- a/vignettes/prior-sensitivity.Rmd +++ b/vignettes/prior-sensitivity.Rmd @@ -95,6 +95,8 @@ the original fit, the `1x` column: another seed and no change to the prior distribution, so the edge is not classified by scale. The same edges carry `fragile = TRUE` in `verdicts(fit)`. + This count also includes edges whose verdict changed but whose Monte Carlo + error was unavailable. The report states how many such edges it includes. A line naming fits whose largest split R-hat exceeds 1.01 appears when there is one. The check uses every fit; that line, and the per-fit diagnostics in