diff --git a/NEWS.md b/NEWS.md index 3b43d3fd..bd5f0312 100644 --- a/NEWS.md +++ b/NEWS.md @@ -165,6 +165,21 @@ released from this line yet. ## Diagnostics +* `prior_sensitivity_check()` now uses the original fit's `verdicts()` flag + in its `fragile` column, replacing `insufficient` and its two subcause + columns. The fourth printed count, "could change on a rerun", includes + flagged edges and changes that could not be assessed because a Monte Carlo + error was unavailable. The report gives the number of these additional edges. + +* The sensitivity check's `saturated` column now indicates an unavailable + chosen-scale Monte Carlo standard error. It no longer requires inclusion + probabilities of exactly 0 or 1. The summary notes explain that blank + diagnostics could not be estimated; they do not imply zero Monte Carlo error. + +* Sensitivity reports identify fits with infinite split-R-hat and advise + checking convergence before interpreting their verdicts. The R-hat + calculation is unchanged. The `Refits:` line now capitalizes NUTS. + * The pairwise summary table for a `bgm()` fit and the difference summary tables for a `bgmCompare()` fit no longer include `share_incl`, the proportion of Monte Carlo variance attributed to uncertainty in inclusion. The reported diff --git a/R/methods_bgmcompare.R b/R/methods_bgmcompare.R index 3c4d3313..77f108f9 100644 --- a/R/methods_bgmcompare.R +++ b/R/methods_bgmcompare.R @@ -239,6 +239,21 @@ print.bgmCompare = function(x, ...) { #' are exempt from the union recode and are never marked). Fits made before #' that field existed print unmarked. #' +#' The inclusion table (`$indicator`) reports the Rao-Blackwellized inclusion +#' probability and its diagnostics for each difference. At each update the sampler +#' records the probability of adding the difference if absent, or retaining it if +#' present. The table summarizes these recorded probabilities. +#' +#' Blank diagnostic cells contain `NA`: the diagnostic could not be estimated. +#' `mcse` and `n_eff` can be unavailable when the recorded probabilities vary +#' too little, often near 0 or 1. This does not establish that Monte Carlo +#' error is zero. Split-R-hat uses a different calculation: identical constant +#' split chains give `NA`, while split chains constant at different values give +#' `Inf`. It can remain finite when `mcse` and `n_eff` are unavailable. +#' +#' Main-effect difference rows are blank when `main_difference_selection = FALSE` +#' because those indicators were not sampled. +#' #' @param object An object of class `bgmCompare`. #' @param ... Currently ignored. #' @@ -351,9 +366,8 @@ print.summary.bgmCompare = function(x, digits = 3, ...) { cat("... (use `summary(fit)$indicator` to see full output)\n") } if(ind_has_na) { - cat("Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion\n") - cat("probability was 0 or 1 in every draw; there is no variation to estimate them\n") - cat("from.\n") + cat("Note: blank diagnostic cells mean these values could not be estimated\n") + cat("from the draws.\n") if(isFALSE(x$arguments$main_difference_selection)) { cat("Main-effect difference rows are blank when main_difference_selection = FALSE\n") cat("left their indicators unsampled.\n") diff --git a/R/methods_bgms.R b/R/methods_bgms.R index 9d4bc681..28ef45de 100644 --- a/R/methods_bgms.R +++ b/R/methods_bgms.R @@ -81,6 +81,19 @@ print.bgms = function(x, ...) { #' #' @description Returns posterior summaries and diagnostics for a fitted `bgms` model. #' +#' @details +#' The inclusion table (`$indicator`) reports the Rao-Blackwellized inclusion +#' probability and its diagnostics for each edge. At each update the sampler +#' records the probability of adding the edge if absent, or retaining it if +#' present. The table summarizes these recorded probabilities. +#' +#' Blank diagnostic cells contain `NA`: the diagnostic could not be estimated. +#' `mcse` and `n_eff` can be unavailable when the recorded probabilities vary +#' too little, often near 0 or 1. This does not establish that Monte Carlo +#' error is zero. Split-R-hat uses a different calculation: identical constant +#' split chains give `NA`, while split chains constant at different values give +#' `Inf`. It can remain finite when `mcse` and `n_eff` are unavailable. +#' #' @param object An object of class `bgms`. #' @param ... Currently ignored. #' @@ -186,8 +199,8 @@ print.summary.bgms = function(x, digits = 3, ...) { print(ind) if(nrow(x$indicator) > .summary_preview_rows) cat("... (use `summary(fit)$indicator` to see full output)\n") if(ind_has_na) { - cat("Note: blank mcse, n_eff and Rhat cells mark edges whose inclusion probability\n") - cat("was 0 or 1 in every draw; there is no variation to estimate them from.\n") + cat("Note: blank diagnostic cells mean these values could not be estimated\n") + cat("from the draws.\n") } cat("\n") } diff --git a/R/prior_sensitivity.R b/R/prior_sensitivity.R index b16e3396..6e413c58 100644 --- a/R/prior_sensitivity.R +++ b/R/prior_sensitivity.R @@ -50,7 +50,8 @@ verdict_from_bf = function(bf, threshold) { #' are given. The original fit is the \code{1x} fit and is not refit. An #' ordinal (omrf) refit starts each chain from the original fit's final #' per-chain state, and a NUTS refit also from its adapted step size and mass -#' matrix; continuous (GGM) and mixed-MRF refits start cold. Every quantity +#' matrix; continuous (GGM) and mixed-MRF refits use default initial values. +#' Every quantity #' the check reports for a fit is that fit's own Rao-Blackwellized inclusion #' probability, its natural log inclusion Bayes factor and verdict, and the #' Monte Carlo standard error of the log Bayes factor from the fit's own @@ -68,22 +69,22 @@ verdict_from_bf = function(bf, threshold) { #' smallest and largest multiplier in the contiguous run of fits around #' \code{1x} that hold the reference verdict. #' -#' \strong{Sufficiency at the chosen scale.} An edge whose per-chain verdicts -#' disagree in the original fit, or whose log Bayes factor lies within -#' \code{max(between-chain band, 2 * MCSE)} of a verdict threshold there, is -#' marked \code{insufficient}: the fit cannot certify the verdict the check -#' starts from, and the printed report lists the edge as not certifiable -#' rather than in a mover category. +#' \strong{The chosen-scale flag.} The \code{fragile} column is the original +#' fit's flag from \code{\link{verdicts}()}: a log Bayes factor lies within +#' two standard errors of an evidence threshold. The fourth printed count, +#' "could change on a rerun", includes these edges and any other edge whose +#' verdict changed but whose Monte Carlo error was unavailable. The report +#' states how many of those additional edges there are. An edge whose standard +#' errors are both undefined is not flagged by \code{verdicts()}. #' -#' \strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the -#' continuous parameters, smallest effective sample size over the continuous -#' parameters and over the indicators, and for NUTS the smallest E-BFMI and -#' the largest energy variance ratio, are in \code{$grid}. The classification -#' uses every fit; the printed report names a fit whose largest -#' split-\eqn{\hat R} exceeds 1.01, and states when a fit did not sample: a -#' parameter constant in every chain at values that differ across chains has -#' an infinite split-\eqn{\hat R}, and that fit's verdicts say nothing about -#' the prior. +#' \strong{Convergence.} The \code{grid} table contains each fit's diagnostics. +#' Its \code{rhat_max} covers the main-effect and pairwise parameter blocks, +#' excluding indicators. All fits enter the classification, including those +#' with convergence problems. The printed report identifies fits with +#' split-\eqn{\hat R} above 1.01 or infinite. Check convergence before +#' interpreting their verdicts. For a refit, you can try increasing +#' \code{warmup} in \code{prior_sensitivity_check()}; for the original fit, +#' rerun \code{bgm()} or \code{bgmCompare()} with more warmup iterations. #' #' \strong{Models with a precision diagonal.} A continuous or mixed fit has a #' Gamma or exponential prior on the precision diagonal with a standardized @@ -154,17 +155,19 @@ verdict_from_bf = function(bf, threshold) { #' factor) and \code{chosen_scale_verdict}; \code{stability_lower} and #' \code{stability_upper}; \code{mover} (\code{stable}, #' \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or -#' \code{NA} for an indicator without a verdict) and \code{moved_at}; -#' \code{insufficient} with its two subcauses \code{insufficient_noisy} -#' (log Bayes factor within Monte Carlo error of a threshold) and -#' \code{insufficient_disagree} (chains disagree on the verdict); -#' \code{saturated} (the original fit's inclusion probability is 0 or 1 in -#' every draw of every chain); +#' \code{NA} when a verdict or the error needed to assess its change is +#' unavailable) and \code{moved_at}; +#' \code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the +#' original fit); \code{saturated} (\code{chosen_scale_mcse} is +#' \code{NA}; this indicates an unavailable error estimate, not necessarily +#' an inclusion probability of exactly 0 or 1); #' then one \code{verdict_x}, \code{log_bf_x} and \code{mcse_x} #' column per multiplier \code{m}, from that fit's own statistics.} #' \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale}, #' \code{original_fit}, \code{iter}, \code{warmup}, \code{seed}, the -#' diagnostics \code{rhat_max}, \code{ess_min}, \code{inclusion_ess_min}, +#' diagnostics \code{rhat_max} (largest split-\eqn{\hat R} over the +#' continuous parameters only), \code{ess_min} (over the continuous +#' parameters), \code{inclusion_ess_min} (over the indicators), #' \code{ebfmi_min}, \code{energy_variance_ratio_max} and #' \code{warmup_incomplete}, \code{sampler_notes}, and \code{seconds}.} #' \item{\code{verdict_by_scale}}{Fit-by-indicator matrix of verdicts.} @@ -481,6 +484,10 @@ prior_sensitivity_engine = function(bgms_object, for(k in seq_along(refit_idx)) fits[[refit_idx[k]]] = refits[[k]] fit_stats = lapply(fits, refit_edge_stats, evidence_threshold = evidence_threshold) + # The chosen-scale flag is verdicts()' near-threshold flag on the original + # fit, so both reports flag the same edges. + v0 = verdicts(bgms_object, evidence_threshold = evidence_threshold) + fit_stats[[s0_idx]]$fragile = v0$fragile[match(fit_stats[[s0_idx]]$edge, v0$parameter)] diag_rows = lapply(fits, refit_diagnostics_row) cls = sensitivity_classification(fit_stats, multipliers, s0_idx, lthr) @@ -570,23 +577,11 @@ sensitivity_classification = function(fit_stats, multipliers, s0_idx, lthr) { mcse_mat = t(stat_mat("mcse_lbf", "numeric")) verdict_mat = t(stat_mat("verdict", "character")) - # --- Sufficiency at the chosen scale, from the original fit ----------------- - # Could a rerun of the original fit with another seed have given this edge a - # different verdict, with no change of prior? Two Monte Carlo error measures - # of the log Bayes factor are available: twice its within-chain standard - # error (mcse_lbf) and its between-chain spread (band_half). The larger of - # the two is the half-width of an error band around the log Bayes factor. If - # one measure is NA (a single-chain fit has no between-chain spread), the - # other sets the half-width; if both are NA, the half-width is NA. The verdict - # is insufficient when that band contains a verdict threshold, +/- log of the - # evidence threshold, or when the chains' individual verdicts disagree. - hw = pmax(s0$band_half, 2 * s0$mcse_lbf, na.rm = TRUE) - straddle = ((s0$lbf - hw < lthr) & (s0$lbf + hw > lthr)) | - ((s0$lbf - hw < -lthr) & (s0$lbf + hw > -lthr)) - disagree = !s0$unanimous - insufficient = disagree | straddle - insufficient_noisy = straddle & insufficient - insufficient_disagree = disagree & insufficient + # --- The chosen-scale flag ------------------------------------------------- + # verdicts()' near-threshold flag on the original fit: could a rerun with + # another seed have given this edge a different verdict, with no change of + # prior? An edge whose standard errors are undefined is not flagged. + fragile = s0$fragile %in% TRUE # --- Mover category, where the verdict first differs, stability bracket ----- ref_v = s0$verdict @@ -634,10 +629,8 @@ sensitivity_classification = function(fit_stats, multipliers, s0_idx, lthr) { stability_upper = stability_upper, mover = mover, moved_at = moved_at, - insufficient = insufficient, - insufficient_noisy = insufficient_noisy, - insufficient_disagree = insufficient_disagree, - saturated = s0$zeroflip, + fragile = fragile, + saturated = s0$saturated, stringsAsFactors = FALSE, row.names = NULL ) vcols = as.data.frame(t(verdict_mat), stringsAsFactors = FALSE) @@ -678,13 +671,10 @@ order_upper_tri_rowmajor = function(p) { #' @title Print a Prior Sensitivity Check #' #' @description -#' Prints the verdict counts at each scale, the number of edges whose verdict -#' held at every scale, changed within Monte Carlo error, changed beyond it, -#' or could not be certified at the chosen scale, the edges that changed -#' beyond Monte Carlo error by name, the fits that did not sample (a -#' parameter constant in every chain at different values, infinite split -#' R-hat), the fits whose largest split R-hat exceeds 1.01, and how the chosen -#' scale compares with the scale the estimated interactions imply. +#' Prints verdict counts by scale, the four sensitivity counts, and the edges +#' whose verdicts changed beyond Monte Carlo error or could change on a rerun. +#' Also reports fits with high or infinite split-R-hat and compares the chosen +#' slab scale with the estimated interaction sizes. #' #' @param x A \code{bgms_prior_sensitivity} object. #' @param max_rows Integer. Maximum edges to name in the table of edges that @@ -760,19 +750,20 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { cat(paste0(" ", out, collapse = "\n"), "\n", sep = "") cat("\n") - # One category per edge: not certifiable at the chosen scale first, then the - # mover category. - uncert = edges$insufficient %in% TRUE | is.na(edges$insufficient) | - is.na(edges$mover) + # One category per edge: the verdicts that could change on a rerun first + # (verdicts()' flag on the original fit, and edges whose verdict change could + # not be compared with an error), then the mover category. + flagged = edges$fragile %in% TRUE + uncert = flagged | is.na(edges$mover) held = !uncert & edges$mover %in% "stable" within = !uncert & edges$mover %in% "within-monte-carlo-error" beyond = !uncert & edges$mover %in% "beyond-monte-carlo-error" counts = c(sum(held), sum(within), sum(beyond), sum(uncert)) labels = c( "verdict held at every scale", - "changed, within Monte Carlo error", - "changed, beyond Monte Carlo error", - "not certifiable at the chosen scale" + "changed within Monte Carlo error", + "changed beyond Monte Carlo error", + "could change on a rerun" ) cat(sprintf(" %-38s %4d\n", labels, counts), sep = "") cat("\n") @@ -797,61 +788,43 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { cat("\n") } - # The edges the original fit cannot certify, and why. + # The verdicts that could change on a rerun, by name; then the edges whose + # verdict change could not be compared with an error. if(sum(uncert) > 0) { - nm = edges$edge[uncert] - shown = utils::head(nm, max_rows) - tail_txt = if(length(nm) > length(shown)) { - sprintf(", and %d more (see $edges)", length(nm) - length(shown)) - } else { - "" - } - one = length(nm) == 1L - noisy = any(edges$insufficient_noisy[uncert], na.rm = TRUE) - disagree = any(edges$insufficient_disagree[uncert], na.rm = TRUE) - cause = if(noisy && disagree) { - "the Bayes factor lies within Monte Carlo error of a threshold or the chains disagree on the verdict" - } else if(disagree) { - "the chains disagree on the verdict" - } else { - "the Bayes factor lies within Monte Carlo error of a threshold" - } - explanation = if(noisy || disagree) { - sprintf( - "In the original fit %s, so a rerun with another seed could change %s verdict without any change of prior.", - cause, if(one) "its" else "the" - ) - } else { - character(0) + if(any(flagged)) { + nm = edges$edge[flagged] + shown = utils::head(nm, max_rows) + tail_txt = if(length(nm) > length(shown)) { + sprintf(", and %d more (see $edges)", length(nm) - length(shown)) + } else { + "" + } + one = length(nm) == 1L + wrap(sprintf( + "%d %s could change on a rerun without any change to the prior distribution: %s%s. %s within two standard errors of an evidence threshold.", + length(nm), if(one) "verdict" else "verdicts", + paste(shown, collapse = ", "), tail_txt, + if(one) "Its log Bayes factor lies" else "Their log Bayes factors lie" + )) } - if(anyNA(edges$insufficient[uncert]) || anyNA(edges$mover[uncert])) { - explanation = c( - explanation, - "Monte Carlo uncertainty could not be assessed for every verdict or verdict change; see $edges." - ) + if(any(uncert & !flagged)) { + n_unknown = sum(uncert & !flagged) + wrap(sprintf( + "The fourth count also includes %d %s whose verdict changed but whose Monte Carlo error was unavailable; see $edges.", + n_unknown, if(n_unknown == 1L) unit$noun else unit$nouns + )) } - explanation = paste(explanation, collapse = " ") - wrap(sprintf( - "%d %s %s not certifiable at the chosen scale: %s%s. %s", - length(nm), if(one) unit$noun else unit$nouns, if(one) "is" else "are", - paste(shown, collapse = ", "), tail_txt, explanation - )) cat("\n") } # Edges whose Monte Carlo error is NA in some fit. mcse_cols = grep("^mcse_x", names(edges), value = TRUE) if(length(mcse_cols) > 0 && anyNA(edges[, mcse_cols])) { - wrap(paste0( - "Monte Carlo standard errors in $edges are NA where a fit's inclusion ", - "probability was 0 or 1 in every draw; there is no variation to estimate ", - "them from." - )) + wrap("NA Monte Carlo standard errors in $edges could not be estimated from the draws.") cat("\n") } - # Fits with a parameter constant at different values across chains, and - # fits whose largest split R-hat exceeds 1.01. + # Report infinite split R-hat separately from finite values above 1.01. g = x$grid if(!is.null(g$rhat_max)) { fit_label = function(i) { @@ -862,11 +835,11 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { one = length(stuck) == 1L wrap(sprintf( paste0( - "The %s %s at least one parameter with infinite split R-hat: ", - "its sub-chains are constant but differ in value. %s verdicts should ", - "be interpreted only after checking convergence." + "The %s %s at least one parameter with infinite split R-hat. ", + "Check convergence before interpreting %s verdicts." ), - list_of(fit_label(stuck)), if(one) "fit includes" else "fits include", if(one) "Its" else "Their" + list_of(fit_label(stuck)), if(one) "fit has" else "fits have", + if(one) "its" else "their" )) cat("\n") } @@ -905,12 +878,13 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) { sprintf("%.0f s", x$runtime_seconds) } n_refit = sum(!g$original_fit) - start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "cold starts" + start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "started from default initial values" it = g$iter[!g$original_fit][1] wu = g$warmup[!g$original_fit][1] + sampler_label = if(identical(x$refit_sampler, "nuts")) "NUTS" else x$refit_sampler cat(sprintf( - "Refits: %d %s refit%s (%d warmup, %d sampling iterations each), %s, %s total.\n", - n_refit, x$refit_sampler, if(n_refit == 1L) "" else "s", wu, it, start_txt, secs + "Refits: %d %s refit%s (%d warmup, %d sampling iterations each), %s, %s total.\n", + n_refit, sampler_label, if(n_refit == 1L) "" else "s", wu, it, start_txt, secs )) cat("See ?prior_sensitivity_check for the construction.\n") invisible(x) diff --git a/R/refit_engine.R b/R/refit_engine.R index 827a3237..aca9dc8b 100644 --- a/R/refit_engine.R +++ b/R/refit_engine.R @@ -256,10 +256,9 @@ verdict_from_lbf = function(lbf, lthr) { # ------------------------------------------------------------------ # Per-edge quantities from one refit: the Rao-Blackwellized inclusion # probability (canonical), its natural log inclusion Bayes factor and verdict, -# the within-chain MCSE of the log BF (RB machinery, PR #182), the between-chain -# half-band (2 * SEM of the grand mean from chain spread, propagated to the log -# BF), per-chain verdict unanimity, and the zero-flip mask. Edge order is the -# native row-major upper triangle. +# the MCSE of the log BF from the summary table's inclusion MCSE by the delta +# method, and whether the inclusion MCSE is unavailable (the saturated column). +# Edge order is the native row-major upper triangle. # ------------------------------------------------------------------ refit_edge_stats = function(fit, evidence_threshold) { raw = get_raw_samples(fit) @@ -267,9 +266,8 @@ refit_edge_stats = function(fit, evidence_threshold) { # exactly. pnm = raw$parameter_names enm = pnm[["indicator"]] %||% pnm[["indicators"]] %||% pnm[["pairwise"]] - M = length(raw$rb_inclusion) - pc = sapply(raw$rb_inclusion, colMeans) # n_edges x M - if(is.null(dim(pc))) pc = matrix(pc, ncol = M) + pc = sapply(raw$rb_inclusion, colMeans) # n_edges x chains + if(is.null(dim(pc))) pc = matrix(pc, ncol = length(raw$rb_inclusion)) pbar = rowMeans(pc) prior_q = if(inherits(fit, "bgmCompare")) { @@ -287,23 +285,14 @@ refit_edge_stats = function(fit, evidence_threshold) { mcse_pip = ind[, "mcse"] lthr = log(evidence_threshold) - lbf = function(p) log((p / (1 - p)) / prior_odds) + lbf_bar = log((pbar / (1 - pbar)) / prior_odds) dfac = 1 / pmax(pbar * (1 - pbar), 1e-6) # d log BF / d p - lbf_bar = lbf(pbar) mcse_lbf = mcse_pip * dfac - se_between = apply(pc, 1, stats::sd) / sqrt(M) - band_half = 2 * se_between * dfac - - verdict = verdict_from_lbf(lbf_bar, lthr) - vc = apply(pc, 2, function(p) verdict_from_lbf(lbf(p), lthr)) # n_edges x M - if(is.null(dim(vc))) vc = matrix(vc, ncol = M) - unanimous = apply(vc, 1, function(v) length(unique(v[!is.na(v)])) <= 1L) - zeroflip = apply(pc, 1, function(p) all(p <= 0) || all(p >= 1)) list( edge = enm, pip = pbar, prior_odds = prior_odds, lbf = lbf_bar, - verdict = verdict, mcse_lbf = mcse_lbf, band_half = band_half, - unanimous = unanimous, zeroflip = zeroflip + verdict = verdict_from_lbf(lbf_bar, lthr), mcse_lbf = mcse_lbf, + saturated = is.na(mcse_pip) ) } diff --git a/R/verdicts.R b/R/verdicts.R index 098c3c7c..1a8e3df9 100644 --- a/R/verdicts.R +++ b/R/verdicts.R @@ -276,7 +276,8 @@ build_verdicts = function(parameter, log_bf, pip, mcse, draws, evidence_threshol #' \item{distance_two_state, distance_rb}{Distance from `log_bf` to the #' nearer verdict boundary, in units of each standard error.} #' \item{fragile}{`TRUE` when either distance is below 2: the -#' near-threshold flag.} +#' near-threshold flag. An indicator whose standard errors are both +#' undefined is not flagged.} #' } #' The evidence threshold is attached as the `evidence_threshold` attribute, #' and whether the fragility flag's operating point covers this kind of diff --git a/man/print.bgms_prior_sensitivity.Rd b/man/print.bgms_prior_sensitivity.Rd index 261d4c4b..3089b8b6 100644 --- a/man/print.bgms_prior_sensitivity.Rd +++ b/man/print.bgms_prior_sensitivity.Rd @@ -19,13 +19,10 @@ changed beyond Monte Carlo error; the rest are counted and left to \code{x}, invisibly. } \description{ -Prints the verdict counts at each scale, the number of edges whose verdict -held at every scale, changed within Monte Carlo error, changed beyond it, -or could not be certified at the chosen scale, the edges that changed -beyond Monte Carlo error by name, the fits that did not sample (a -parameter constant in every chain at different values, infinite split -R-hat), the fits whose largest split R-hat exceeds 1.01, and how the chosen -scale compares with the scale the estimated interactions imply. +Prints verdict counts by scale, the four sensitivity counts, and the edges +whose verdicts changed beyond Monte Carlo error or could change on a rerun. +Also reports fits with high or infinite split-R-hat and compares the chosen +slab scale with the estimated interaction sizes. } \seealso{ \code{\link{prior_sensitivity_check}()} diff --git a/man/prior_sensitivity_check.Rd b/man/prior_sensitivity_check.Rd index 665fc890..7a8629ca 100644 --- a/man/prior_sensitivity_check.Rd +++ b/man/prior_sensitivity_check.Rd @@ -68,17 +68,19 @@ inclusion probability, the original fit's \code{chosen_scale_pip}, factor) and \code{chosen_scale_verdict}; \code{stability_lower} and \code{stability_upper}; \code{mover} (\code{stable}, \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or -\code{NA} for an indicator without a verdict) and \code{moved_at}; -\code{insufficient} with its two subcauses \code{insufficient_noisy} -(log Bayes factor within Monte Carlo error of a threshold) and -\code{insufficient_disagree} (chains disagree on the verdict); -\code{saturated} (the original fit's inclusion probability is 0 or 1 in -every draw of every chain); +\code{NA} when a verdict or the error needed to assess its change is +unavailable) and \code{moved_at}; +\code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the +original fit); \code{saturated} (\code{chosen_scale_mcse} is +\code{NA}; this indicates an unavailable error estimate, not necessarily +an inclusion probability of exactly 0 or 1); then one \code{verdict_x}, \code{log_bf_x} and \code{mcse_x} column per multiplier \code{m}, from that fit's own statistics.} \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale}, \code{original_fit}, \code{iter}, \code{warmup}, \code{seed}, the -diagnostics \code{rhat_max}, \code{ess_min}, \code{inclusion_ess_min}, +diagnostics \code{rhat_max} (largest split-\eqn{\hat R} over the +continuous parameters only), \code{ess_min} (over the continuous +parameters), \code{inclusion_ess_min} (over the indicators), \code{ebfmi_min}, \code{energy_variance_ratio_max} and \code{warmup_incomplete}, \code{sampler_notes}, and \code{seconds}.} \item{\code{verdict_by_scale}}{Fit-by-indicator matrix of verdicts.} @@ -117,7 +119,8 @@ fit's \code{iter} and \code{warmup} unless \code{iter} and \code{warmup} are given. The original fit is the \code{1x} fit and is not refit. An ordinal (omrf) refit starts each chain from the original fit's final per-chain state, and a NUTS refit also from its adapted step size and mass -matrix; continuous (GGM) and mixed-MRF refits start cold. Every quantity +matrix; continuous (GGM) and mixed-MRF refits use default initial values. +Every quantity the check reports for a fit is that fit's own Rao-Blackwellized inclusion probability, its natural log inclusion Bayes factor and verdict, and the Monte Carlo standard error of the log Bayes factor from the fit's own @@ -135,22 +138,22 @@ verdict differs. \code{stability_lower} and \code{stability_upper} are the smallest and largest multiplier in the contiguous run of fits around \code{1x} that hold the reference verdict. -\strong{Sufficiency at the chosen scale.} An edge whose per-chain verdicts -disagree in the original fit, or whose log Bayes factor lies within -\code{max(between-chain band, 2 * MCSE)} of a verdict threshold there, is -marked \code{insufficient}: the fit cannot certify the verdict the check -starts from, and the printed report lists the edge as not certifiable -rather than in a mover category. - -\strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the -continuous parameters, smallest effective sample size over the continuous -parameters and over the indicators, and for NUTS the smallest E-BFMI and -the largest energy variance ratio, are in \code{$grid}. The classification -uses every fit; the printed report names a fit whose largest -split-\eqn{\hat R} exceeds 1.01, and states when a fit did not sample: a -parameter constant in every chain at values that differ across chains has -an infinite split-\eqn{\hat R}, and that fit's verdicts say nothing about -the prior. +\strong{The chosen-scale flag.} The \code{fragile} column is the original +fit's flag from \code{\link{verdicts}()}: a log Bayes factor lies within +two standard errors of an evidence threshold. The fourth printed count, +"could change on a rerun", includes these edges and any other edge whose +verdict changed but whose Monte Carlo error was unavailable. The report +states how many of those additional edges there are. An edge whose standard +errors are both undefined is not flagged by \code{verdicts()}. + +\strong{Convergence.} The \code{grid} table contains each fit's diagnostics. +Its \code{rhat_max} covers the main-effect and pairwise parameter blocks, +excluding indicators. All fits enter the classification, including those +with convergence problems. The printed report identifies fits with +split-\eqn{\hat R} above 1.01 or infinite. Check convergence before +interpreting their verdicts. For a refit, you can try increasing +\code{warmup} in \code{prior_sensitivity_check()}; for the original fit, +rerun \code{bgm()} or \code{bgmCompare()} with more warmup iterations. \strong{Models with a precision diagonal.} A continuous or mixed fit has a Gamma or exponential prior on the precision diagonal with a standardized diff --git a/man/summary.bgmCompare.Rd b/man/summary.bgmCompare.Rd index 49a71cd9..60bfb85f 100644 --- a/man/summary.bgmCompare.Rd +++ b/man/summary.bgmCompare.Rd @@ -39,6 +39,21 @@ The per-group counts behind the mark are in observation counts per variable (\code{NULL} for Blume-Capel variables, which are exempt from the union recode and are never marked). Fits made before that field existed print unmarked. + +The inclusion table (\verb{$indicator}) reports the Rao-Blackwellized inclusion +probability and its diagnostics for each difference. At each update the sampler +records the probability of adding the difference if absent, or retaining it if +present. The table summarizes these recorded probabilities. + +Blank diagnostic cells contain \code{NA}: the diagnostic could not be estimated. +\code{mcse} and \code{n_eff} can be unavailable when the recorded probabilities vary +too little, often near 0 or 1. This does not establish that Monte Carlo +error is zero. Split-R-hat uses a different calculation: identical constant +split chains give \code{NA}, while split chains constant at different values give +\code{Inf}. It can remain finite when \code{mcse} and \code{n_eff} are unavailable. + +Main-effect difference rows are blank when \code{main_difference_selection = FALSE} +because those indicators were not sampled. } \examples{ \donttest{ diff --git a/man/summary.bgms.Rd b/man/summary.bgms.Rd index 99b58486..7e013ad2 100644 --- a/man/summary.bgms.Rd +++ b/man/summary.bgms.Rd @@ -17,6 +17,19 @@ An object of class \code{summary.bgms} with posterior summaries. \description{ Returns posterior summaries and diagnostics for a fitted \code{bgms} model. } +\details{ +The inclusion table (\verb{$indicator}) reports the Rao-Blackwellized inclusion +probability and its diagnostics for each edge. At each update the sampler +records the probability of adding the edge if absent, or retaining it if +present. The table summarizes these recorded probabilities. + +Blank diagnostic cells contain \code{NA}: the diagnostic could not be estimated. +\code{mcse} and \code{n_eff} can be unavailable when the recorded probabilities vary +too little, often near 0 or 1. This does not establish that Monte Carlo +error is zero. Split-R-hat uses a different calculation: identical constant +split chains give \code{NA}, while split chains constant at different values give +\code{Inf}. It can remain finite when \code{mcse} and \code{n_eff} are unavailable. +} \examples{ \donttest{ fit = bgm(x = Wenchuan[, 1:3]) diff --git a/man/verdicts.Rd b/man/verdicts.Rd index 27f2df9d..441cf510 100644 --- a/man/verdicts.Rd +++ b/man/verdicts.Rd @@ -37,7 +37,8 @@ where the Rao-Blackwellized draws are constant to double precision.} \item{distance_two_state, distance_rb}{Distance from \code{log_bf} to the nearer verdict boundary, in units of each standard error.} \item{fragile}{\code{TRUE} when either distance is below 2: the -near-threshold flag.} +near-threshold flag. An indicator whose standard errors are both +undefined is not flagged.} } The evidence threshold is attached as the \code{evidence_threshold} attribute, and whether the fragility flag's operating point covers this kind of diff --git a/tests/testthat/_snaps/summary-marking.md b/tests/testthat/_snaps/summary-marking.md index d994f4cf..fcc44b9a 100644 --- a/tests/testthat/_snaps/summary-marking.md +++ b/tests/testthat/_snaps/summary-marking.md @@ -31,9 +31,8 @@ B (main) B-C (pairwise) C (main) - Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion - probability was 0 or 1 in every draw; there is no variation to estimate them - from. + Note: blank diagnostic cells mean these values could not be estimated + from the draws. Main-effect difference rows are blank when main_difference_selection = FALSE left their indicators unsampled. diff --git a/tests/testthat/test-mcmc-diagnostics.R b/tests/testthat/test-mcmc-diagnostics.R index ca924256..f34f8397 100644 --- a/tests/testthat/test-mcmc-diagnostics.R +++ b/tests/testthat/test-mcmc-diagnostics.R @@ -369,6 +369,24 @@ test_that("a single brief excursion does not inflate Rhat", { expect_lt(rhat, 1.01) }) +test_that("split Rhat preserves disagreement when parameter units change", { + set.seed(227) + x = array(rnorm(800), c(200, 4, 1)) + x[, 3:4, 1] = x[, 3:4, 1] + 4 + rhat = function(draws) bgms:::.compute_rhat_cpp(bgms:::split_chains(draws)) + expect_gt(rhat(x), 2) + expect_equal(rhat(x * 1e-9), rhat(x), tolerance = 1e-12) + + # Distinct, exactly representable constants must retain the Inf alarm. + stuck = array(0, c(100, 2, 1)) + stuck[, 2, 1] = 2^-30 + expect_identical(rhat(stuck), Inf) + + # Infinite split Rhat does not imply that a whole chain never moved. + moved = array(c(rep(0, 50), rep(1, 50)), c(100, 1, 1)) + expect_identical(rhat(moved), Inf) +}) + test_that("all-constant-and-equal sub-chains give NA", { x = array(1, dim = c(1000, 4, 1)) rhat = bgms:::.compute_rhat_cpp(bgms:::split_chains(x)) diff --git a/tests/testthat/test-prior-sensitivity.R b/tests/testthat/test-prior-sensitivity.R index c6e31aeb..e63e858d 100644 --- a/tests/testthat/test-prior-sensitivity.R +++ b/tests/testthat/test-prior-sensitivity.R @@ -22,15 +22,14 @@ skip_unless_slow = function() { # Per-fit statistics in the shape refit_edge_stats() returns, for n edges. fake_stats = function(lbf, mcse, prior_q = 0.5, lthr = log(10), - unanimous = TRUE, zeroflip = FALSE) { + fragile = FALSE, saturated = FALSE) { n = length(lbf) prior_odds = rep(prior_q / (1 - prior_q), n) pip = plogis(lbf + log(prior_odds)) list( edge = paste0("e", seq_len(n)), pip = pip, prior_odds = prior_odds, lbf = lbf, verdict = verdict_from_lbf(lbf, lthr), mcse_lbf = mcse, - band_half = rep(0, n), unanimous = rep_len(unanimous, n), - zeroflip = rep_len(zeroflip, n) + fragile = rep_len(fragile, n), saturated = rep_len(saturated, n) ) } @@ -92,23 +91,35 @@ test_that("the classification compares each verdict change with the two fits' Mo }) -test_that("sufficiency at the chosen scale is the only not-certifiable criterion", { +test_that("the chosen-scale flag is the original fit's verdicts() flag", { lthr = log(10) multipliers = c(0.5, 1, 2) - # Edge 1 sits within 2 * MCSE of the presence threshold at 1x; edge 2 is - # within error at 2x only; edge 3 has disagreeing chains at 1x. - s0 = fake_stats(c(2.2, 1.0, 1.5), mcse = c(0.1, 0.1, 0.1), unanimous = c(TRUE, TRUE, FALSE)) + # Edge 1 is flagged by verdicts() in the original fit; edge 2 is near a + # threshold at 2x only, which is not a flag. + s0 = fake_stats(c(2.2, 1.0, 1.5), mcse = c(0.1, 0.1, 0.1), fragile = c(TRUE, FALSE, FALSE)) s_lo = fake_stats(c(2.2, 1.0, 1.5), mcse = c(0.1, 0.1, 0.1)) s_hi = fake_stats(c(2.2, 2.5, 1.5), mcse = c(0.1, 0.1, 0.1)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) - expect_equal(cls$edges$insufficient, c(TRUE, FALSE, TRUE)) - expect_equal(cls$edges$insufficient_noisy, c(TRUE, FALSE, FALSE)) - expect_equal(cls$edges$insufficient_disagree, c(FALSE, FALSE, TRUE)) + expect_equal(cls$edges$fragile, c(TRUE, FALSE, FALSE)) + expect_false(any(c("insufficient", "insufficient_noisy", "insufficient_disagree") %in% names(cls$edges))) # Edge 2 is classified on its own change, not on its nearness to a threshold # at the other scale. expect_equal(cls$edges$mover[2], "beyond-monte-carlo-error") }) +test_that("the check's flag on a fit is verdicts()' flag on that fit", { + skip_on_cran() + fit = bgm(Wenchuan[1:150, 1:5], + iter = 300, warmup = 300, chains = 2, seed = 3, + display_progress = "none" + ) + ps = prior_sensitivity_check(fit, scale_multipliers = 2, seed = 3) + v = verdicts(fit) + expect_identical(ps$edges$fragile, v$fragile[match(ps$edges$edge, v$parameter)]) + # saturated is the edges whose chosen-scale MCSE is NA. + expect_identical(ps$edges$saturated, is.na(ps$edges$chosen_scale_mcse)) +}) + test_that("saturated and verdict-less indicators are handled", { lthr = log(10) @@ -116,14 +127,14 @@ test_that("saturated and verdict-less indicators are handled", { # Edge 1 is at inclusion probability 1 in every draw of the original fit # (infinite log BF, no MCSE) and becomes finite presence at 2x; edge 2 has # no verdict. - s0 = fake_stats(c(Inf, NA), mcse = c(NA, NA), zeroflip = c(TRUE, FALSE)) + s0 = fake_stats(c(Inf, NA), mcse = c(NA, NA), saturated = c(TRUE, FALSE)) s0$verdict[2] = NA_character_ - s_lo = fake_stats(c(Inf, NA), mcse = c(NA, NA), zeroflip = c(TRUE, FALSE)) + s_lo = fake_stats(c(Inf, NA), mcse = c(NA, NA), saturated = c(TRUE, FALSE)) s_hi = fake_stats(c(4, NA), mcse = c(0.2, NA)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) expect_equal(cls$edges$mover, c("stable", NA)) expect_true(cls$edges$saturated[1]) - expect_false(cls$edges$insufficient[1]) + expect_false(cls$edges$fragile[1]) expect_true(is.na(cls$edges$stability_lower[2])) # A verdict change away from a saturated indicator cannot be compared with # an error, since the saturated fit has none. @@ -138,30 +149,29 @@ test_that("an unknown Monte Carlo error is not assessed", { multipliers = c(0.5, 1, 2) # Edge 1: the refit whose verdict differs has no MCSE, so the change cannot # be compared with an error and the mover category is NA. Edge 2: the - # original fit has no MCSE and no band, so its sufficiency is NA. + # original fit has no MCSE; verdicts() does not flag an undefined standard + # error, so the edge is not flagged. s0 = fake_stats(c(1.0, 1.0), mcse = c(0.1, NA)) - s0$band_half[2] = NA_real_ s_lo = fake_stats(c(1.0, 1.0), mcse = c(0.1, 0.1)) s_hi = fake_stats(c(2.5, 1.0), mcse = c(NA, 0.1)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) expect_true(is.na(cls$edges$mover[1])) expect_equal(cls$edges$moved_at[1], 2) - expect_false(cls$edges$insufficient[1]) - expect_true(is.na(cls$edges$insufficient[2])) + expect_false(cls$edges$fragile[1]) + expect_false(cls$edges$fragile[2]) expect_equal(cls$edges$mover[2], "stable") - # A single-chain fit has no between-chain band; the MCSE alone sets the - # half-width. - s1 = fake_stats(c(2.2, 1.0), mcse = c(0.1, 0.1)) - s1$band_half[] = NA_real_ - cls1 = sensitivity_classification(list(s1, s1, s1), multipliers, 2L, lthr) - expect_equal(cls1$edges$insufficient, c(TRUE, FALSE)) }) test_that("print() reports the counts, the movers and high R-hat fits", { lthr = log(10) multipliers = c(0.5, 1, 2) - s0 = fake_stats(c(3.0, 1.0, 2.0, -3.0, 2.6), mcse = c(0.2, 0.1, 0.2, 0.2, 0.3)) + # Edges 3 and 5 are flagged by verdicts() in the original fit, so they count + # as verdicts that could change on a rerun rather than as movers. + s0 = fake_stats(c(3.0, 1.0, 2.0, -3.0, 2.6), + mcse = c(0.2, 0.1, 0.2, 0.2, 0.3), + fragile = c(FALSE, FALSE, TRUE, FALSE, TRUE) + ) s_lo = fake_stats(c(3.2, 2.5, 2.1, -2.9, 2.8), mcse = c(0.2, 0.1, 0.2, 0.2, 0.3)) s_hi = fake_stats(c(2.9, 1.1, 2.35, -3.1, 2.2), mcse = c(0.2, 0.1, 0.2, 0.2, 0.3)) cls = sensitivity_classification(list(s_lo, s0, s_hi), multipliers, 2L, lthr) @@ -169,20 +179,19 @@ test_that("print() reports the counts, the movers and high R-hat fits", { ps = fake_sensitivity(cls, multipliers, rhat_max = c(1.03, 1.005, 1.02), s_hat = 0.2) out = gsub("\\s+", " ", paste(utils::capture.output(print(ps)), collapse = " ")) expect_match(out, "fits at 0.5x, 1x and 2x the chosen scale") - # Edges 3 and 5 lie within 2 * MCSE of the presence threshold in the - # original fit, so they are not certifiable rather than movers. expect_match(out, "verdict held at every scale 2") - expect_match(out, "changed, within Monte Carlo error 0") - expect_match(out, "changed, beyond Monte Carlo error 1") - expect_match(out, "not certifiable at the chosen scale 2") - expect_match(out, "2 edges are not certifiable at the chosen scale: e3, e5") - expect_match(out, "within Monte Carlo error of a threshold, so a rerun") + expect_match(out, "changed within Monte Carlo error 0") + expect_match(out, "changed beyond Monte Carlo error 1") + expect_match(out, "could change on a rerun 2") + expect_match(out, "2 verdicts could change on a rerun without any change to the prior distribution: e3, e5") + expect_match(out, "Their log Bayes factors lie within two standard errors of an evidence threshold") + expect_false(grepl("not certifiable", out, fixed = TRUE)) expect_match(out, "1 edge changed verdict beyond Monte Carlo error") expect_match(out, "e2 presence undecided undecided 0.5x") expect_match(out, "Largest split R-hat above 1.01: 0.5x fit 1.03 and 2x fit 1.02") expect_match(out, "the slab is 5 times that") expect_match(out, "toward absence") - expect_match(out, "2 nuts refits") + expect_match(out, "Refits: 2 NUTS refits") # A changed verdict with an unavailable MCSE belongs in the fourth count. unknown = fake_stats(2.5, mcse = NA_real_) @@ -192,11 +201,16 @@ test_that("print() reports the counts, the movers and high R-hat fits", { ) out_unknown = utils::capture.output(print(fake_sensitivity(cls_unknown, multipliers))) count_lines = grep( - "^ (verdict held|changed,|not certifiable)", out_unknown, + "^ (verdict held|changed |could change)", out_unknown, value = TRUE ) counts = as.integer(sub(".* +([0-9]+)$", "\\1", count_lines)) expect_equal(c(fourth = counts[4], total = sum(counts)), c(fourth = 1L, total = 1L)) + expect_false(any(cls_unknown$edges$fragile)) + expect_match( + paste(out_unknown, collapse = " "), + "The fourth count also includes 1 edge whose verdict changed" + ) # Missing uncertainty must not hide the explanation for ordinary noisy edges. s_lo_mixed = s_lo @@ -206,11 +220,11 @@ test_that("print() reports the counts, the movers and high R-hat fits", { utils::capture.output(print(fake_sensitivity(cls_mixed, multipliers))), collapse = " " )) - expect_match(out_mixed, "not certifiable at the chosen scale 3") - expect_match(out_mixed, "within Monte Carlo error of a threshold, so a rerun") - expect_match(out_mixed, "Monte Carlo uncertainty could not be assessed for every verdict or verdict change") - # An NA-only group has no known threshold or chain-disagreement cause. - expect_false(any(grepl("rerun with another seed", out_unknown, fixed = TRUE))) + expect_match(out_mixed, "could change on a rerun 3") + expect_match(out_mixed, "2 verdicts could change on a rerun without any change to the prior distribution: e3, e5") + expect_match(out_mixed, "The fourth count also includes 1 edge whose verdict changed but whose Monte Carlo error was unavailable") + # An NA-only group names no flagged verdict. + expect_false(any(grepl("without any change to the prior distribution", out_unknown, fixed = TRUE))) # Every fit at or below 1.01, and a scale within a factor two: neither # sentence appears, and the verdict table has every scale. @@ -225,20 +239,20 @@ test_that("print() reports the counts, the movers and high R-hat fits", { # out of the numeric R-hat line. ps3 = fake_sensitivity(cls, multipliers, rhat_max = c(1.03, 1.005, Inf), s_hat = 0.8) out3 = gsub("\\s+", " ", paste(utils::capture.output(print(ps3)), collapse = " ")) - expect_match(out3, "The 2x fit includes at least one parameter with infinite split R-hat") - expect_match(out3, "Its verdicts should be interpreted only after checking convergence") + expect_match(out3, "The 2x fit has at least one parameter with infinite split R-hat") + expect_match(out3, "Check convergence before interpreting its verdicts") expect_match(out3, "Largest split R-hat above 1.01: 0.5x fit 1.03\\.") - expect_false(grepl("Monte Carlo standard errors in $edges", out3, fixed = TRUE)) + expect_false(grepl("NA Monte Carlo standard errors in $edges", out3, fixed = TRUE)) # An NA Monte Carlo error in some fit is explained in one sentence. cls_na = cls cls_na$edges$mcse_x2[1] = NA_real_ ps5 = fake_sensitivity(cls_na, multipliers, s_hat = 0.8) out5 = gsub("\\s+", " ", paste(utils::capture.output(print(ps5)), collapse = " ")) - expect_match(out5, "Monte Carlo standard errors in \\$edges are NA where a fit's inclusion probability was 0 or 1 in every draw") + expect_match(out5, "NA Monte Carlo standard errors in \\$edges could not be estimated from the draws") ps4 = fake_sensitivity(cls, multipliers, rhat_max = c(Inf, 1.005, Inf), s_hat = 0.8) out4 = gsub("\\s+", " ", paste(utils::capture.output(print(ps4)), collapse = " ")) - expect_match(out4, "The 0.5x and 2x fits include at least one parameter with infinite split R-hat") - expect_match(out4, "Their verdicts should be interpreted only after checking convergence") + expect_match(out4, "The 0.5x and 2x fits have at least one parameter with infinite split R-hat") + expect_match(out4, "Check convergence before interpreting their verdicts") expect_false(grepl("Largest split R-hat", out4, fixed = TRUE)) }) diff --git a/vignettes/diagnostics.Rmd b/vignettes/diagnostics.Rmd index 3648661d..ba12b789 100644 --- a/vignettes/diagnostics.Rmd +++ b/vignettes/diagnostics.Rmd @@ -84,20 +84,16 @@ The inclusion table carries its own diagnostics: summary(fit)$indicator ``` -- **mean**, **mcse**, **sd**, **n_eff**, and **Rhat** describe the - Rao-Blackwellized inclusion probability. The sampler records a one-step - inclusion draw at every indicator update, and that draw is continuous, so the - standard Monte Carlo machinery applies to it. They are `NA` when that draw - carries too little variability for the machinery to work on, which happens by - two routes. The first is an indicator whose Rao-Blackwellized draws are - constant to double precision, where the inclusion probability sits at its - numerical bound; `sd` is then exactly `0`. The second is a draw whose - variance is non-zero but falls below the numerical floor of the - autocovariance kernel: `mcse` and `n_eff` come back `NA` while `sd` and - `Rhat` are still reported. In the table above, `intrusion-flash` takes the - second route — an `sd` of about `2e-11` with `NA` `mcse` and `n_eff` and a - finite `Rhat`. Either way the reading is the same: the inclusion probability - is pinned, not poorly estimated. +The inclusion table summarizes the Rao-Blackwellized inclusion probabilities +recorded at each update: the probability of adding an absent edge or retaining +a present one. It reports their mean, Monte Carlo standard error (`mcse`), +standard deviation (`sd`), effective sample size (`n_eff`), and split-R-hat. + +Blank diagnostic cells contain `NA`. The recorded probabilities may vary too +little to estimate `mcse` and `n_eff`, often when they are close to 0 or 1. +This does not establish that Monte Carlo error is zero. Split-R-hat is computed +separately and can remain finite. Identical constant split chains give `NA`; +split chains constant at different values give `Inf`. ## Traceplots diff --git a/vignettes/prior-sensitivity-ps.rds b/vignettes/prior-sensitivity-ps.rds index 4345e222..048d091b 100644 Binary files a/vignettes/prior-sensitivity-ps.rds and b/vignettes/prior-sensitivity-ps.rds differ diff --git a/vignettes/prior-sensitivity.Rmd b/vignettes/prior-sensitivity.Rmd index f7e64f94..8d5ce373 100644 --- a/vignettes/prior-sensitivity.Rmd +++ b/vignettes/prior-sensitivity.Rmd @@ -82,17 +82,21 @@ The four counts below that table classify every edge against the verdict of the original fit, the `1x` column: - *verdict held at every scale*: the edge has the same verdict in every fit. -- *changed, within Monte Carlo error*: the verdict differs at some scale, but +- *changed within Monte Carlo error*: the verdict differs at some scale, but the change in the log Bayes factor from the original fit to that fit is smaller than twice the combined Monte Carlo standard error of the two fits. Two runs of the same analysis could have produced this difference. -- *changed, beyond Monte Carlo error*: the change is larger than that. The +- *changed beyond Monte Carlo error*: the change is larger than that. The verdict depends on the scale. These edges are named, with their verdict at each scale and the scale nearest the chosen one at which it differs. -- *not certifiable at the chosen scale*: in the original fit the log Bayes - factor lies within Monte Carlo error of a verdict threshold, or the chains - disagree on the verdict. The verdict the check starts from is not settled, - so the edge is not classified; running the original fit longer settles it. +- *could change on a rerun*: `verdicts()` flags the edge in the original fit, + because its log Bayes factor lies within two standard errors of an evidence + threshold. The verdict the check starts from could change on a rerun with + another seed and no change to the prior distribution, so the edge is not + classified by scale. The same edges carry `fragile = TRUE` in + `verdicts(fit)`. + This count also includes edges whose verdict changed but whose Monte Carlo + error was unavailable. The report states how many such edges it includes. A line naming fits whose largest split R-hat exceeds 1.01 appears when there is one. The check uses every fit; that line, and the per-fit diagnostics in @@ -109,9 +113,9 @@ moves near-threshold verdicts. Bayes factor, Monte Carlo standard error and verdict; the `mover` category and `moved_at`; `stability_lower` and `stability_upper`, the smallest and largest multiplier in the contiguous run of fits around `1x` that hold the -original verdict; the `insufficient` flag with its two subcauses; and one -verdict, log Bayes factor and Monte Carlo standard error column per scale, -each from that fit's own statistics. +original verdict; the `fragile` flag of `verdicts()` on the original fit and +the `saturated` flag; and one verdict, log Bayes factor and Monte Carlo +standard error column per scale, each from that fit's own statistics. ```{r, eval = FALSE} head(ps$edges)