Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,21 @@ released from this line yet.

## Diagnostics

* `prior_sensitivity_check()` now uses the original fit's `verdicts()` flag
in its `fragile` column, replacing `insufficient` and its two subcause
columns. The fourth printed count, "could change on a rerun", includes
flagged edges and changes that could not be assessed because a Monte Carlo
error was unavailable. The report gives the number of these additional edges.

* The sensitivity check's `saturated` column now indicates an unavailable
chosen-scale Monte Carlo standard error. It no longer requires inclusion
probabilities of exactly 0 or 1. The summary notes explain that blank
diagnostics could not be estimated; they do not imply zero Monte Carlo error.

* Sensitivity reports identify fits with infinite split-R-hat and advise
checking convergence before interpreting their verdicts. The R-hat
calculation is unchanged. The `Refits:` line now capitalizes NUTS.

* The pairwise summary table for a `bgm()` fit and the difference summary
tables for a `bgmCompare()` fit no longer include `share_incl`, the proportion
of Monte Carlo variance attributed to uncertainty in inclusion. The reported
Expand Down
20 changes: 17 additions & 3 deletions R/methods_bgmcompare.R
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,21 @@ print.bgmCompare = function(x, ...) {
#' are exempt from the union recode and are never marked). Fits made before
#' that field existed print unmarked.
#'
#' The inclusion table (`$indicator`) reports the Rao-Blackwellized inclusion
#' probability and its diagnostics for each difference. At each update the sampler
#' records the probability of adding the difference if absent, or retaining it if
#' present. The table summarizes these recorded probabilities.
#'
#' Blank diagnostic cells contain `NA`: the diagnostic could not be estimated.
#' `mcse` and `n_eff` can be unavailable when the recorded probabilities vary
#' too little, often near 0 or 1. This does not establish that Monte Carlo
#' error is zero. Split-R-hat uses a different calculation: identical constant
#' split chains give `NA`, while split chains constant at different values give
#' `Inf`. It can remain finite when `mcse` and `n_eff` are unavailable.
#'
#' Main-effect difference rows are blank when `main_difference_selection = FALSE`
#' because those indicators were not sampled.
#'
#' @param object An object of class `bgmCompare`.
#' @param ... Currently ignored.
#'
Expand Down Expand Up @@ -351,9 +366,8 @@ print.summary.bgmCompare = function(x, digits = 3, ...) {
cat("... (use `summary(fit)$indicator` to see full output)\n")
}
if(ind_has_na) {
cat("Note: blank mcse, n_eff and Rhat cells mark indicators whose inclusion\n")
cat("probability was 0 or 1 in every draw; there is no variation to estimate them\n")
cat("from.\n")
cat("Note: blank diagnostic cells mean these values could not be estimated\n")
cat("from the draws.\n")
if(isFALSE(x$arguments$main_difference_selection)) {
cat("Main-effect difference rows are blank when main_difference_selection = FALSE\n")
cat("left their indicators unsampled.\n")
Expand Down
17 changes: 15 additions & 2 deletions R/methods_bgms.R
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,19 @@ print.bgms = function(x, ...) {
#'
#' @description Returns posterior summaries and diagnostics for a fitted `bgms` model.
#'
#' @details
#' The inclusion table (`$indicator`) reports the Rao-Blackwellized inclusion
#' probability and its diagnostics for each edge. At each update the sampler
#' records the probability of adding the edge if absent, or retaining it if
#' present. The table summarizes these recorded probabilities.
#'
#' Blank diagnostic cells contain `NA`: the diagnostic could not be estimated.
#' `mcse` and `n_eff` can be unavailable when the recorded probabilities vary
#' too little, often near 0 or 1. This does not establish that Monte Carlo
#' error is zero. Split-R-hat uses a different calculation: identical constant
#' split chains give `NA`, while split chains constant at different values give
#' `Inf`. It can remain finite when `mcse` and `n_eff` are unavailable.
#'
#' @param object An object of class `bgms`.
#' @param ... Currently ignored.
#'
Expand Down Expand Up @@ -186,8 +199,8 @@ print.summary.bgms = function(x, digits = 3, ...) {
print(ind)
if(nrow(x$indicator) > .summary_preview_rows) cat("... (use `summary(fit)$indicator` to see full output)\n")
if(ind_has_na) {
cat("Note: blank mcse, n_eff and Rhat cells mark edges whose inclusion probability\n")
cat("was 0 or 1 in every draw; there is no variation to estimate them from.\n")
cat("Note: blank diagnostic cells mean these values could not be estimated\n")
cat("from the draws.\n")
}
cat("\n")
}
Expand Down
190 changes: 82 additions & 108 deletions R/prior_sensitivity.R
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ verdict_from_bf = function(bf, threshold) {
#' are given. The original fit is the \code{1x} fit and is not refit. An
#' ordinal (omrf) refit starts each chain from the original fit's final
#' per-chain state, and a NUTS refit also from its adapted step size and mass
#' matrix; continuous (GGM) and mixed-MRF refits start cold. Every quantity
#' matrix; continuous (GGM) and mixed-MRF refits use default initial values.
#' Every quantity
#' the check reports for a fit is that fit's own Rao-Blackwellized inclusion
#' probability, its natural log inclusion Bayes factor and verdict, and the
#' Monte Carlo standard error of the log Bayes factor from the fit's own
Expand All @@ -68,22 +69,22 @@ verdict_from_bf = function(bf, threshold) {
#' smallest and largest multiplier in the contiguous run of fits around
#' \code{1x} that hold the reference verdict.
#'
#' \strong{Sufficiency at the chosen scale.} An edge whose per-chain verdicts
#' disagree in the original fit, or whose log Bayes factor lies within
#' \code{max(between-chain band, 2 * MCSE)} of a verdict threshold there, is
#' marked \code{insufficient}: the fit cannot certify the verdict the check
#' starts from, and the printed report lists the edge as not certifiable
#' rather than in a mover category.
#' \strong{The chosen-scale flag.} The \code{fragile} column is the original
#' fit's flag from \code{\link{verdicts}()}: a log Bayes factor lies within
#' two standard errors of an evidence threshold. The fourth printed count,
#' "could change on a rerun", includes these edges and any other edge whose
#' verdict changed but whose Monte Carlo error was unavailable. The report
#' states how many of those additional edges there are. An edge whose standard
#' errors are both undefined is not flagged by \code{verdicts()}.
#'
#' \strong{Convergence.} Each fit's largest split-\eqn{\hat R} over the
#' continuous parameters, smallest effective sample size over the continuous
#' parameters and over the indicators, and for NUTS the smallest E-BFMI and
#' the largest energy variance ratio, are in \code{$grid}. The classification
#' uses every fit; the printed report names a fit whose largest
#' split-\eqn{\hat R} exceeds 1.01, and states when a fit did not sample: a
#' parameter constant in every chain at values that differ across chains has
#' an infinite split-\eqn{\hat R}, and that fit's verdicts say nothing about
#' the prior.
#' \strong{Convergence.} The \code{grid} table contains each fit's diagnostics.
#' Its \code{rhat_max} covers the main-effect and pairwise parameter blocks,
#' excluding indicators. All fits enter the classification, including those
#' with convergence problems. The printed report identifies fits with
#' split-\eqn{\hat R} above 1.01 or infinite. Check convergence before
#' interpreting their verdicts. For a refit, you can try increasing
#' \code{warmup} in \code{prior_sensitivity_check()}; for the original fit,
#' rerun \code{bgm()} or \code{bgmCompare()} with more warmup iterations.
#'
#' \strong{Models with a precision diagonal.} A continuous or mixed fit has a
#' Gamma or exponential prior on the precision diagonal with a standardized
Expand Down Expand Up @@ -154,17 +155,19 @@ verdict_from_bf = function(bf, threshold) {
#' factor) and \code{chosen_scale_verdict}; \code{stability_lower} and
#' \code{stability_upper}; \code{mover} (\code{stable},
#' \code{within-monte-carlo-error}, \code{beyond-monte-carlo-error}, or
#' \code{NA} for an indicator without a verdict) and \code{moved_at};
#' \code{insufficient} with its two subcauses \code{insufficient_noisy}
#' (log Bayes factor within Monte Carlo error of a threshold) and
#' \code{insufficient_disagree} (chains disagree on the verdict);
#' \code{saturated} (the original fit's inclusion probability is 0 or 1 in
#' every draw of every chain);
#' \code{NA} when a verdict or the error needed to assess its change is
#' unavailable) and \code{moved_at};
#' \code{fragile} (\code{\link{verdicts}()}' near-threshold flag on the
#' original fit); \code{saturated} (\code{chosen_scale_mcse} is
#' \code{NA}; this indicates an unavailable error estimate, not necessarily
#' an inclusion probability of exactly 0 or 1);
#' then one \code{verdict_x<m>}, \code{log_bf_x<m>} and \code{mcse_x<m>}
#' column per multiplier \code{m}, from that fit's own statistics.}
#' \item{\code{grid}}{One row per fit: \code{multiplier}, \code{scale},
#' \code{original_fit}, \code{iter}, \code{warmup}, \code{seed}, the
#' diagnostics \code{rhat_max}, \code{ess_min}, \code{inclusion_ess_min},
#' diagnostics \code{rhat_max} (largest split-\eqn{\hat R} over the
#' continuous parameters only), \code{ess_min} (over the continuous
#' parameters), \code{inclusion_ess_min} (over the indicators),
#' \code{ebfmi_min}, \code{energy_variance_ratio_max} and
#' \code{warmup_incomplete}, \code{sampler_notes}, and \code{seconds}.}
#' \item{\code{verdict_by_scale}}{Fit-by-indicator matrix of verdicts.}
Expand Down Expand Up @@ -481,6 +484,10 @@ prior_sensitivity_engine = function(bgms_object,
for(k in seq_along(refit_idx)) fits[[refit_idx[k]]] = refits[[k]]

fit_stats = lapply(fits, refit_edge_stats, evidence_threshold = evidence_threshold)
# The chosen-scale flag is verdicts()' near-threshold flag on the original
# fit, so both reports flag the same edges.
v0 = verdicts(bgms_object, evidence_threshold = evidence_threshold)
fit_stats[[s0_idx]]$fragile = v0$fragile[match(fit_stats[[s0_idx]]$edge, v0$parameter)]
diag_rows = lapply(fits, refit_diagnostics_row)

cls = sensitivity_classification(fit_stats, multipliers, s0_idx, lthr)
Expand Down Expand Up @@ -570,23 +577,11 @@ sensitivity_classification = function(fit_stats, multipliers, s0_idx, lthr) {
mcse_mat = t(stat_mat("mcse_lbf", "numeric"))
verdict_mat = t(stat_mat("verdict", "character"))

# --- Sufficiency at the chosen scale, from the original fit -----------------
# Could a rerun of the original fit with another seed have given this edge a
# different verdict, with no change of prior? Two Monte Carlo error measures
# of the log Bayes factor are available: twice its within-chain standard
# error (mcse_lbf) and its between-chain spread (band_half). The larger of
# the two is the half-width of an error band around the log Bayes factor. If
# one measure is NA (a single-chain fit has no between-chain spread), the
# other sets the half-width; if both are NA, the half-width is NA. The verdict
# is insufficient when that band contains a verdict threshold, +/- log of the
# evidence threshold, or when the chains' individual verdicts disagree.
hw = pmax(s0$band_half, 2 * s0$mcse_lbf, na.rm = TRUE)
straddle = ((s0$lbf - hw < lthr) & (s0$lbf + hw > lthr)) |
((s0$lbf - hw < -lthr) & (s0$lbf + hw > -lthr))
disagree = !s0$unanimous
insufficient = disagree | straddle
insufficient_noisy = straddle & insufficient
insufficient_disagree = disagree & insufficient
# --- The chosen-scale flag -------------------------------------------------
# verdicts()' near-threshold flag on the original fit: could a rerun with
# another seed have given this edge a different verdict, with no change of
# prior? An edge whose standard errors are undefined is not flagged.
fragile = s0$fragile %in% TRUE

# --- Mover category, where the verdict first differs, stability bracket -----
ref_v = s0$verdict
Expand Down Expand Up @@ -634,10 +629,8 @@ sensitivity_classification = function(fit_stats, multipliers, s0_idx, lthr) {
stability_upper = stability_upper,
mover = mover,
moved_at = moved_at,
insufficient = insufficient,
insufficient_noisy = insufficient_noisy,
insufficient_disagree = insufficient_disagree,
saturated = s0$zeroflip,
fragile = fragile,
saturated = s0$saturated,
stringsAsFactors = FALSE, row.names = NULL
)
vcols = as.data.frame(t(verdict_mat), stringsAsFactors = FALSE)
Expand Down Expand Up @@ -678,13 +671,10 @@ order_upper_tri_rowmajor = function(p) {
#' @title Print a Prior Sensitivity Check
#'
#' @description
#' Prints the verdict counts at each scale, the number of edges whose verdict
#' held at every scale, changed within Monte Carlo error, changed beyond it,
#' or could not be certified at the chosen scale, the edges that changed
#' beyond Monte Carlo error by name, the fits that did not sample (a
#' parameter constant in every chain at different values, infinite split
#' R-hat), the fits whose largest split R-hat exceeds 1.01, and how the chosen
#' scale compares with the scale the estimated interactions imply.
#' Prints verdict counts by scale, the four sensitivity counts, and the edges
#' whose verdicts changed beyond Monte Carlo error or could change on a rerun.
#' Also reports fits with high or infinite split-R-hat and compares the chosen
#' slab scale with the estimated interaction sizes.
#'
#' @param x A \code{bgms_prior_sensitivity} object.
#' @param max_rows Integer. Maximum edges to name in the table of edges that
Expand Down Expand Up @@ -760,19 +750,20 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) {
cat(paste0(" ", out, collapse = "\n"), "\n", sep = "")
cat("\n")

# One category per edge: not certifiable at the chosen scale first, then the
# mover category.
uncert = edges$insufficient %in% TRUE | is.na(edges$insufficient) |
is.na(edges$mover)
# One category per edge: the verdicts that could change on a rerun first
# (verdicts()' flag on the original fit, and edges whose verdict change could
# not be compared with an error), then the mover category.
flagged = edges$fragile %in% TRUE
uncert = flagged | is.na(edges$mover)
held = !uncert & edges$mover %in% "stable"
within = !uncert & edges$mover %in% "within-monte-carlo-error"
beyond = !uncert & edges$mover %in% "beyond-monte-carlo-error"
counts = c(sum(held), sum(within), sum(beyond), sum(uncert))
labels = c(
"verdict held at every scale",
"changed, within Monte Carlo error",
"changed, beyond Monte Carlo error",
"not certifiable at the chosen scale"
"changed within Monte Carlo error",
"changed beyond Monte Carlo error",
"could change on a rerun"
)
cat(sprintf(" %-38s %4d\n", labels, counts), sep = "")
cat("\n")
Expand All @@ -797,61 +788,43 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) {
cat("\n")
}

# The edges the original fit cannot certify, and why.
# The verdicts that could change on a rerun, by name; then the edges whose
# verdict change could not be compared with an error.
if(sum(uncert) > 0) {
nm = edges$edge[uncert]
shown = utils::head(nm, max_rows)
tail_txt = if(length(nm) > length(shown)) {
sprintf(", and %d more (see $edges)", length(nm) - length(shown))
} else {
""
}
one = length(nm) == 1L
noisy = any(edges$insufficient_noisy[uncert], na.rm = TRUE)
disagree = any(edges$insufficient_disagree[uncert], na.rm = TRUE)
cause = if(noisy && disagree) {
"the Bayes factor lies within Monte Carlo error of a threshold or the chains disagree on the verdict"
} else if(disagree) {
"the chains disagree on the verdict"
} else {
"the Bayes factor lies within Monte Carlo error of a threshold"
}
explanation = if(noisy || disagree) {
sprintf(
"In the original fit %s, so a rerun with another seed could change %s verdict without any change of prior.",
cause, if(one) "its" else "the"
)
} else {
character(0)
if(any(flagged)) {
nm = edges$edge[flagged]
shown = utils::head(nm, max_rows)
tail_txt = if(length(nm) > length(shown)) {
sprintf(", and %d more (see $edges)", length(nm) - length(shown))
} else {
""
}
one = length(nm) == 1L
wrap(sprintf(
"%d %s could change on a rerun without any change to the prior distribution: %s%s. %s within two standard errors of an evidence threshold.",
length(nm), if(one) "verdict" else "verdicts",
paste(shown, collapse = ", "), tail_txt,
if(one) "Its log Bayes factor lies" else "Their log Bayes factors lie"
))
}
if(anyNA(edges$insufficient[uncert]) || anyNA(edges$mover[uncert])) {
explanation = c(
explanation,
"Monte Carlo uncertainty could not be assessed for every verdict or verdict change; see $edges."
)
if(any(uncert & !flagged)) {
n_unknown = sum(uncert & !flagged)
wrap(sprintf(
"The fourth count also includes %d %s whose verdict changed but whose Monte Carlo error was unavailable; see $edges.",
n_unknown, if(n_unknown == 1L) unit$noun else unit$nouns
))
}
explanation = paste(explanation, collapse = " ")
wrap(sprintf(
"%d %s %s not certifiable at the chosen scale: %s%s. %s",
length(nm), if(one) unit$noun else unit$nouns, if(one) "is" else "are",
paste(shown, collapse = ", "), tail_txt, explanation
))
cat("\n")
}

# Edges whose Monte Carlo error is NA in some fit.
mcse_cols = grep("^mcse_x", names(edges), value = TRUE)
if(length(mcse_cols) > 0 && anyNA(edges[, mcse_cols])) {
wrap(paste0(
"Monte Carlo standard errors in $edges are NA where a fit's inclusion ",
"probability was 0 or 1 in every draw; there is no variation to estimate ",
"them from."
))
wrap("NA Monte Carlo standard errors in $edges could not be estimated from the draws.")
cat("\n")
}

# Fits with a parameter constant at different values across chains, and
# fits whose largest split R-hat exceeds 1.01.
# Report infinite split R-hat separately from finite values above 1.01.
g = x$grid
if(!is.null(g$rhat_max)) {
fit_label = function(i) {
Expand All @@ -862,11 +835,11 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) {
one = length(stuck) == 1L
wrap(sprintf(
paste0(
"The %s %s at least one parameter with infinite split R-hat: ",
"its sub-chains are constant but differ in value. %s verdicts should ",
"be interpreted only after checking convergence."
"The %s %s at least one parameter with infinite split R-hat. ",
"Check convergence before interpreting %s verdicts."
),
list_of(fit_label(stuck)), if(one) "fit includes" else "fits include", if(one) "Its" else "Their"
list_of(fit_label(stuck)), if(one) "fit has" else "fits have",
if(one) "its" else "their"
))
cat("\n")
}
Expand Down Expand Up @@ -905,12 +878,13 @@ print.bgms_prior_sensitivity = function(x, max_rows = 10L, ...) {
sprintf("%.0f s", x$runtime_seconds)
}
n_refit = sum(!g$original_fit)
start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "cold starts"
start_txt = if(isTRUE(x$warm)) "started from the original fit's final state" else "started from default initial values"
it = g$iter[!g$original_fit][1]
wu = g$warmup[!g$original_fit][1]
sampler_label = if(identical(x$refit_sampler, "nuts")) "NUTS" else x$refit_sampler
cat(sprintf(
"Refits: %d %s refit%s (%d warmup, %d sampling iterations each), %s, %s total.\n",
n_refit, x$refit_sampler, if(n_refit == 1L) "" else "s", wu, it, start_txt, secs
"Refits: %d %s refit%s (%d warmup, %d sampling iterations each), %s, %s total.\n",
n_refit, sampler_label, if(n_refit == 1L) "" else "s", wu, it, start_txt, secs
))
cat("See ?prior_sensitivity_check for the construction.\n")
invisible(x)
Expand Down
Loading
Loading