Repository navigation
fix(sensitivity): use verdicts() flags and clarify diagnostic reporting - #227
Merged
Merged
Conversation
… one saturation rule for the summary cells and the check prior_sensitivity_check() takes its chosen-scale flag from verdicts() on the original fit, so the two reports count the same edges. The between-chain band and the per-chain verdict test are gone, as are the $edges columns insufficient, insufficient_noisy and insufficient_disagree; the flag is the fragile column. The printed count labels are "verdict held at every scale", "changed within Monte Carlo error", "changed beyond Monte Carlo error" and "could change on a rerun"; the sentence under them names the flagged edges and states that their log Bayes factors lie within two standard errors of an evidence threshold. The split-R-hat kernel applies the ESS kernel's variance floor: a sub-chain with biased variance below 1e-15 counts as constant, so a Rao-Blackwellized inclusion series constant to numerical precision has NA for mcse, n_eff and Rhat together. The check's saturated column is the same criterion (the chosen-scale MCSE is NA). The note under the summary tables and in the check says "0 or 1 in every draw of every chain, or constant to numerical precision"; the Details of summary.bgms, summary.bgmCompare and the diagnostics vignette explain why the series is constant. The sentence for a fit with an infinite split-R-hat says the sampler did not move and that the fit's verdicts do not describe the posterior distribution at that scale; the help page states that rhat_max covers the continuous parameters only and what such a refit needs. The Refits line prints NUTS. ?verdicts states that an indicator whose standard errors are undefined is not flagged. The vignette's saved check object is regenerated.
…t the blank-cell and infinite-R-hat wording The split-R-hat kernel is unchanged from develop: a sub-chain counts as constant only when its draws are exactly equal, so the Rhat of an inclusion series that is constant to numerical precision is computed, as before. The saturated column of prior_sensitivity_check() stays the edges whose chosen-scale Monte Carlo standard error is NA. The Details of summary.bgms and summary.bgmCompare and the diagnostics vignette say that mcse and n_eff are NA when the series has no usable variation and that Rhat is NA only for exactly constant chains; the recorded value for an included edge is the probability of retaining it; the cutoff is a variation of about 3e-8, not a removal probability of 1e-8. The infinite-R-hat sentence states what the diagnostic shows: a parameter constant within every split chain at values that differ between split chains, which the sampler did not explore. The help page says such a refit may need a longer warmup or a cold start. NEWS states that the fourth count is the verdicts() flag plus the edges whose verdict change could not be compared with an error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sensitivity check now uses
verdicts()to flag near-threshold verdicts in the original fit. Itsfragilecolumn replacesinsufficientand the two subcause columns. The fourth printed count also includes changed verdicts whose Monte Carlo error is unavailable; the report states how many additional edges it includes.The
saturatedcolumn now records an unavailable chosen-scale MCSE. This does not necessarily mean the inclusion probability is exactly 0 or 1. Summary notes and documentation explain blank diagnostics without treating them as evidence of zero error.Fits with infinite split R-hat are named, with advice to check convergence before interpreting their verdicts. The help page explains how to increase warmup iterations. The R-hat calculation is unchanged from develop; regression tests protect against losing convergence warnings when parameter units change.
The printed notes, help pages, NEWS, and vignettes use shorter explanations. The
Refits:line capitalizes NUTS and describes default initial values without “cold start” terminology.Validation:
test-mcmc-diagnostics.R,test-prior-sensitivity.R(including slow and certification tiers),test-summary-marking.R, andtest-verdicts.Rpass. Lintr, package styler, andgit diff --checkpass. Documentation regenerated and the summary snapshot updated.