NaN-handling for Energy Score#123
Open
frazane wants to merge 5 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds nan_policy handling to the multivariate Energy Score family (ES/twES/owES/vrES), mirroring the existing CRPS NaN-handling approach. It wires NaN member omission through the existing weighted estimators (by zero-weighting invalid members) and fixes two weighted numba energy kernels that become exercised by the new omit path.
Changes:
- Added a
nan_policyparameter toes_ensemble,twes_ensemble,owes_ensemble, andvres_ensemble, and integrated it into the scoring flow (including proper interaction withens_w). - Reworked
apply_nan_policy_ens_mvto align/maskens_wand to supportpropagate/raise/omitconsistently with the univariate helper. - Fixed weighted numba energy gufunc issues (scalar indexing of
ow, andwbarnormalisation for outcome-weighted energy) and added comprehensive tests across policies and weight scenarios.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_wenergy.py |
Adds NaN-policy tests for owES/vrES/twES and a uniform-ens_w equivalence guard for the weighted kernels. |
tests/test_energy.py |
Adds NaN-policy coverage for es_ensemble, including NaNs in forecasts, NaNs in ens_w, and estimator-specific omit behavior. |
scoringrules/core/utils.py |
Reworks apply_nan_policy_ens_mv to return aligned ens_w and implement omit-by-zero-weighting for invalid members. |
scoringrules/core/energy/_gufuncs_w.py |
Fixes weighted numba ow/vr energy gufunc bugs (scalar indexing and wbar computation). |
scoringrules/_energy.py |
Threads nan_policy through public Energy Score APIs and routes omit cases through the weighted code paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
nan_policysupport to the multivariate Energy Score family, following the same approach introduced for the ensemble CRPS in #118. This is the first multivariate metric to gain NaN handling; the pattern is meant to carry over to the remaining multivariate scores.As with CRPS, a member is treated as invalid if any of its variables (or its weight) is NaN, and
nan_policyacceptspropagate(default),raise, andomit. Theomitpolicy zero-weights invalid members and routes the score through the existing weighted estimators, so it reduces to wiring rather than new numerics. NaN in the observations always propagates.What changed
nan_policytoes_ensemble,twes_ensemble,owes_ensembleandvres_ensemble.apply_nan_policy_ens_mvhelper (added in NaN-handling for ensemble CRPS #118 but unused and untested) to mirror the univariateapply_nan_policy_ens_uv: it now folds invalid members into the ensemble weights and returns the alignedens_w.ens_wis actually exercised on the numba backend (which theomitpath now does): a scalar-indexing error (ow = ow[0]) in the outcome-weighted and vertically-rescaled kernels, and a wrong normalisation (np.mean(fw)instead ofnp.sum(fw * ens_w)) in the outcome-weighted kernel.omitagainst the score computed over the surviving members (for both thenrgandfairestimators), NaN weights, and the weighted-ens_wpaths that the kernel fixes depend on.Notes
omitis not implemented for theakrandakr_circpermestimators (they are roll/permutation-based and cannot be expressed through member weighting), matching how CRPS treats itsakr/intestimators. They raiseNotImplementedError.ow = ow[0]pattern still exists in the weighted multivariate kernel-score gufuncs (core/kernels/_gufuncs_w.py); left as a follow-up since it is outside the Energy Score scope here.