CVaR Phase 2/3: CLI + decomposition, maximize support, and docs

[DLWoodruff]

What

Combined Phase 2 + Phase 3 of CVaR risk management (follow-on to #746, which landed the core transform and EF tests). Per the design doc doc/designs/cvar_design.md, this adds the command-line / generic_cylinders surface, completes maximize support, and ships the user docs — so the risk-averse objective λ·E[Cost] + β·CVaR_α(Cost) is usable end-to-end across the EF solve and every cylinder.

Phases 2 and 3 are combined into one PR at the maintainer's request. Opened as a draft.

CLI + decomposition (Phase 2)

• config: cvar_args() adds --cvar, --cvar-weight (β), --cvar-alpha (α), --cvar-mean-weight (λ); wired into mpisppy/generic/parsing.py.
• generic_cylinders: when --cvar is set, scenario_creator is wrapped with cvar_scenario_creator (after the ADMM block; guarded against bundles/ADMM, which it can't yet compose with). Because η is appended to the root node it is "just another first-stage variable," so EF, PH/APH, Lagrangian, subgradient, FWPH, and xhat all inherit risk aversion with no algorithm changes.
• example: examples/farmer/farmer_generic.bash and examples/run_all.py gain risk-averse --cvar invocations.

Maximize support (Phase 3)

add_cvar now handles maximization via the PySP lower-tail mirror: the excess variable becomes NonPositiveReals and the excess constraint flips to δ_s ≤ Cost_s − η. The objective expression and sense are unchanged, so the same --cvar flags maximize λ·E[Reward] + β·CVaR_α of the worst-case (lowest) rewards. This replaces the Phase 1 NotImplementedError guard (which addressed @bknueven's review on #746).

Setting rho with CVaR (important)

The VaR variable η has a much larger cost scale than typical model variables (it lives on the objective scale), so a uniform rho stalls PH. Measured on the 3-scenario farmer (--cvar-weight 2 --cvar-alpha 0.8, EF-CVaR optimum -220700):

rho strategy	result after up to 100 PH iters
--default-rho 1	40% gap — bounds never close (η stuck)
--grad-rho --grad-order-stat 0.5	converges to the optimum (0% gap)
--sep-rho	converges to the optimum (0% gap)

The new docs and example scripts recommend and use a cost-aware rho.

Docs (Phase 3)

New doc/src/risk_management.rst (registered in the toctree): CLI + programmatic usage, the rho/η guidance above, the maximize note, a zhat4xhat confidence-interval note (it evaluates whichever objective is active, so it automatically uses the risk-averse one), and the verbatim single-root-stage / not-time-consistent caveat from design §6.5. CVaR flags also added to generic_cylinders.rst.

Tests

Extended existing (already CI- and run_coverage.bash-wired) files:

• test_cvar.py: maximize closed-form EF (lower-tail) + pure CVaR, a structural maximize check, and a serial PH-on-CVaR run asserting a valid (rho-independent) outer bound. The maximize closed form: rewards {10,20,30,40} uniform, α=0.6 ⇒ E=25, lower-VaR η*=20, lower-CVaR=13.75.
• test_generic_cylinders.py: --EF --cvar through the CLI matches a directly-built EF-CVaR objective.
• test_with_cylinders.py: PH hub + Lagrangian (outer) and + xhatshuffle (inner) bracket the EF-CVaR optimum — the rho-independent bound sandwich.

All green locally (serial via pytest; test_with_cylinders.py via mpiexec -np 2); ruff clean; docs build succeeds.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.63%. Comparing base (32190ff) to head (0f94c09).

Files with missing lines	Patch %	Lines
mpisppy/generic_cylinders.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #751      +/-   ##
==========================================
+ Coverage   74.59%   74.63%   +0.03%     
==========================================
  Files         166      166              
  Lines       21532    21544      +12     
==========================================
+ Hits        16062    16079      +17     
+ Misses       5470     5465       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bknueven]

Any reason for the additional fudge factor? I wouldn't mind 1e-10 or so, but 1e-4 seems like a bug.

[DLWoodruff]

Good catch — no real reason, the relative fudge factor was overkill and could mask a genuinely wrong-side bound. Tightened to 1e-8 * abs(ef_opt) (just round-off absorption) in 120cbe8; both CVaR tests still pass.

[bknueven]

Same question here.

[DLWoodruff]

Same fix here — now 1e-8 * abs(ef_opt) in 120cbe8.

[bknueven]

I would like to spend more time to understand this better, but that shouldn't hold it up, and I might not get said time for at least another week.