Automatic report generation

[SteSeg]

Summary

This PR introduces an end-to-end report generation API for benchmarks, including optional report creation directly from Benchmark.run(). The report workflow builds structured report metadata, renders plots (absolute + C/E), and adds validation quality plots and observable-level summary charts based on verbosity. Reports can be rendered to YAML and PDF.

What is new

• Benchmark.run(..., generate_report=False, report_config=None) now optionally generates a report after a benchmark run.
• Default reference results are pulled from results_database/{benchmark_name}/experiment.h5 when available.
• A complete reporting pipeline:
  • Build report data from ResultSource entries.
  • Render YAML and PDF outputs.
  • Add plot artifacts and PDF pages for absolute, C/E, quality metrics, and observable summaries.

API overview

1) Run-time report generation

bench.run(generate_report=True)

This triggers report generation after the run completes. If report_config is omitted, defaults are:

• output_dir=report/
• include_yaml=True
• include_pdf=True
• verbosity=2

2) Manual report pipeline

Use this when you want to customize the report or run it outside Benchmark.run().

from pathlib import Path
from openmc_fusion_benchmarks.benchmark_results import BenchmarkResults
from openmc_fusion_benchmarks.report import ReportConfig, ResultSource, build_report, render_pdf, render_yaml

reference = BenchmarkResults.from_file("reference_results.h5")
candidate = BenchmarkResults.from_file("benchmark_results.h5")

sources = [
    ResultSource(name="reference", kind="experiment", results=reference),
    ResultSource(name="calculation", kind="calculation", results=candidate),
]

config = ReportConfig(
    output_dir=Path("report"),
    include_yaml=True,
    include_pdf=True,
    verbosity=2,
)

report = build_report(sources, config)
render_yaml(report, config.output_dir / "report.yaml")
render_pdf(report, config.output_dir / "report.pdf", config.output_dir / "plots")

3) Config objects

• ReportConfig:

  • output_dir: output directory root.
  • include_yaml: enable YAML output.
  • include_pdf: enable PDF output.
  • plot_tallies: optional list of tallies to plot.
  • verbosity: controls level of detail for plots/sections.

• ResultSource:

  • name: display name.
  • kind: typically experiment (reference) or calculation (candidate).
  • results: a BenchmarkResults instance.
  • tally_names: optional list of tallies to include.

Workflow details

Build report data

build_report() collects:

• benchmark metadata from specifications and run metadata
• source metadata (file paths, tally list)
• plot definitions for each selected tally

Render outputs

• render_yaml() writes a machine-readable report summary.
• render_pdf() composes a full PDF report:
  • Summary page (benchmark title, reference, validation case)
  • Optional specification summary (verbosity > 0)
  • Per-tally plots for absolute values + C/E
  • Observable/tally-level summary charts
  • Quality evaluation plots (validation metrics)

Validation quality plots (verbosity-driven)

A dedicated "Quality evaluation" section is added with per-point metrics. Metrics included depend on verbosity:

• Verbosity 0: C/E
• Verbosity 1: C/E, chi2 contribution
• Verbosity 2 (default): C/E, relative deviation, combined uncertainty, chi2 contribution
• Verbosity 3: C/E, relative deviation, absolute deviation, combined uncertainty, normalized residual, chi2 contribution

Each plot includes the mathematical expression and a short description of the metric.

Observable summary charts

A dedicated "Observable summary" page aggregates per-tally metrics into bar charts. Metrics depend on verbosity:

• Verbosity 0: rms_relative_deviation
• Verbosity 1: rms_relative_deviation, reduced_chi2
• Verbosity 2: mean_bias, mean_abs_relative_deviation, rms_relative_deviation, reduced_chi2
• Verbosity 3: adds mean_abs_normalized_residual

Notes

• The report uses calculation results from benchmark_results.h5 and reference results from experiment.h5 if available.
• Grading/status metrics are intentionally disabled in report output; only quantitative metrics are shown.
• Report generation is non-blocking for workflows that do not require reporting (default is off).

[coveralls]

Coverage Report for CI Build 26114921953

Coverage decreased (-0.9%) to 68.415%

Details

• Coverage decreased (-0.9%) from the base build.
• Patch coverage: No coverable lines changed in this PR.
• 283 coverage regressions across 3 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

283 previously-covered lines in 3 files lost coverage.

File	Lines Losing Coverage	Coverage
/usr/share/miniconda3/envs/test-env/lib/python3.12/site-packages/openmc_fusion_benchmarks/report/renderers.py	177	18.3%
/usr/share/miniconda3/envs/test-env/lib/python3.12/site-packages/openmc_fusion_benchmarks/report/plots.py	55	50.0%
/usr/share/miniconda3/envs/test-env/lib/python3.12/site-packages/openmc_fusion_benchmarks/benchmark.py	51	84.02%

---

Coverage Stats

Relevant Lines:	2479
Covered Lines:	1696
Line Coverage:	68.41%
Coverage Strength:	0.68 hits per line

---

💛 - Coveralls