Phase 6: Report generation

[teresa-ortega]

Proposed changes

Adds examples/beluga/report.ipynb, a Jupyter cookbook that shows how to explore and visualise APE results from a completed Beluga benchmark run using lambkin.data.access and lambkin.data.evo.

The notebook covers:

• Explore available data — list variants and iterations using access.iterations().
• APE timeseries by variant — individual iterations in light color, per-variant mean overlaid in bold.
• Stats summary table — RMSE mean/std, mean, and max per variant, aggregated across iterations.
• RMSE comparison — bar chart with error bars across variants.
• Pandas DataFrame — snippet showing how to convert results to a long-format DataFrame for advanced analysis with seaborn.
• Export to HTML — how to share the notebook as a standalone HTML file with jupyter nbconvert.

No new SDK modules or dependencies are introduced — the notebook uses only the existing lambkin.data API.

Type of change

• 🐛 Bugfix (change which fixes an issue)
• 🚀 Feature (change which adds functionality)
• 📚 Documentation (change which fixes or extends documentation)

💥 No breaking changes

Checklist

Put an x in the boxes that apply. This is simply a reminder of what we will require before merging your code.

• Lint and unit tests (if any) pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Additional comments

N/A

[xaru8145]

Before we go further in the review of this implementation, I want you to think through some scenarios:

1. What happens when a user wants RPE instead of APE?
2. What happens when they want both APE and RPE in the same report?
3. What happens when they only care about RMSE and not the full stats table?

Walk through how a user would achieve each of these with the current report.generate() API. What does that tell you about the generality of this approach?

[teresa-ortega]

You're right, the current report.generate() is too opinionated — it hardcodes APE, always generates all three sections, and can't combine multiple metrics in the same report.

I see two ways to fix this:

Option A : composable cells

Instead of a single generate(), the SDK exposes building blocks that the user composes:

import lambkin.data.report as report

nb = report.notebook()
nb.add(report.timeseries("output.ape.zip", title="APE"))
nb.add(report.timeseries("output.rpe.zip", title="RPE"))
nb.add(report.stats_table("output.ape.zip"))
nb.save(output_dir / "report.ipynb")

This handles all three scenarios cleanly. More API surface, but fully flexible.

Option B : generate() with parameters

Keep a single entry point but make it configurable:

report.generate(
    ctx,
    files=["output.ape.zip", "output.rpe.zip"],
    sections=["timeseries", "stats"],
)

Which direction do you prefer?

[xaru8145]

Framing: The subject should be the notebook as a workflow, not lambkin.data. Something like:

Benchmark results can be explored interactively in a Jupyter notebook after a run. The included report.ipynb provides a starting point; lambkin.data functions accept a plain path so no benchmark context is needed:

[xaru8145]

The code snippet also reads like a Python script — using print() and plt.show() — which works against the notebook framing. In a notebook, cell outputs render inline; you don't need either.

Rather than writing a new snippet, just point to the Beluga example notebook and note that it can be adapted for different metrics. That's faster and more honest about what the user should actually do.

[xaru8145]

in a notebook, outputs render inline, nothing is "printed". Change to "displays" or "shows".

Suggested change

	The notebook plots APE timeseries by variant, prints a stats summary table, and generates an RMSE comparison bar chart. All figures are saved under `results/`.
	The notebook plots APE timeseries by variant, shows a stats summary table, and generates an RMSE comparison bar chart. All figures are saved under `results/`.