fix: latent_variables guide reuses quick fit + draws from PDF (PyAutoLabs/autolens_workspace#230) (#123)

Consistency change mirroring the autolens_workspace fix. The guide ran a
bespoke Nautilus fit and computed latents over the full posterior; now it reuses
the shared `_quick_fit.py` result via the aggregator and computes latents over
20 PDF draws (`samples.samples_drawn_randomly_via_pdf_from`). Adds a
"Controlling the Cost via Config" docstring (latent.yaml = which latents,
output.yaml latent_draw_via_pdf = how many draws).

Cold run 63s. Notebook regenerated.


Claude-Session: https://claude.ai/code/session_01XwGFQNneASyEoHJm3RGecm

Co-authored-by: Jammy2211 <JNightingale2211@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: 3 people

llms-full.txt

@@ -231,7 +231,7 @@ AUTO-GENERATED by PyAutoBuild — do not edit by hand; regenerate with generate.
 - [Database: Introduction](scripts/guides/results/database/start_here.py): The default behaviour of model-fitting results output is to be written to hard-disc in folders. These are simple to navigate and manually check.
   - Contents: Unique Identifiers, Dataset, Results From Hard Disk, Building a Database File From an Output Folder, Writing Directly To Database, Files, Generators, Model, Search, Samples, Wrap Up
 - [Results: Latent Variables](scripts/guides/results/latent_variables.py): A latent variable is a quantity derived from the model parameters that is not itself sampled by the non-linear search — its posterior is induced by the posterior over the parameters. PyAutoGalaxy ships a curated catalogue of galaxy-level latent variables out of the box, controlled by ``autogalaxy/config/latent.yaml``. This tutorial shows what the catalogue contains, how to toggle individual latents on or off via the workspace config, how to load latent results from a completed fit, and how to extend ``AnalysisImaging`` with a user-defined latent when the curated set isn't enough.
-  - Contents: Galaxy Latents in PyAutoGalaxy: The curated default catalogue from ``autogalaxy/config/latent.yaml``., Toggling Latents: The workspace ``config/latent.yaml`` override and why it shadows the library default., Model Fit: A quick fit that produces real latent output for the loading section below., Loading Latent Results: Reading ``latent/samples.csv`` / ``latent_summary.json`` via, Extending with a Custom Latent: Subclass ``ag.AnalysisImaging``, override ``LATENT_KEYS`` and, Contributing Upstream: When your custom latent is general enough, consider promoting it to the library.
+  - Contents: Galaxy Latents in PyAutoGalaxy: The curated default catalogue from ``autogalaxy/config/latent.yaml``., Toggling Latents: The workspace ``config/latent.yaml`` override and why it shadows the library default., Model Fit: Reuse the shared quick fit (``_quick_fit.py``) that produces real latent output., Loading Latent Results: ``analysis.compute_latent_samples`` over a subset of PDF draws, and the two, Extending with a Custom Latent: Subclass ``ag.AnalysisImaging``, override ``LATENT_KEYS`` and, Contributing Upstream: When your custom latent is general enough, consider promoting it to the library.
 - [Results: Start Here](scripts/guides/results/start_here.py): After a model-fit completes, nearly everything a user could need is written to the `output/` folder. Most of it can be loaded back into full Python objects with a single line of code, via either `.json` files (for model objects like the `Galaxies`, `Model` and samples) or `.fits` files (for imaging products like the model image, residuals and per-galaxy images).
   - Contents: Model Fit, Info
 - [Results: CSV](scripts/guides/results/workflow/csv_make.py): This example is a results workflow example, which means it provides tool to set up an effective workflow inspecting and interpreting the large libraries of modeling results.

notebooks/guides/results/latent_variables.ipynb

@@ -24,9 +24,9 @@
         "\n",
         " - Galaxy Latents in PyAutoGalaxy: The curated default catalogue from ``autogalaxy/config/latent.yaml``.\n",
         " - Toggling Latents: The workspace ``config/latent.yaml`` override and why it shadows the library default.\n",
-        " - Model Fit: A quick fit that produces real latent output for the loading section below.\n",
-        " - Loading Latent Results: Reading ``latent/samples.csv`` / ``latent_summary.json`` via\n",
-        "   ``analysis.compute_latent_samples(result.samples)`` and the standard ``Samples`` API.\n",
+        " - Model Fit: Reuse the shared quick fit (``_quick_fit.py``) that produces real latent output.\n",
+        " - Loading Latent Results: ``analysis.compute_latent_samples`` over a subset of PDF draws, and the two\n",
+        "   config surfaces (``latent.yaml`` / ``output.yaml``) that control which latents and how many draws.\n",
         " - Extending with a Custom Latent: Subclass ``ag.AnalysisImaging``, override ``LATENT_KEYS`` and\n",
         "   ``compute_latent_variables`` to add your own derived quantity.\n",
         " - Contributing Upstream: When your custom latent is general enough, consider promoting it to the library."
@@ -115,14 +115,13 @@
         "'''\n",
         "__Model Fit__\n",
         "\n",
-        "To make the loading and extending sections below concrete, we run a quick model fit on the standard simple-\n",
-        "imaging dataset that ships with the workspace. The model is a single galaxy with a Sersic bulge \u2014 keeping it\n",
-        "small so the example runs in a reasonable time.\n",
+        "The loading and extending sections below need a completed fit to read latents from. Rather than run a bespoke\n",
+        "fit here, we reuse the shared quick fit that the other results guides use: ``_quick_fit.py`` writes a capped\n",
+        "single-galaxy (Sersic bulge + Exponential disk) fit of the standard ``simple`` imaging dataset to\n",
+        "``output/results_folder/``. It is idempotent \u2014 it returns immediately if those results already exist \u2014 so the\n",
+        "non-linear search is paid once across the whole guide suite rather than repeated in every example.\n",
         "\n",
-        "We pass ``magzero=25.0`` to ``ag.AnalysisImaging`` so ``total_galaxy_0_flux_mujy`` populates with a real value\n",
-        "rather than NaN. If you forget it, ``total_galaxy_0_flux_mujy`` will be NaN and the library will log a single\n",
-        "warning per process noting the conversion was skipped \u2014 the fit itself is unaffected, and the raw\n",
-        "``total_galaxy_0_flux`` column populates normally.\n",
+        "We then load that fit's samples via the aggregator, exactly as ``start_here.py`` and ``aggregator/models.py`` do.\n",
         "'''"
       ],
       "outputs": [],
@@ -132,17 +131,38 @@
       "cell_type": "code",
       "metadata": {},
       "source": [
-        "dataset_name = \"simple\"\n",
-        "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name\n",
+        "import subprocess\n",
+        "import sys\n",
+        "\n",
+        "subprocess.run(\n",
+        "    [sys.executable, \"scripts/guides/results/_quick_fit.py\"],\n",
+        "    check=True,\n",
+        ")\n",
         "\n",
-        "if not dataset_path.exists():\n",
-        "    import subprocess\n",
-        "    import sys\n",
+        "from autofit.aggregator.aggregator import Aggregator\n",
         "\n",
-        "    subprocess.run(\n",
-        "        [sys.executable, \"scripts/imaging/simulator.py\"],\n",
-        "        check=True,\n",
-        "    )\n",
+        "agg = Aggregator.from_directory(directory=Path(\"output\") / \"results_folder\")\n",
+        "samples = list(agg)[0].samples"
+      ],
+      "outputs": [],
+      "execution_count": null
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The samples carry the parameter posterior; the ``analysis`` carries the machinery that turns each posterior\n",
+        "draw into latent values. We rebuild the dataset and an ``ag.AnalysisImaging`` with ``magzero=25.0`` so\n",
+        "``total_galaxy_0_flux_mujy`` populates with a real value rather than NaN (without it that column is NaN and the\n",
+        "library logs a single warning per process; the raw ``total_galaxy_0_flux`` column populates normally)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {},
+      "source": [
+        "dataset_name = \"simple\"\n",
+        "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name\n",
         "\n",
         "dataset = ag.Imaging.from_fits(\n",
         "    data_path=dataset_path / \"data.fits\",\n",
@@ -158,18 +178,7 @@
         ")\n",
         "dataset = dataset.apply_mask(mask=mask)\n",
         "\n",
-        "galaxy_model = af.Model(ag.Galaxy, redshift=0.5, bulge=af.Model(ag.lp.Sersic))\n",
-        "model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))\n",
-        "\n",
-        "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=False, magzero=25.0)\n",
-        "\n",
-        "search = af.Nautilus(\n",
-        "    name=\"cookbook_latent_variables\",\n",
-        "    n_live=50,\n",
-        "    n_like_max=300,\n",
-        ")\n",
-        "\n",
-        "result = search.fit(model=model, analysis=analysis)"
+        "analysis = ag.AnalysisImaging(dataset=dataset, use_jax=False, magzero=25.0)"
       ],
       "outputs": [],
       "execution_count": null
@@ -180,21 +189,34 @@
       "source": [
         "__Loading Latent Results__\n",
         "\n",
-        "The fit above produces ``latent/samples.csv`` and ``latent/latent_summary.json`` under the search's output\n",
-        "directory. You can read them back into a ``Samples`` object via ``analysis.compute_latent_samples(result.samples)``.\n",
-        "The returned object exposes the same API as the parameter ``Samples`` \u2014 ``median_pdf``, ``max_log_likelihood``,\n",
-        "``values_at_sigma_1``, and so on \u2014 but reports on the induced latent posterior rather than the parameter posterior.\n",
+        "``analysis.compute_latent_samples(samples)`` reads the posterior into a ``Samples`` object that exposes the same\n",
+        "API as the parameter ``Samples`` \u2014 ``median_pdf``, ``max_log_likelihood``, ``values_at_sigma_1``, and so on \u2014 but\n",
+        "reports on the induced latent posterior. Because both ``total_galaxy_0_flux`` and ``total_galaxy_0_flux_mujy`` are\n",
+        "enabled in this workspace, the returned instance exposes both attributes; enabling additional latents in the\n",
+        "workspace yaml makes them all appear as attributes on the same instance.\n",
+        "\n",
+        "__Controlling the Cost via Config__\n",
+        "\n",
+        "Latents are computed by reconstructing a fit for every posterior sample, so the cost scales with the number of\n",
+        "samples. Two workspace config files control this:\n",
+        "\n",
+        " - ``config/latent.yaml`` \u2014 controls *which* latents are computed. Disable the ones you don't need.\n",
+        "\n",
+        " - ``config/output.yaml`` \u2014 ``latent_draw_via_pdf`` / ``latent_draw_via_pdf_size`` control *how many* posterior\n",
+        "   draws the latents are computed over when a live search updates. Drawing a representative subset from the PDF\n",
+        "   gives faithful latent errors at a fraction of the every-sample cost.\n",
         "\n",
-        "Because both ``total_galaxy_0_flux`` and ``total_galaxy_0_flux_mujy`` are enabled in this workspace, the\n",
-        "returned instance exposes both attributes. If you enable additional latents in the workspace yaml, they all\n",
-        "appear as attributes on the same instance."
+        "Here we mirror that draw-from-PDF behaviour explicitly with ``samples.samples_drawn_randomly_via_pdf_from``,\n",
+        "computing the latents over 20 PDF draws so this guide runs quickly while still producing a real, representative\n",
+        "latent posterior. For a publication-quality result, compute over all samples (or a larger number of draws)."
       ]
     },
     {
       "cell_type": "code",
       "metadata": {},
       "source": [
-        "latent_samples = analysis.compute_latent_samples(result.samples)\n",
+        "latent_draws = samples.samples_drawn_randomly_via_pdf_from(total_draws=20)\n",
+        "latent_samples = analysis.compute_latent_samples(latent_draws)\n",
         "\n",
         "median_instance = latent_samples.median_pdf()\n",
         "print(f\"Median PDF total_galaxy_0_flux: {median_instance.total_galaxy_0_flux}\")\n",

scripts/guides/results/latent_variables.py

@@ -19,9 +19,9 @@
 
  - Galaxy Latents in PyAutoGalaxy: The curated default catalogue from ``autogalaxy/config/latent.yaml``.
  - Toggling Latents: The workspace ``config/latent.yaml`` override and why it shadows the library default.
- - Model Fit: A quick fit that produces real latent output for the loading section below.
- - Loading Latent Results: Reading ``latent/samples.csv`` / ``latent_summary.json`` via
-   ``analysis.compute_latent_samples(result.samples)`` and the standard ``Samples`` API.
+ - Model Fit: Reuse the shared quick fit (``_quick_fit.py``) that produces real latent output.
+ - Loading Latent Results: ``analysis.compute_latent_samples`` over a subset of PDF draws, and the two
+   config surfaces (``latent.yaml`` / ``output.yaml``) that control which latents and how many draws.
  - Extending with a Custom Latent: Subclass ``ag.AnalysisImaging``, override ``LATENT_KEYS`` and
    ``compute_latent_variables`` to add your own derived quantity.
  - Contributing Upstream: When your custom latent is general enough, consider promoting it to the library.
@@ -84,26 +84,35 @@
 """
 __Model Fit__
 
-To make the loading and extending sections below concrete, we run a quick model fit on the standard simple-
-imaging dataset that ships with the workspace. The model is a single galaxy with a Sersic bulge — keeping it
-small so the example runs in a reasonable time.
+The loading and extending sections below need a completed fit to read latents from. Rather than run a bespoke
+fit here, we reuse the shared quick fit that the other results guides use: ``_quick_fit.py`` writes a capped
+single-galaxy (Sersic bulge + Exponential disk) fit of the standard ``simple`` imaging dataset to
+``output/results_folder/``. It is idempotent — it returns immediately if those results already exist — so the
+non-linear search is paid once across the whole guide suite rather than repeated in every example.
 
-We pass ``magzero=25.0`` to ``ag.AnalysisImaging`` so ``total_galaxy_0_flux_mujy`` populates with a real value
-rather than NaN. If you forget it, ``total_galaxy_0_flux_mujy`` will be NaN and the library will log a single
-warning per process noting the conversion was skipped — the fit itself is unaffected, and the raw
-``total_galaxy_0_flux`` column populates normally.
+We then load that fit's samples via the aggregator, exactly as ``start_here.py`` and ``aggregator/models.py`` do.
 """
-dataset_name = "simple"
-dataset_path = Path("dataset") / "imaging" / dataset_name
+import subprocess
+import sys
+
+subprocess.run(
+    [sys.executable, "scripts/guides/results/_quick_fit.py"],
+    check=True,
+)
+
+from autofit.aggregator.aggregator import Aggregator
 
-if not dataset_path.exists():
-    import subprocess
-    import sys
+agg = Aggregator.from_directory(directory=Path("output") / "results_folder")
+samples = list(agg)[0].samples
 
-    subprocess.run(
-        [sys.executable, "scripts/imaging/simulator.py"],
-        check=True,
-    )
+"""
+The samples carry the parameter posterior; the ``analysis`` carries the machinery that turns each posterior
+draw into latent values. We rebuild the dataset and an ``ag.AnalysisImaging`` with ``magzero=25.0`` so
+``total_galaxy_0_flux_mujy`` populates with a real value rather than NaN (without it that column is NaN and the
+library logs a single warning per process; the raw ``total_galaxy_0_flux`` column populates normally).
+"""
+dataset_name = "simple"
+dataset_path = Path("dataset") / "imaging" / dataset_name
 
 dataset = ag.Imaging.from_fits(
     data_path=dataset_path / "data.fits",
@@ -119,32 +128,34 @@
 )
 dataset = dataset.apply_mask(mask=mask)
 
-galaxy_model = af.Model(ag.Galaxy, redshift=0.5, bulge=af.Model(ag.lp.Sersic))
-model = af.Collection(galaxies=af.Collection(galaxy=galaxy_model))
-
 analysis = ag.AnalysisImaging(dataset=dataset, use_jax=False, magzero=25.0)
 
-search = af.Nautilus(
-    name="cookbook_latent_variables",
-    n_live=50,
-    n_like_max=300,
-)
-
-result = search.fit(model=model, analysis=analysis)
-
 """
 __Loading Latent Results__
 
-The fit above produces ``latent/samples.csv`` and ``latent/latent_summary.json`` under the search's output
-directory. You can read them back into a ``Samples`` object via ``analysis.compute_latent_samples(result.samples)``.
-The returned object exposes the same API as the parameter ``Samples`` — ``median_pdf``, ``max_log_likelihood``,
-``values_at_sigma_1``, and so on — but reports on the induced latent posterior rather than the parameter posterior.
+``analysis.compute_latent_samples(samples)`` reads the posterior into a ``Samples`` object that exposes the same
+API as the parameter ``Samples`` — ``median_pdf``, ``max_log_likelihood``, ``values_at_sigma_1``, and so on — but
+reports on the induced latent posterior. Because both ``total_galaxy_0_flux`` and ``total_galaxy_0_flux_mujy`` are
+enabled in this workspace, the returned instance exposes both attributes; enabling additional latents in the
+workspace yaml makes them all appear as attributes on the same instance.
+
+__Controlling the Cost via Config__
+
+Latents are computed by reconstructing a fit for every posterior sample, so the cost scales with the number of
+samples. Two workspace config files control this:
+
+ - ``config/latent.yaml`` — controls *which* latents are computed. Disable the ones you don't need.
+
+ - ``config/output.yaml`` — ``latent_draw_via_pdf`` / ``latent_draw_via_pdf_size`` control *how many* posterior
+   draws the latents are computed over when a live search updates. Drawing a representative subset from the PDF
+   gives faithful latent errors at a fraction of the every-sample cost.
 
-Because both ``total_galaxy_0_flux`` and ``total_galaxy_0_flux_mujy`` are enabled in this workspace, the
-returned instance exposes both attributes. If you enable additional latents in the workspace yaml, they all
-appear as attributes on the same instance.
+Here we mirror that draw-from-PDF behaviour explicitly with ``samples.samples_drawn_randomly_via_pdf_from``,
+computing the latents over 20 PDF draws so this guide runs quickly while still producing a real, representative
+latent posterior. For a publication-quality result, compute over all samples (or a larger number of draws).
 """
-latent_samples = analysis.compute_latent_samples(result.samples)
+latent_draws = samples.samples_drawn_randomly_via_pdf_from(total_draws=20)
+latent_samples = analysis.compute_latent_samples(latent_draws)
 
 median_instance = latent_samples.median_pdf()
 print(f"Median PDF total_galaxy_0_flux: {median_instance.total_galaxy_0_flux}")

workspace_index.json

@@ -693,16 +693,19 @@
     "contents": [
       "Galaxy Latents in PyAutoGalaxy: The curated default catalogue from ``autogalaxy/config/latent.yaml``.",
       "Toggling Latents: The workspace ``config/latent.yaml`` override and why it shadows the library default.",
-      "Model Fit: A quick fit that produces real latent output for the loading section below.",
-      "Loading Latent Results: Reading ``latent/samples.csv`` / ``latent_summary.json`` via",
+      "Model Fit: Reuse the shared quick fit (``_quick_fit.py``) that produces real latent output.",
+      "Loading Latent Results: ``analysis.compute_latent_samples`` over a subset of PDF draws, and the two",
       "Extending with a Custom Latent: Subclass ``ag.AnalysisImaging``, override ``LATENT_KEYS`` and",
       "Contributing Upstream: When your custom latent is general enough, consider promoting it to the library."
     ],
     "cross_refs": [
       "../../../autofit_workspace/scripts/cookbooks/latent_variables.py",
+      "_quick_fit.py",
+      "aggregator/models.py",
       "autogalaxy/imaging/model/latent.py",
       "euclid_strong_lens_modeling_pipeline/util.py",
       "scripts/guides/units/flux.py",
+      "start_here.py",
       "test_autogalaxy/imaging/test_latent.py"
     ],
     "notebook": "notebooks/guides/results/latent_variables.ipynb",