Run the card stages over several eras: one workflow whose branches are the cards

.github/copilot-instructions.md

@@ -52,13 +52,28 @@ histogram?*
 
 - The FLAF era is `era` (`--era`); `period` is refused by FLAF as a parameter, option or keyword
   and survives only as the AnaTuple column.
-- `era` is FLAF's `EraParameter(max_len=1)`: a tuple holding one real era. It is read as
-  `self.era[0]` — `self.era` itself in a path, a label or a command is the tuple, not the name.
-  Requirements pass one real era as a tuple: `X.req(self, era=(e,), …)`.
-- A meta-era (`era_groups:` in the card) never goes into `era`, because FLAF's `Setup` knows only
-  real eras: it goes into `meta_era`, and `era` holds one of its sub-eras as the `Setup` anchor.
-  `datacard_era` (`meta_era`, else `era[0]`) keys the outputs and store paths, and
-  `get_sub_eras()` lists the real eras whose histograms feed a card.
+- Every task here takes several eras (`allow_multi_era = True` on `StatInferenceTask`, without
+  FLAF's era branches): `era` is the sorted tuple of `--era`, which must hold every real era
+  behind the configuration's cards (`card_eras()` raises naming the card and the missing
+  eras). An era's own settings are read through `self.setup_for(era)`; a setting the run reads
+  once through `self.common(getter, what)` or `statinference_settings()`, which raise when
+  the eras disagree. FLAF's `check_era_rename.py`, run in CI, rejects `self.era[...]` and an
+  `era=` keyword in `req()`; `self.global_params`, `self.setup`, `self.fs_*`
+  (except `fs_default`) and `self.datasets` are not checked statically and raise `EraScopeError` only
+  when run over several eras, which the unit tests do (three eras) — flag them in review.
+- The tasks a run starts from that are not workflows (`ResonantLimitsTask`, the plot tasks,
+  `ResonantLimitsAndHistPlotTask`) check the eras against the cards when they are constructed
+  (`check_card_eras`), so the error stops `law run` before scheduling; the card workflows check
+  when they build their map, never in `__init__` (FLAF constructs them over single eras).
+- `CreateDatacardsTask` and `PreprocessShapesTask` are one workflow each whose branches are the
+  cards (`eras:` of the configuration, in its order); `branch_data` is the card era, which may
+  be a group era (`era_groups:`), never in `era` because FLAF's `Setup` knows only real eras.
+  `get_sub_eras(card_era)` lists the real eras whose histograms feed a card; `MergedHists`
+  reads each with `hist_era` from that era's own `fs_HistTuple`.
+- Outputs are keyed on the version and the card, the job state and the logs on the version and
+  the class (`store_parts()`, `remote_log_dir_target()`), never on `--era`, so a run over more
+  eras resumes the same jobs. One datacard configuration per version: the job file is named
+  after the branch numbers, i.e. the positions in `eras:`.
 
 ### Binning optimisation
 
@@ -102,6 +117,6 @@ Verified 2026-08-27; re-check before relying on any of it.
 | Layout | `dc_make/` (datacard construction: `maker.py`, `create_datacards.py`, `process.py`, `model.py`, `uncertainty.py`, `binner.py`), `bin_opt/` (binning optimisation and limits), `common/` (`tools.py`, `param_parse.py`), `law/tasks.py`, `fit_val/`, `config/`, `legacy/`, `test/` (unit tests and their scratch analysis) |
 | Consumers | HH_bbtautau and HH_bbWW use this as a submodule; H_mumu does not |
 | Cards | analysis-side, e.g. `HH_bbWW/config/Datacards/CI_card.yaml` |
-| Tests | `test/test_law_chain.py` (the law tasks on the real FLAF and law classes over a scratch analysis, `test/statinference_fixture.py`: per card the outputs, inputs and payload commands) and `test/test_shape_plots.py` (which histograms a card's shape plots show, through `input_categories`); no ROOT, Combine or dhi; run by `unit-tests` against a pinned FLAF commit |
+| Tests | `test/test_law_chain.py` (the law tasks on the real FLAF and law classes over a scratch analysis, `test/statinference_fixture.py`: per card the outputs, inputs and payload commands), `test/test_card_eras.py` (the card workflows over several eras) and `test/test_shape_plots.py` (which histograms a card's shape plots show, through `input_categories`); no ROOT, Combine or dhi; run by `unit-tests` against a pinned FLAF commit |
 | Workflows | `formatting-check`, `unit-tests`, `repo-sanity-checks` (FLAF's sanity checks and FLAF's `run_tools/check_era_rename.py` for leftovers of the `period` → `era` rename), `trigger-flaf-integration`. Formatting is checked automatically — do not comment on it. A change to an assertion of `test/test_law_chain.py` (not to its helpers) changes what the chain is guaranteed to read, write or run — comment on it |
 | Integration test | Triggered by `@cms-flaf-bot please test`; the pipeline configuration lives in `cms-flaf/FLAF_ci` |

.github/workflows/repo-sanity-checks.yaml

@@ -26,7 +26,7 @@ jobs:
           repository: cms-flaf/FLAF
           # A fixed FLAF commit: the checker's rules change with FLAF, and this repository adopts
           # a new rule set by moving the pin, in a change that also fixes what the new rules find.
-          ref: e8e42edb2f1626a0875c29ceb5be0992942fb887
+          ref: 4d9602cdbcdf6d09ba4566b4f83e3878e39975ca
           path: flaf_check
           sparse-checkout: run_tools/check_era_rename.py
           sparse-checkout-cone-mode: false

.github/workflows/unit-tests.yaml

@@ -46,4 +46,5 @@ jobs:
         run: >-
           python -m pytest -q
           StatInference/test/test_law_chain.py
+          StatInference/test/test_card_eras.py
           StatInference/test/test_shape_plots.py

README.md

@@ -101,13 +101,15 @@ rm -r output/binning_v2/tauTau_2018/workers
 
 `test/test_law_chain.py` checks the law tasks on the real FLAF and law classes over a scratch
 analysis (`test/statinference_fixture.py`): for each card, what it writes, which merged
-histograms it reads and the commands it runs. `test/test_shape_plots.py` checks which
-histograms the shape plots of a card show, for categories read through `input_categories`.
+histograms it reads and the commands it runs. `test/test_card_eras.py` checks the card
+workflows over several eras, and `test/test_shape_plots.py` which histograms the shape plots
+of a card show, for categories read through `input_categories`.
 They need no ROOT, Combine or dhi. The `unit-tests` workflow runs them against a pinned FLAF
 commit; to run them locally, put a FLAF checkout at that commit (`ref:` in
 `.github/workflows/unit-tests.yaml`) next to this one and run from the directory holding both:
 
 ```bash
 pip install pyyaml numpy luigi==3.8.1 law==0.1.21 pytest
-PYTHONPATH=$PWD python -m pytest -q StatInference/test/test_law_chain.py StatInference/test/test_shape_plots.py
+PYTHONPATH=$PWD python -m pytest -q StatInference/test/test_law_chain.py \
+    StatInference/test/test_card_eras.py StatInference/test/test_shape_plots.py
 ```

law/CreateDatacardsTask.py

@@ -6,7 +6,11 @@
 from string import Template
 
 from FLAF.RunKit.run_tools import ps_call
-from FLAF.run_tools.law_customizations import HTCondorWorkflow, copy_param
+from FLAF.run_tools.law_customizations import (
+    EraDependentValueError,
+    HTCondorWorkflow,
+    copy_param,
+)
 
 from StatInference.common.tools import CategoryNaming, InputCategories, importROOT
 
@@ -15,76 +19,64 @@
 
 
 class CreateDatacardsTask(StatInferenceTask, HTCondorWorkflow, law.LocalWorkflow):
-    """The datacards for one era, from the shapes the chain was pointed at.
+    """The datacards of the configuration, one branch per card (`eras:`, in its order).
 
     Runs dc_make/create_datacards.py on whatever `<era>/<variable>/<variable>.root` tree
     it is given -- the configuration's `preprocess:` step if it declares one, otherwise
-    the merged histograms directly -- and does no binning of its own. For a meta-era
-    (`meta_era` set, `era` one of its real sub-eras) it holds every sub-era's
-    histograms at once, which is what the summed shape behind each datacard bin is built
-    from, and why the stacked shape plots are drawn here.
+    the merged histograms directly -- and does no binning of its own. The card of a group
+    era holds every sub-era's histograms at once, which is what the summed shape behind
+    each datacard bin is built from, and why the stacked shape plots are drawn here.
     """
 
     max_runtime = copy_param(HTCondorWorkflow.max_runtime, 2.0)
     n_cpus = copy_param(HTCondorWorkflow.n_cpus, 1)
 
-    # If set, build datacards for this meta-era instead of self.era[0]. self.era[0] must
-    # then be one of its real sub-eras -- it is only used to construct a valid FLAF Setup
-    # (Setup.getGlobal requires a real, known era; a meta-era name isn't one).
-    meta_era = luigi.Parameter(default="")
-
     # Stacked plots of the shapes going into the cards. This task is where they belong:
     # it is the only one holding every sub-era's histograms at once, which is what the
     # merged shape (and hence each datacard bin) is built from.
     make_plots = luigi.BoolParameter(default=True)
 
-    def get_sub_eras(self):
-        """Real eras whose histograms feed into datacard_era: its constituent
-        sub-eras for a meta-era, otherwise just [self.era[0]]."""
-        return self.get_era_groups().get(self.datacard_era, [self.era[0]])
-
-    def input_hist_reqs(self):
-        """{key: task} for the shapes the cards are built from.
+    def input_hist_reqs(self, card_eras):
+        """{key: task} for the shapes the cards of `card_eras` are built from.
 
-        The configuration's `preprocess:` step if it declares one, otherwise the merged
-        histograms directly. Either way what arrives is "<era>/<variable>/<variable>.root",
-        so nothing below here knows which it got.
+        The configuration's `preprocess:` step if it declares one -- its workflow over the
+        same cards (law hands the --branches selection on), or from a branch its branch of
+        the same card -- otherwise the merged histograms directly. Either way what arrives
+        is "<era>/<variable>/<variable>.root", so nothing below here knows which it got.
         """
         if self.preprocess_config():
-            return {
-                "PreprocessShapes": PreprocessShapesTask.req(
-                    self, meta_era=self.meta_era, branches=()
-                )
-            }
+            return {"PreprocessShapes": PreprocessShapesTask.req(self)}
         return {
             f"MergedHists_{era}_{variable}": req
             for (era, variable), req in self.merged_hist_reqs(
-                self.get_sub_eras()
+                self.sub_eras_of(card_eras)
             ).items()
         }
 
-    def workflow_requires(self):
+    def stage_workflow_requires(self):
         # Merged with the base class's rather than replacing them: FLAF's HTCondorWorkflow
         # puts the software bundles a submitted job unpacks in there, and returning only
         # our own inputs drops them, so the jobs start without the bundle they need.
-        reqs = super().workflow_requires()
-        reqs.update(self.input_hist_reqs())
+        reqs = super().stage_workflow_requires()
+        # the selected cards only (the map honours --branches)
+        reqs.update(self.input_hist_reqs(self.get_branch_map().values()))
         return reqs
 
     def requires(self):
-        return list(self.input_hist_reqs().values())
+        return list(self.input_hist_reqs([self.branch_data]).values())
 
     def create_branch_map(self):
-        return {0: None}
+        return dict(enumerate(self.card_eras()))
 
     def output(self):
         # fs_default. Note that combine cannot read these directly:
         # ResonantLimitsTask mirrors them back to datacards_dir() before handing them to
         # dhi -- see ResonantLimitsTask.stage_datacards.
-        return self.output_dir_target(self.version, "Datacards", self.datacard_era)
+        return self.output_dir_target(self.version, "Datacards", self.branch_data)
 
     def run(self):
-        statInf_entry = self.global_params["StatInference"]
+        card_era = self.branch_data
+        statInf_entry = self.statinference_settings()
         config = self.datacard_config_path()
         # ${ERA} in hist_bins names the era the cards are for, the same way the model's
         # input_file_pattern does. Each era has its own binning -- derived from its own
@@ -93,7 +85,7 @@ def run(self):
         hist_bins = (
             os.path.join(
                 self.ana_path(),
-                Template(hist_bins_rel).safe_substitute(ERA=self.datacard_era),
+                Template(hist_bins_rel).safe_substitute(ERA=card_era),
             )
             if hist_bins_rel
             else None
@@ -107,12 +99,14 @@ def run(self):
                 # PreprocessShapesTask already wrote the "<era>/<variable>/<variable>.root"
                 # layout, so its output directory is the base input_file_pattern resolves
                 # against -- nothing to stage.
-                reqs = self.input_hist_reqs()["PreprocessShapes"]
+                reqs = self.input_hist_reqs([card_era])["PreprocessShapes"]
                 base_dir_local = stack.enter_context(reqs.output().localize("r"))
             else:
                 targets = {
                     key: req.output()
-                    for key, req in self.merged_hist_reqs(self.get_sub_eras()).items()
+                    for key, req in self.merged_hist_reqs(
+                        self.get_sub_eras(card_era)
+                    ).items()
                 }
                 self.check_inputs(targets)
                 base_dir_local = stack.enter_context(self.stage_inputs(targets))
@@ -127,7 +121,7 @@ def run(self):
                 "--config",
                 config,
                 "--eras",
-                self.datacard_era,
+                card_era,
             ]
             if hist_bins:
                 cmd += ["--hist-bins", hist_bins]
@@ -165,21 +159,33 @@ def _surviving_axis(base_dir):
         slices = node.get("slices") if isinstance(node, dict) else None
         return 0 if slices and "y_range" in slices[0] else 1
 
-    def plot_variable(self, variable, axis=1):
+    def plot_variable(self, variable, sub_eras, axis=1):
         """The variable whose axis the shapes are binned along, for histograms.yaml.
 
         For 2D->1D rebinned input, the entry of the 2D variable's ``var_list: [x, y]``
-        picked by `axis` (from _surviving_axis()); for 1D input, the variable itself.
+        picked by `axis` (from _surviving_axis()); for 1D input, the variable itself. The
+        plotted shapes are the sum over `sub_eras`, so their histograms.yaml must agree.
         """
-        try:
-            import FLAF.Common.Setup as Setup
-
-            hists = Setup.Setup(self.ana_path(), self.era[0], self.version).hists
-            var_list = hists[variable].get("var_list")
-            if var_list and len(var_list) > 1:
-                return var_list[axis]
-        except Exception as e:
-            print(f"Warning: no var_list for {variable} ({e}); plotting it as itself")
+        # None: the era's histograms.yaml has no entry for the variable
+        var_lists = {}
+        for era in sub_eras:
+            hists = self.setup_for(era).hists
+            try:
+                var_lists[era] = hists[variable].get("var_list")
+            except KeyError:
+                var_lists[era] = None
+        var_list = var_lists[sub_eras[0]]
+        if any(other != var_list for other in var_lists.values()):
+            raise EraDependentValueError(
+                f"{self.get_task_family()}: the var_list of {variable} in histograms.yaml "
+                f"differs between the eras summed in one plot ({var_lists})"
+            )
+        if variable not in self.setup_for(sub_eras[0]).hists:
+            print(
+                f"Warning: no histograms.yaml entry for {variable}; plotting it as itself"
+            )
+        elif var_list and len(var_list) > 1:
+            return var_list[axis]
         return variable
 
     @staticmethod
@@ -298,6 +304,8 @@ def plot_rebinned_shapes(self, base_dir, config, output_dir):
         import glob
 
         cfg = self.get_config_data()
+        card_era = self.branch_data
+        sub_eras = self.get_sub_eras(card_era)
         plots_dir = os.path.join(output_dir, "plots")
         os.makedirs(plots_dir, exist_ok=True)
         plotter = os.path.join(self.ana_path(), "FLAF", "Analysis", "HistPlotter.py")
@@ -309,7 +317,7 @@ def plot_rebinned_shapes(self, base_dir, config, output_dir):
         for variable in self.get_required_variables():
             inputs = [
                 p
-                for era in self.get_sub_eras()
+                for era in sub_eras
                 for p in glob.glob(
                     os.path.join(base_dir, era, variable, f"{variable}.root")
                 )
@@ -376,25 +384,28 @@ def plot_rebinned_shapes(self, base_dir, config, output_dir):
                 "--globalConfig",
                 os.path.join(
                     self.ana_path(),
-                    self.global_params["analysis_config_area"],
+                    self.common(
+                        lambda setup: setup.global_params["analysis_config_area"],
+                        "analysis_config_area",
+                    ),
                     "global.yaml",
                 ),
                 # The surviving axis of the rebinned shapes, and already at its final
                 # binning -- so no --rebin, which would coarsen it back to the histograms.yaml
                 # grid and undo the whole point of the rebinning.
                 "--var",
-                self.plot_variable(variable, self._surviving_axis(base_dir)),
+                self.plot_variable(variable, sub_eras, self._surviving_axis(base_dir)),
                 # The plotted shapes are the sum over every sub-era, so the label has to
                 # name the combination: HistPlotter reads config/plot/<year>.yaml for the
                 # luminosity, and a sub-era's file states that sub-era's luminosity alone.
                 # --era stays a real era -- it builds the Setup, which does not know
                 # meta-era names.
                 "--year",
-                self.datacard_era,
+                card_era,
                 "--ana_path",
                 self.ana_path(),
                 "--era",
-                self.era[0],
+                sub_eras[0],
                 "--LAWrunVersion",
                 self.version,
                 "--wantSignals",

law/MergedHists.py

@@ -18,13 +18,16 @@ class MergedHists(StatInferenceTask, law.ExternalTask):
     """
 
     variable = luigi.Parameter(description="Hists_merged variable (directory) name")
+    # The era the file belongs to, one of `era`: a task over several eras requires the
+    # files of each, each from that era's own fs_HistTuple.
+    hist_era = luigi.Parameter(description="era of the merged histograms")
 
     def output(self):
         return self.remote_target(
             self.input_hists_version,
             "Hists_merged",
-            self.era[0],
+            self.hist_era,
             self.variable,
             f"{self.variable}.root",
-            fs=self.fs_HistTuple,
+            fs=self.setup_for(self.hist_era).get_fs("HistTuple"),
         )

law/PlotPullsAndImpactsTask.py

@@ -40,6 +40,8 @@ class PlotPullsAndImpactsTask(DhiPlotMixin, StatInferenceTask):
     # NO_STR leaves each of those at its own default.
     workflow = luigi.Parameter(default=law.parameter.NO_STR)
 
+    check_card_eras = True
+
     def requires(self):
         # ResonantLimitsTask is what writes the combined cards this reads.
         return [ResonantLimitsTask.req(self)]

law/PlotResonantLimitsTask.py

@@ -43,6 +43,8 @@ class PlotResonantLimitsTask(DhiPlotMixin, StatInferenceTask):
     # NO_STR leaves each of those at its own default.
     workflow = luigi.Parameter(default=law.parameter.NO_STR)
 
+    check_card_eras = True
+
     def requires(self):
         # ResonantLimitsTask is what mirrors the cards to datacards_dir(), which is what
         # the globs below resolve against -- the plots cannot be built before it has run.