[Backport to 17_0_X (#50691)] SoA Schema evolution by Electricks94 · Pull Request #51316 · cms-sw/cmssw

Electricks94 · 2026-06-25T07:35:05Z

PR description:

Extension of the custom streamer of the SoA Backend to encounter for the following cases:

Added columns/eigen columns/scalars are initialised to 0
Removed or rearranged columns are handle correctly by ROOT
columns with changed type (e.g. double to float) are handle by ROOTm this holds also for Eigen columns and scalars
Eigen columns with changed dimensions for example Eigen::Matrix<float, 4, 2> -> Eigen::Matrix<float, 3, 2>; result in a meaningfull error at reading
Columns with enum types are stored as integers in the SoA backend and only exposed as enums. Hence, the schema evolution behaves exactly like in the case of integers and also 8 byte large enums are supported which is not case with ROOT normally (see Add support for enums with non-default size root-project/root#17009)
Examples using ioread rules are provided to showcase how complex cases (e.g. using custom types for SoA columns) can be handled

PR validation:

A set of SoA Layouts is provided which evolve from a base SoA Layout. The base SoA Layout is stored multiple times using a different type alias which is then changed in the code to simulate an evolving class. The created ROOT files are then used to validate the reading of the previously stored Collections.

This PR is an extension of #50487

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This is a backport of #50691 to 17_0_X

@felicepantaleo fyi

cmsbuild · 2026-06-25T07:35:34Z

cms-bot internal usage

cmsbuild · 2026-06-25T07:37:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51316/49931

Found files with invalid states:
- DataFormats/PortableTestObjects/test/SoAReader_cfg.py:
  - Added: dd0da50
  - Deleted: 6753a15
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionSoAProducer.cc:
  - Added: 6753a15
  - Modified: 6c27bd8, c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/EvolutionTwoAnalyzer.cc:
  - Added: 6753a15
  - Modified: 6c27bd8, c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/EvolutionFourAnalyzer.cc:
  - Added: 6c27bd8
  - Modified: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/EvolutionFiveAnalyzer.cc:
  - Added: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SchemaEvolutionFiveReader.py:
  - Added: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SchemaEvolutionZeroReader.py:
  - Added: 6753a15
  - Modified: 6c27bd8
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SchemaEvolutionTwoReader.py:
  - Added: 6753a15
  - Modified: 6c27bd8
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionAnalyzer.cc:
  - Added: dd0da50
  - Modified: 37a3a70, cbce0c9
  - Deleted: 6753a15
- DataFormats/PortableTestObjects/test/SchemaEvolutionFourReader.py:
  - Added: 6c27bd8
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/EvolutionOneAnalyzer.cc:
  - Added: 6753a15
  - Modified: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SchemaEvolutionThreeReader.py:
  - Added: 6c27bd8
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/EvolutionThreeAnalyzer.cc:
  - Added: 6c27bd8
  - Modified: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/SchemaEvolutionProducer.cc:
  - Added: dd0da50
  - Modified: 37a3a70, cbce0c9
  - Deleted: 6753a15
- DataFormats/PortableTestObjects/interface/SchemaEvolutionSoA.h:
  - Added: dd0da50
  - Modified: 37a3a70, cbce0c9, 6753a15, 6c27bd8, c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/plugins/EvolutionZeroAnalyzer.cc:
  - Added: 6753a15
  - Modified: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SchemaEvolutionWriter.py:
  - Added: 6753a15
  - Modified: 6c27bd8, c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/interface/SchemaEvolutionHostCollection.h:
  - Added: dd0da50
  - Modified: 37a3a70, 6753a15, 6c27bd8, c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SoAWriter_cfg.py:
  - Added: dd0da50
  - Deleted: 6753a15
- DataFormats/PortableTestObjects/test/TestSchemaEvolution.sh:
  - Added: 6c27bd8
  - Modified: c869640
  - Deleted: 94a1e25
- DataFormats/PortableTestObjects/test/SchemaEvolutionOneReader.py:
  - Added: 6753a15
  - Modified: 6c27bd8
  - Deleted: 94a1e25

cmsbuild · 2026-06-25T07:37:32Z

A new Pull Request was created by @Electricks94 for CMSSW_17_0_X.

It involves the following packages:

DataFormats/Portable (heterogeneous)
DataFormats/SoATemplate (heterogeneous)
HeterogeneousCore/TestModules (****)

The following packages do not have a category, yet:

HeterogeneousCore/TestModules
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@cmsbuild, @fwyzard, @makortel can you please review it and eventually sign? Thanks.
@makortel, @missirol, @mmusich, @rovere this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

Backported from SoA Schema evolution #50691

fwyzard · 2026-06-25T13:27:34Z

type ngt

fwyzard · 2026-06-25T13:27:38Z

backport #50691

fwyzard · 2026-06-25T13:29:42Z

enable gpu

fwyzard · 2026-06-25T13:29:45Z

please test

fwyzard · 2026-06-25T14:40:31Z

+heterogeneous

cmsbuild · 2026-06-25T17:19:09Z

-1

Failed Tests: RelVals-AMD_MI300X
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3bc521/54272/summary.html
COMMIT: 23ae7c9
CMSSW: CMSSW_17_0_X_2026-06-25-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51316/54272/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_MI300X

34634.40334634.403_TTbar_14TeV+Run4D121PU_Patatrack_PixelOnlyAlpaka_Validation/step2_TTbar_14TeV+Run4D121PU_Patatrack_PixelOnlyAlpaka_Validation.log
34634.75134634.751_TTbar_14TeV+Run4D121PU_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D121PU_HLT75e33TimingAlpaka.log

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 54
DQMHistoTests: Total histograms compared: 4337615
DQMHistoTests: Total failures: 45
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4337550
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 53 files compared)
Checked 232 log files, 203 edm output root files, 54 DQM output files
TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

You potentially removed 25 lines from the logs
Reco comparison results: 389 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 222745
DQMHistoTests: Total failures: 33995
DQMHistoTests: Total nulls: 31
DQMHistoTests: Total successes: 188719
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 3 / 12 workflows

NVIDIA_H100 Comparison Summary

Summary:

You potentially removed 5 lines from the logs
Reco comparison results: 380 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 222745
DQMHistoTests: Total failures: 29185
DQMHistoTests: Total nulls: 33
DQMHistoTests: Total successes: 193527
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 2 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

You potentially removed 14 lines from the logs
Reco comparison results: 417 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 222745
DQMHistoTests: Total failures: 31747
DQMHistoTests: Total nulls: 31
DQMHistoTests: Total successes: 190967
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 3 / 12 workflows

NVIDIA_T4 Comparison Summary

Summary:

You potentially removed 10 lines from the logs
Reco comparison results: 401 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 222745
DQMHistoTests: Total failures: 29754
DQMHistoTests: Total nulls: 34
DQMHistoTests: Total successes: 192957
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34634.712_TTbar_14TeV+Run4D121PU_lstOnGPUIters01TrackingOnly step3 max memory diff 76.3 exceeds +/- 30.0 MiB

fwyzard and others added 10 commits June 25, 2026 09:32

Extend SoA read streamer to support schema evolution

7313cfa

Start working on test

dd0da50

Added check for Eigen type evolution

37a3a70

Store enum as integer in SoA backend

cbce0c9

Added test case

6753a15

Added test cases and documentation to SoA schema evolution

6c27bd8

Extended Schema Evolution with custom io rules

c869640

Extended Documentation

89bda4c

Move Schemaevolution tests to HeterogeniousCore

94a1e25

made README in DataFormats Portable clearer

23ae7c9

cmsbuild added this to the CMSSW_17_0_X milestone Jun 25, 2026

cmsbuild added pending-signatures tests-pending orp-pending new-package-pending code-checks-pending heterogeneous-pending changes-dataformats labels Jun 25, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 25, 2026

cmsbuild added backport ngt labels Jun 25, 2026

cmsbuild removed the tests-pending label Jun 25, 2026

cmsbuild added the tests-started label Jun 25, 2026

cmsbuild added heterogeneous-approved and removed heterogeneous-pending labels Jun 25, 2026

cmsbuild added tests-rejected and removed tests-started labels Jun 25, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Backport to 17_0_X (#50691)] SoA Schema evolution#51316

[Backport to 17_0_X (#50691)] SoA Schema evolution#51316
Electricks94 wants to merge 10 commits into
cms-sw:CMSSW_17_0_Xfrom
Electricks94:SoASerialisation_170X

Electricks94 commented Jun 25, 2026

Uh oh!

cmsbuild commented Jun 25, 2026 •

edited

Loading

Uh oh!

cmsbuild commented Jun 25, 2026

Uh oh!

cmsbuild commented Jun 25, 2026 •

edited

Loading

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

cmsbuild commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Electricks94 commented Jun 25, 2026

Uh oh!

cmsbuild commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 25, 2026

Uh oh!

cmsbuild commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

fwyzard commented Jun 25, 2026

Uh oh!

cmsbuild commented Jun 25, 2026

Failed RelVals-AMD_MI300X

Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Max Memory Comparisons exceeding threshold NVIDIA_T4

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cmsbuild commented Jun 25, 2026 •

edited

Loading

cmsbuild commented Jun 25, 2026 •

edited

Loading