Switch to writing files in CDF5 (NETCDF3_64BIT_DATA) format #333
Merged
Conversation
61ac0cf to
8873379
Compare
This is E3SM's preferred fromat for large files. Since E3SM can't support `NETCDF4` format, it is the only option that we can be sure will work for both small and large meshes, and saves us the hassle of having different fromats for different mesh sizes. Good performance does require first writing a `NETCDF4` file from `xarray` and then converting to `NETCDF3_64BIT_DATA`. This is handled by a new version of `mpas_tools`.
This ensures that they have the desired format.
These are causing warnings during local builds of the docs
Collaborator
Author
TestingI ran the |
Collaborator
Author
|
I'm going to merged this. If it turns out to be a problem, we can move back to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is E3SM's preferred fromat for large files. Since E3SM can't support
NETCDF4format, it is the only option that wecan be sure will work for both small and large meshes, and saves us the hassle of having different fromats for different mesh sizes.
Good performance does require first writing a
NETCDF4file fromxarrayand then converting toNETCDF3_64BIT_DATA. This is handled by a new version ofmpas_tools, see MPAS-Dev/MPAS-Tools#633.Copilot summary
This pull request introduces changes to improve NetCDF file handling and streamline the codebase in
polaris. Key updates include modifying the default NetCDF format, refactoring imports for clarity, and adding functionality to set default I/O types for streams. Below is a breakdown of the most important changes:NetCDF File Handling Improvements:
NETCDF3_64BIT_DATAinpolaris/default.cfgto align with improved compatibility and performance. The output engine was also switched fromscipytonetcdf4.polaris/model_step.pyto set the default I/O type for streams when the format isNETCDF3_64BIT_DATA.set_default_io_typefunction inpolaris/streams.pyto set theio_typeattribute for streams and immutable streams, except for immutable streams named 'mesh'.Codebase Refactoring:
polaris/mesh/spherical.py, replacingxarraywith an aliasxrand addingwrite_netcdffor improved readability and functionality.xarray.DataArrayandxarray.open_datasetto use the aliasxrfor consistency inpolaris/mesh/spherical.py. [1] [2]runmethod inpolaris/mesh/spherical.pyto rewrite temporary mesh files in the desired NetCDF format usingwrite_netcdf.Checklist
api.md) has any new or modified class, method and/or functions listedTestingcomment in the PR documents testing used to verify the changes