Skip to content

ENH: 2D chunks for all#285

Draft
jklymak wants to merge 4 commits into
MITgcm:masterfrom
jklymak:enh-2D-for-all
Draft

ENH: 2D chunks for all#285
jklymak wants to merge 4 commits into
MITgcm:masterfrom
jklymak:enh-2D-for-all

Conversation

@jklymak

@jklymak jklymak commented Sep 27, 2021

Copy link
Copy Markdown
Contributor

This allows 2D chunking for all mds files, not just the LLC cases.

This is a total hack at this point/proof of concept. However, it works for me, and allows me to get 2-D chunks and parallel work on a machine that doesn't have enough memory for my model run (for some reason I can't discern - each variable is only 5 Gb).

  • needs docs
  • needs tests
  • probably needs to be properly interested with the other chunking options.
with xm.open_mdsdataset(data_dir, prefix=['means'], 
                                endian="<", geometry='cartesian',
                                chunks="2D") as ds:
   print(ds)

Here Nz=400:

...
Data variables:
    VVEL     (time, Z, YG, XC) float64 dask.array<chunksize=(1, 1, 1216, 1440), meta=np.ndarray>
    UVEL     (time, Z, YC, XG) float64 dask.array<chunksize=(1, 1, 1216, 1440), meta=np.ndarray>
    THETA    (time, Z, YC, XC) float64 dask.array<chunksize=(1, 1, 1216, 1440), meta=np.ndarray>
...

@mjlosch

mjlosch commented Apr 16, 2026

Copy link
Copy Markdown
Member

@jklymak I realise this is old, but would it be possible to contribute a short test for this new functionality?

@mjlosch

mjlosch commented Apr 16, 2026

Copy link
Copy Markdown
Member

@IvanaEscobar The tests all passed for python 3.11 to 3.13 on my local machine. Not sure what happened here, looks like it has to do with the CI itself.

@jklymak

jklymak commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

@mjlosch possibly this summer, but not very soon! I'll move to draft, or anyone is more than welcome to take this over.

@jklymak jklymak marked this pull request as draft April 16, 2026 23:32
@IvanaEscobar

Copy link
Copy Markdown
Collaborator

@mjlosch the error is addressed in PR #358 . The CI needed updated gfortran compilers in each environment tested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants