Optimizing climatology calculation with Xarray and Dask

[fmaussion]

Thanks all for your suggestions!

Let’s start with the pure .mean() swap issue.

@kirill.kzb

Trying out Dask configuration with 1 thread per worker would be helpful.

Thanks! I tried it as well, no big change as far as I can see. Rendered notebook (including rich dask output) is here

This results in a swap overfill. You can see a screen recording here

@kirill.kzb

without knowing data format it’s hard to say if chunking is appropriate for a given data source, lattitude=50 is kinda narrow, unless your data is stored in tall narrow chunks on disk that is.

How do I know how the data is stored? In the notebook I offer a link to download the data if anyone is interested.

@rabernat

Regarding the Dask swap memory issue, I feel like that really needs to be raised with Dask itself. Have you opened an issue somewhere?

I haven’t yet. Its a “minor” issue (it works, albeit with some unwanted swap use), and may be related to linux in particular?