I’ve been experiencing curious behaviour when using xarray and dask-jobqueue to parallelise the computation of a monthly climatology from zarr-format data stored on an ultra-fast parallel file storage system (https://www.beegfs.io/content/).
The memory load on each worker increases very rapidly throughout the operation, to values substantially larger than what I believe any one worker should have at any one time based on the chunking. This does not occur (or at least occurs to a much lesser degree) when the data is on a different file system.
It’s almost as though the workers are unable to reduce and process data at a sufficient rate to keep up with reading input. This probably makes no sense.
Given that the issue seems to be related to the hardware I’m using, it’s difficult for me to provide a reproducible example. So, I’m seeking input regarding how to debug and understand this issue. My apologies in advance if this is not the right forum for this type of question.