# Very big memory load when using fast parallel file system

**URL:** <https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160>\
**Category:** HPC\
**Created:** [September 23, 2019, 3:34am UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160 "2019-09-23T03:34:07Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dougiesquire](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dougiesquire/32/1323_2.png) [@dougiesquire](https://discourse.pangeo.io/u/dougiesquire)\
**Post date:** [September 23, 2019, 3:34am UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/1 "2019-09-23T03:34:07Z")

</div>

I’ve been experiencing curious behaviour when using xarray and dask-jobqueue to parallelise the computation of a monthly climatology from zarr-format data stored on an ultra-fast parallel file storage system ([https://www.beegfs.io/content/](https://www.beegfs.io/content/)).

The memory load on each worker increases very rapidly throughout the operation, to values substantially larger than what I believe any one worker should have at any one time based on the chunking. This does not occur (or at least occurs to a much lesser degree) when the data is on a different file system.

It’s almost as though the workers are unable to reduce and process data at a sufficient rate to keep up with reading input. This probably makes no sense.

Given that the issue seems to be related to the hardware I’m using, it’s difficult for me to provide a reproducible example. So, I’m seeking input regarding how to debug and understand this issue. My apologies in advance if this is not the right forum for this type of question.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 23, 2019, 1:51pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/2 "2019-09-23T13:51:16Z")

</div>

Hi @dougiesquire - welcome to the forum and thanks for your message!

What you have encountered is a long-standing issue with dask. There is a github issue for it:

> <https://github.com/dask/distributed/issues/2602>

  
and a lot of active development happening in this area.

I strongly encourage you to engage on that github issue and describe your experience. Particularly useful would be to see the actual code you are using to trigger this memory overload. The dask gurus might have some useful workarounds.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 24, 2019, 7:21pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/3 "2019-09-24T19:21:32Z")

</div>

Hi @dougiesquire–just wanted to follow up. If you could weigh in on the linked github issue, it would probably add a lot to the discussion and motivate the dask developers to continue to focus on this issue. The more people they see chiming in, the higher priority it might receive.

Let me know if there is any way I can help you with this.

---

<div class="post-metadata">

**Author:** ![geynard](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/geynard/32/2924_2.png) [@geynard](https://discourse.pangeo.io/u/geynard)\
**Post date:** [September 24, 2019, 7:54pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/4 "2019-09-24T19:54:01Z")

</div>

Hey, just chiming here to say that the problem might not come from Dask. We’ve experienced a similar behaviour with Ifremer cluster:

> <https://github.com/dask/dask-jobqueue/issues/178>

This was due to a problem in their system configuration, not sure where though. This happened on Lustre file system.

Maybe you can test the snippet given in the issue, and tell us if you can reproduce your problem.

---

<div class="post-metadata">

**Author:** ![geynard](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/geynard/32/2924_2.png) [@geynard](https://discourse.pangeo.io/u/geynard)\
**Post date:** [September 24, 2019, 8:15pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/5 "2019-09-24T20:15:49Z")

</div>

> [@dougiesquire](#):
>
> It’s almost as though the workers are unable to reduce and process data at a sufficient rate to keep up with reading input. This probably makes no sense.

Forgot one thing: you may want to use the Dask Dashboard to see if your guess is correct. It would tell you a lot of information about what volume is being stored in memory on Dask side!

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dougiesquire/32/1323_2.png) [@dougiesquire](https://discourse.pangeo.io/u/dougiesquire)\
**Post date:** [September 25, 2019, 12:15am UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/6 "2019-09-25T00:15:44Z")

</div>

Thanks for the feedback, @rabernat and @geynard!

I’ve posted this issue on the recommended github thread.

> [@geynard](#):
>
> Forgot one thing: you may want to use the Dask Dashboard to see if your guess is correct. It would tell you a lot of information about what volume is being stored in memory on Dask side!

I should have been clearer in my original post - my references to “memory load” are as diagnosed by the Dask Dashboard. I’m pretty confident that my case is associated with Dask.

---

<div class="post-metadata">

**Author:** ![jbusecke](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jbusecke/32/118_2.png) [@jbusecke](https://discourse.pangeo.io/u/jbusecke)\
**Post date:** [September 30, 2019, 9:06pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/7 "2019-09-30T21:06:31Z")

</div>

I have recently faced the same problem. Thanks for the link @rabernat, ill post on the github issue.

---

<div class="post-metadata">

**Author:** ![pbranson](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/pbranson/32/156_2.png) [@pbranson](https://discourse.pangeo.io/u/pbranson)\
**Post date:** [October 8, 2019, 11:57am UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/8 "2019-10-08T11:57:52Z")

</div>

@dougiesquire I have had similar issues and the work around I have used for the CSIRO HPC is to allow spilling to disk as the nodes have local storage accessible at $TMPDIR - I believe this is better than using a location on the parallel file system due to the many small reads and writes that happen with caching. The workers will go orange, but your calculation may still complete without killing the workers.

Here is the jobqueue.yml that I use:

distributed:  
worker:  
memory:  
target: 0.6  
spill: 0.75  
pause: 0.80 # fraction at which we pause worker threads  
terminate: 0.95 # fraction at which we terminate the worker

jobqueue:  
slurm:  
cores: 4  
memory: 32GB  
processes: 2  
queue: h2  
walltime: 120  
local-directory: $TMPDIR

When working on Pawsey HPC, the nodes dont have local storage, but if you are happy to use containers with Shifter you can create a so called ‘per-node-cache’ that is overlaid into the container filesystem and reduces the metadata load on the parallel file system, example here:

> <https://github.com/pbranson/pangeo-hpc-shifter/blob/master/start_worker.sh>

  
increase the **size=4G** part to suit your needs, I have used up to 40G per worker without issue.

Hopefully some of the ideas that @TomAugspurger and @rabernat have discussed in those github issues help in a future dask release.

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dougiesquire/32/1323_2.png) [@dougiesquire](https://discourse.pangeo.io/u/dougiesquire)\
**Post date:** [October 11, 2019, 12:42am UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/9 "2019-10-11T00:42:05Z")

</div>

Thanks @pbranson! I’m hopeful of potential solutions implemented within dask, In the meantime, spilling to disk at least means the jobs will finish and these are great tips for Australian HPC systems.

---

<div class="post-metadata">

**Author:** ![Thomas\_Moore](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/thomas_moore/32/2035_2.png) [@Thomas\_Moore](https://discourse.pangeo.io/u/Thomas_Moore)\
**Post date:** [November 13, 2019, 3:53am UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/10 "2019-11-13T03:53:22Z")

</div>

Not surprisingly, within a similar CSIRO Pangeo HPC environment including our new fast BeeGFS storage, I seem to be hitting similar issues. @pbranson my attempt to set the local-directory to $TMPDIR doesn’t appear to assist? This is anecdotal as I’ve not had time to do lots of comparisons. As someone not well versed in investigating **Dask** code under the hood I’m wondering what I could be logging or documenting that might help others explore a fix in **Dask**?

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [November 13, 2019, 12:49pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/11 "2019-11-13T12:49:14Z")

</div>

Would there be a way to get me access to one of these machines so I can take a look at things? Or is anyone able to do a screen share with me?

---

<div class="post-metadata">

**Author:** ![Thomas\_Moore](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/thomas_moore/32/2035_2.png) [@Thomas\_Moore](https://discourse.pangeo.io/u/Thomas_Moore)\
**Post date:** [November 13, 2019, 5:17pm UTC](https://discourse.pangeo.io/t/very-big-memory-load-when-using-fast-parallel-file-system/160/12 "2019-11-13T17:17:51Z")

</div>

Thanks for the reply @TomAugspurger - I’ll DM you.