Developing online CMIP6 ENSO tool: issues with unresponsive cluster and looking for potential collaborators

Hi all,

I am a grad student working on tropical climate dynamics. I was excited to learn about Pangeo as it seems like an incredible way to do some more in depth analysis of CMIP6 data without having to download large 4d ocean variables on slow bandwidth.

So I went ahead to develop some code, which essentially extracts ocean velocities and temperature (along with wind stress and heat fluxes) in a tropical Pacific box with the aim of assessing feedbacks associated with ENSO across a selection of CMIP6 models.

While the code runs fine for 3d fields (like wind stress), it almost always crashes for 4d ocean variables. I get things like:

CancelledError: (‘truediv-5a4b2c564a815459f94e3086e8d792ac’, 112)

and

distributed.client - ERROR - Failed to reconnect to scheduler after 50.00 seconds, closing client
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>
asyncio.exceptions.CancelledError

It happens seemingly regardless of how big my cluster is, I’ve tried changing chunk sizes and still my experience is that anytime the cluster gets in contact with a 4d variable, it almost always becomes unresponsive.

So my questions are:
Is this a problem with the dask and chunks that are set up in a wrong way that I can fix with more knowledge? OR

Is my aim of carrying out (heavy) calculations with 4d ocean variables even well suited for pangeo cloud?

I also have a proposition: If anyone is interested in developing an online, straightforward tool for assessing ENSO dynamics in CMIP6, and how they change with global warming, and have experience dask/Pangeo/etc., I would really love to collaborate. I think my idea is awesome and could be very useful for the climate science community, but I’ve also realized that I may need some help turning it into a reality. At least using Pangeo, because right now I feel a little bit like giving up on it.

Thanks for reading my post!
Regards
Ulla

1 Like

Thanks for sharing this exciting idea!

What you want to do sounds reasonable and possible with Pangeo cloud. But it’s true that working with 4D CMIP6 data can get complicated and computationally expensive.

Your computation is probably crashing because the workers are using too much memory. There are two basic solutions:

  • just configure your cluster with more memory
  • tweak your code to be more memory efficient.

For the latter, there are lots of options, but it’s hard to make a specific suggestion without more detail. Could you share your code?

@jbusecke is our resident expert on CMIP6 cloud processing. I’m sure he would have some good suggestions.

Hi,

Thank you very much for your helpful reply. I’ve made a repository on Github that I’ve shared with you and @jbusecke. Let me know if it works. I’d be more than happy to receive tips for how to make the code more memory efficient. And to get your input on whether what I’m trying to do could work well on Pangeo.

Ulla

Dear Ulla,

Sorry for the late reply. I took a (very brief) look at your code and have a few comments.

  1. I think the main bottleneck will be “To get ready to perform the analysis, I like to regrid the ocean data, to make it easier to work with. Note, this section may have to be modified depending on which model you use.” Depending on how you set this up xesmf can consume a lot of memory. Is there a way you can break down the different steps and try to compute intermediate results to see “where” the calculation breaks?

  2. Shameless self promotion alert!!! I am actively developing a package to make working with cmip6 data easier (and my primary use case currently are 4D ocean variables). There are some examples in there that could be helpful. I am also actively working on some higher level tools that will simplify your ‘bookkeeping’ in the first few cells.

  3. Could you write out the full calculation needed for the ENSO stability index? I am not very familiar with the topic, and would like to learn more.

Overall the idea seems really cool. Let me know if the above helps you to narrow down the problem. Happy to help further if needed.

2 Likes

Hi jbusecke

  1. So you suspect it is the regridding that is swallowing memory? Interesting. I will follow your approach to find out where it breaks down.

  2. Wow, awesome, I will definitely check it out. Not self-promotion when it’s highly relevant :wink:

  3. Yes, I will upload the remainder of the code to github. Actually, the end goal is to have everything online, but because the cluster kept crashing, I decided to take some things offline for now.

Will be in touch shortly!

1 Like