Question on DASK efficiency

Hey @oceanusofi! I came across this thread and just wanted to share that I have also had a lot of trouble with that package. I thought it would speed up my processing because I could use dask, but for reasons I don’t understand I’ve found it to be considerably slower than the standard package for calculating marine heatwaves (which xmhw is built off of).

I find the standard package to be a bit more awkward to use (it’s numpy based, not xarray based), but it runs much faster and you can still parallelize the code using dask arrays. Here is a snippet of doing that with some fake data:

# Create fake input data
lat_size, long_size = 100, 100
data = da.random.random_integers(0, 30, size=(1_000, long_size, lat_size), chunks=(-1, 10, 10))  # size = (time, longitude, latitude)
time = np.arange(730_000, 731_000)  # time in ordinal days

# define a wrapper for the climatology
def func1d_climatology(arr, time):
   _, point_clim = mhw.detect(time, arr)
   # return climatology
   return point_clim['seas']

# define a wrapper for the threshold
def func1d_threshold(arr, time):
   _, point_clim = mhw.detect(time, arr)
   # return threshold
   return point_clim['thresh']

# output arrays
full_climatology = da.zeros_like(data)
full_threshold = da.zeros_like(data)

climatology = da.apply_along_axis(func1d_climatology, 0, data, time=time, dtype=data.dtype, shape=(1000,))
threshold = da.apply_along_axis(func1d_threshold, 0, data, time=time, dtype=data.dtype, shape=(1000,))
climatology = climatology.compute()
threshold = threshold.compute()

That code runs the function twice, once for each of the outputs that I want (the climatology and the threshold values). I am almost positive that dask has a way where you don’t have to do that, but I haven’t figured it out.

I hope this is helpful!

1 Like

Thanks for sharing @rwegener2!

Has someone opened an issue on xmhw to report that the package is not working as advertised?

When she was working in our group, @hillaryscannell also developed a package for MHW tracking: