# Xarray operations (e.g., preprocess) running locally (post open\_mfdataset) instead of on Dask distributed cluster

**URL:** <https://discourse.pangeo.io/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/4637>\
**Category:** Uncategorized\
**Tags:** zarr\
**Created:** [October 29, 2024, 7:16am UTC](https://discourse.pangeo.io/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/4637 "2024-10-29T07:16:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lbesnard](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/lbesnard/32/2549_2.png) [@lbesnard](https://discourse.pangeo.io/u/lbesnard)\
**Post date:** [October 29, 2024, 7:16am UTC](https://discourse.pangeo.io/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/4637/1 "2024-10-29T07:16:27Z")

</div>

For reference, I’ve also asked the same question at

> **[remote cluster with dask distributed uses the deployment machine's memory and...](https://github.com/dask/distributed/discussions/8913)**
>
> I have a python script, running from my personal laptop (Machine A). This script starts a Coiled cluster with dask distributed. Then NetCDFs are open together with xr.open\_mfdataset. Things go a bi...

and in the coiled slack page. I will update this discussion in case I have a positive answer from those places.

I’m using dask.distributed and a coiled cluster to open with xarray multiple NetCDF files. Nothing fancy here.

But there is also a preprocess\_xarray function running during the opening of files. This preprocess\_xarray function is completely “empty” in my 1st showcase example. However, all the NetCDF files are being downloaded to the machine running the cluster (I’m monitoring my network usage). If this function had stuff in it, It would use my local memory …

I’ve tried many things, adding a @delayed decorator to force this function to run on the cluster… and so on with no success.

I have a reproducible example here [code to showcase issues with dask distributed running preprocess on local machine instead of cluster · GitHub](https://gist.github.com/lbesnard/97bdf0b4af9fa340e8ef47aa20b3cc93) . this is a very simplified version of a bigger script available at [GitHub - aodn/aodn\_cloud\_optimised: Cloud optimised data formats](https://github.com/aodn/aodn_cloud_optimised)

Also the reason why the preprocess\_xarray is not part of the class in my example is because that function can’t be serialized and used with partial.

Here is another version of that gist, but slightly different. The preprocess function is creating a dummy variable, and all of the memory of my local machine is being used

> <https://gist.github.com/lbesnard/0732b7572983ea5ede5e8d7bcbe809fc>

Cheers,  
Loz

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [November 2, 2024, 3:31am UTC](https://discourse.pangeo.io/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/4637/2 "2024-11-02T03:31:55Z")

</div>

Answered there…looks like your client is not running anything?

---

<div class="post-metadata">

**Author:** ![lbesnard](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/lbesnard/32/2549_2.png) [@lbesnard](https://discourse.pangeo.io/u/lbesnard)\
**Post date:** [November 13, 2024, 3:13am UTC](https://discourse.pangeo.io/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/4637/3 "2024-11-13T03:13:01Z")

</div>

for reference, answered here [Xarray operations (e.g., preprocess) running locally (post open\_mfdataset) instead of on Dask distributed cluster · dask/distributed · Discussion #8913 · GitHub](https://github.com/dask/distributed/discussions/8913#discussioncomment-11127547)

cause is a s3fs bug