Hello all,
I am using the following function to load with dask a subset of a 4D matrix with original dimensions (time=365,depth=32,lat=3057,lon=4321) in ipython:
def load_octemp(yearsel,lon1,lon2,lat1,lat2,time,n):
year = yearsel
chunks = {'time_counter':1,'deptht':32,'y':3057,'x':4321}
dataset = xr.open_dataset('/home/folder/files/T_2016_2D.nc',chunks=chunks,engine='h5netcdf')
vars()['TEMP'+str(n)]= dataset.votemper.isel(time_counter=slice(0,365),deptht=slice(n,n+1),y=slice(lat1,lat2),x=slice(lon1,lon2))
vars()['temp'+str(n)] = vars()['TEMP'+str(n)].compute()
return vars()['temp'+str(n)]
Then, I would like to parallelize the loading of the subset of the file (final dimensions: 365 * 32 * 50 * 40) with the above function, across the depth dimension (That is, i would like to load simultaneously 32 3D maps of [time=365,lat=50,lon=40], each of them corresponding to 1 vertical level) using concurrent futures & the client.map() function like this:
from dask.distributed import Client, LocalCluster
cluster = LocalCluster()
client = Client(cluster)## Here cluster is the adress of the scheduler that I am connecting to.
client = Client(threads_per_worker=2, n_workers=16,processes=True)
Just for a test I tried to parallelize the loading of the 3D maps of only 2 vertical levels like this:
a=[ n for n in range(20,22)]
cores=os.cpu_count()
with concurrent.futures.ProcessPoolExecutor(cores) as ex:
foo = client.map(load_octemp,repeat(yearsel),repeat(lon1),repeat(lon2),repeat(lat1),repeat(lat2),repeat(time),a)
test= client.gather(foo)
The output variable ‘‘test’’ is a list of two 4D matrices of dimensions: [365,1,50,40]. Each matrix inside this list file, has its own metadata and values computed.
However, in order to continue with my calculations I would like to convert the test variable, from a list into a 4D xarray with dimensions:[365,2,50,40]. I tried to do it like this:
testf=xr.DataArray(test1)
The output testf is now an xarray DataArray. However, the metadata of the matrices that were inside the list are now completely lost.
Any ideas of how to retain the metadata of the matrices when converting the list into xarray?
I am assuming that the client.map() function outputs only list type of objects?Is it not possible to output xarray objects?
Sofi