Processing large (too large for memory) xarray datasets, and writing to netcdf

The problem is that there are other variables you are trying to write that don’t have time as a dimension: the coordinate variables x and y. You don’t want to re-write these every time you write a new timestep. Xarray is telling you how to solve it! Call

dd = dd.drop(['x', 'y'])

before calling dd.to_zarr().

Another thing to note: make sure ds_template uses dask-backed arrays for all of the data variables (with your desired chunk structure). Otherwise you will end up writing a lot of unnecessary data.

1 Like