The likely problem here is that the Dask graph for that array is absolutely huge, which bogs everything down.
Instead, use this pattern:
Just use one single Dask chunk for the Xarray dataset
Specify chunk size via encoding
Here’s an example of that from our Serverless Data Cube Demo :
def create_dataset_schema(self, storage) -> None:
storage.initialize()
big_ds = (
xr_zeros(self.geobox, chunks=-1, dtype="uint16")
.expand_dims(
{
"band": self.bands,
"time": self.time_data,
}
)
.transpose(..., "band")
).to_dataset(name=self.varname)
big_ds.attrs["title"] = "Sentinel 2 Data Cube"
lon_encoding = optimize_coord_encoding(big_ds.longitude.values, self.dx)
lat_encoding = optimize_coord_encoding(big_ds.latitude.values, -self.dx)
encoding = {
"longitude": {"chunks": big_ds.longitude.shape, **lon_encoding},
"latitude": {"chunks": big_ds.latitude.shape, **lat_encoding},
show original
An even better solution would be to stop overloading to_zarr and actually implement schema creation in Xarray. That has been discussed extensively here:
opened 08:25PM - 19 Oct 23 UTC
enhancement
topic-zarr
### Is your feature request related to a problem?
A leaf from https://github.co… m/pydata/xarray/issues/8245, which has a bullet:
> compute=False is arguably a less-than-obvious kwarg meaning "write metadata". Maybe this should be a method, maybe it's a candidate for renaming? Or maybe make_template can be an abstraction over it
I've also noticed that for large arrays, running `compute=False` can take several minutes, despite the indexes being very small. I think this is because it's building a dask task graph — which is then discarded, since the array is written from different machines with the `region` pattern.
### Describe the solution you'd like
Would introducing a `metadata_only` parameter to `to_zarr` help here:
- Better name
- No dask graph
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
4 Likes