Extremly slow write to S3 bucket with xarray.Dataset.to_zarr

[rabernat]

Thanks for sharing Masha! Writing a dataset of this size should take seconds, not hours.

@martindurant - do you have any idea what might be happening here? We have had similar reports of very slow writes with xarray → zarr → s3fs → s3, e.g.

Any tips on avoiding high AWS S3 cost when storing Zarr's with lots of objects

Hello, We are generating Zarr stores and push final data products to AWS S3 bucket. We were hit by unexpected high cost (~3K to store 40 data products) from AWS S3 to store our test data sets. When using (250, 10, 10) chunking for (date, x, y) dimensions, we ended up with average ~3-4M objects per single Zarr store, thus the high S3 cost. Our normal way of operation is to generate the data set locally and to push it to the S3 bucket. When new data that contributes to the data set becomes avai…

@mlukis has provided a nice minimal reproducible example we could use for debugging this.