Best way of copying and rechunking a remote zarr store

Uncategorized
2

Hi Guido,

The init step doesn’t have to be expensive — make the template lazy and single-chunked, and set the target chunking through encoding rather than through the template’s own chunks:

import dask.array as da
import numpy as np
import xarray as xr

# one lazy chunk per variable: nothing is allocated, and the graph stays
# at one task per variable no matter how big the dataset is
data_vars = {
    name: xr.DataArray(
        da.full(shape, np.float32("nan"), chunks=-1, dtype="float32"),
        dims=dims,
        attrs=src[name].attrs,
    )
    for name, ... in ...
}
ds_empty = xr.Dataset(data_vars, coords=coords, attrs=attrs)

# the on-disk chunking is declared here, not by the template's dask chunks
# also set compression here
encoding = {v: {"chunks": target_chunks} for v in ds_empty.data_vars}

ds_empty.to_zarr(TARGET, compute=False, mode="w", encoding=encoding)

Keeps the deferred write graph tiny. If the template is numpy-backed, or chunked at the target chunk size, you either allocate the whole array or build one task per output chunk — which is where the memory blowup at init comes from. Good older thread on exactly this:

Decoupling template chunks from store chunks via encoding means you can change the target layout without touching how the template is built.

Side questions:

  • why using zarr_format=2?
  • have you considered icechunk-era5 which comes with dual chunking to avoid having to download and rechunk yourself?