Extremely slow rechunking of Zarr store with xarray

[rabernat]

Thanks for the question. We’ll do our best to help. I’d like to first offer some feedback on how to make your post more easily “answerable.”

1. Include code / data examples that will allow us to reproduce your problem.
2. Your post contains several related but distinct questions. You might get a better response by just asking one question at a time.

Let me try to enumerate the distinct questions

@mliukis

Python’s rechunker package did a much better job by rechunking 5.4Gb Zarr in a “reasonable” 1.5 hours. To our surprise, access time is actually ~2x longer than for the Zarr store chunked at x and y of 10 (6s vs 3.4s). Is there an explanation for it?

2x longer than what? How are you measuring “access time”? I don’t understand what you are asking here., so I’m not going to try to answer this part unless you can provide some clarification.

@mliukis

I also tried to provide chunks values as encoding parameters at the time of Zarr creation with xarray.to_zarr(), but that didn’t seem to work as read back in Zarr store seems to be using chunk sizes as determined by the Zarr and not the ones I provided.

This sounds like it could be a bug with Xarray. Xarray should respect the chunks attribute the variable encoding when writing to Zarr. To check this, I wrote a short test:

import xarray as xr
import numpy as np
import zarr

ds = xr.DataArray(
    np.arange(100),
    name='foo',
    dims='x'
).to_dataset()
ds['foo'].encoding = {'chunks': 10}
ds.to_zarr('test.zarr', mode='w')

zgroup = zarr.open('test.zarr')
assert zgroup['foo'].chunks == (10,)

This verifies that Xarray is using the chunks specified in the encoding when writing the Zarr array. Without seeing your code, it’s hard to know what is going wrong in your case.

@mliukis

Any suggestions or advice on how to handle chunking for the Zarr store which is ever “growing” in one of the dimensions is much appreciated!

It sounds like you want to create a Zarr array that is contiguous in time but chunked in the other dimenions (x,y ); and then you want to append to your array in time. This will never work well in Zarr. The reason is fundamental: whenever you make a write that touches a Zarr chunk, the entire chunk has to be rewritten. Zarr does not support “partial chunk writes.” (Note that other storage formats like TileDB might work better here.) So every time you append to your array that is chunked this way, you basically have to rewrite the entire file. If your ingestion process requires you to append to a Zarr array along the time dimension, you are definitely better off chunking the array in time from the beginning.

However, it also sounds like you want to do timeseries analysis at each point. This requires an access pattern that is orthogonal to your write pattern. This was the same issue discussed in this epic thread.

Best practices to go from 1000s of netcdf files to analyses on a HPC cluster?

What? 8759 netcdf files totalling 17TB of HYCOM ocean model (u,v) velocity data at two depth levels (and bottom velocity), at hourly time steps. So the total data arrays of interest are [9000 (X) by 7055 (Y) by 8759 (time) by 2 (Depth)] for both u and v; Using nco tools, I can reduce these files as an example to 365 netcdf files, totaling 4.3TB, one file per day (24 hourly steps), for one depth level only, so data arrays are 9000 (X) by 7055 (Y) by 8759 (time) for the u,v components. How? Us…

The best solution we can offer is the rechunker package, which you are already using.

@mliukis

Python’s rechunker package did a much better job by rechunking 5.4Gb Zarr in a “reasonable” 1.5 hours

This should definitely not take so long. On a fast machine, you should be able to rechunk 5.4 GB of data in just a few seconds. If you provide more details of how you are invoking rechunker, maybe we can help you improve this.

@mliukis

We end up with a Zarr store where chunking is determined automatically by Zarr.

This is probably not a good idea. You should think hard about your access patterns and choose chunks that are optimized for your use case. There is no universal “good” chunking schemes. Everything depends on how you will access the data. Just remember the main rule: there are no partial reads / writes to Zarr chunks. If your operation touches a single item in a chunk, the whole chunk needs to be read / written.

As a final question, let me ask this: why Zarr? Did you try a more mature format like HDF5 before deciding that you needed Zarr? How did that go?