# Map\_blocks and to\_zarr(region=)

**URL:** <https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019>\
**Category:** Data\
**Created:** [December 13, 2021, 5:00pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019 "2021-12-13T17:00:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jamesmcc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jamesmcc/32/1180_2.png) [@Jamesmcc](https://discourse.pangeo.io/u/Jamesmcc)\
**Post date:** [December 13, 2021, 5:00pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/1 "2021-12-13T17:00:23Z")

</div>

Season’s Greetings!

I’ve recently been processing some fairly large model output from sub-daily resolution to daily resolution. I have 3 different spatial resolution. With the highest-resolution data i really run into a processing bottleneck. This is the display of the zarr store containing two variables to reprocess to daily which cause the bottleneck

 ![Screen Shot 2021-12-13 at 8.49.41 AM](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/a/a10b7b866869fa64ffc8dd3083a8e719753920bd.png)

Main points:

- this data set is not something to hold in memory
- daily averages only change the temporal dimension (8:1) but not the spatial dimension (so this is going to be embarrassingly parallel in space)

To me this just shouts map\_blocks with to\_zarr(region) on each block/region!

I’ve followed 3 mains approaches to the problem, without any breakthroughs or major successes. There were several variations on these but here’s the summary of the 3 approaches: a) things I tried, b) what happened, and c) my a priori expectations when trying to process just a modest (~6000 x ~4000) spatial sub region of one the above variables (times below are for processing a single variable, using generally ~20 workers, though the number of workers above this amount didnt have a huge impact).

* * *

1  
I “rolled my own” naively without map\_blocks: ~40hours to compute. This may not be the worst thing, but the order of my computations was not really optimized when I reviewed these in the end. Map blocks helps lay out (and check) all the relevant input and output sizes, and it therefore seems to aid in ordering the computations correctly compared to going it alone. When looking at the dask workers in the dash board, the computations have lots of holes, like swiss cheese. This further suggests potential optimization.

2  
I put to\_zarr() inside the function passed to map\_blocks and returned a dummy value of 1 for each output chunk: this was super flaky and unreliable, sometimes it would work, but sometimes promised keys would go missing and there would be timeouts. A priori, this would be the best approach (and maybe i need to revisit it when my cluster is not having a bad day? I’m not sure it was or was not having a bad day). But it was basically random if my 9 output chunk test dataset would successfully complete. Any thoughts about what is going on here? I frankly dont remember much about the activity of the workers in time because it was too erratic.

3  
Loop over subsets/regions of the input data and call map\_blocks on these subsets, then write the returned data to\_zarr(region=): this completed in about ~36 hours. This is stable, but not much more efficient that #1. More details of the approach: outside the loop, identify a full-space slice of time to process on the input, and then do (without computing)

```auto
input           
.isel(time=time_in_slice)
.resample(time="1D").mean(dim='time')
.chunk(dict(
    time=time_out_chunk_size, x=space_chunk_size, y=space_chunk_size))
.to_zarr(compute=False, append_dim='time')

```

Then for each separate spatial chunk of this temporal subset/slice, do

```auto
map_blocks(
    avg_daily, 
    input.isel(time=time_in_slice, x=x_slice, y=y_slice)
).load().to_zarr(region)

```

The dask dashboard looks great when reading and computing daily averages and rechunking, very tight block across all workers. But between those dask operations and the to\_zarr(region) part (which is nicely parallel), there is a very large gap without any dask activity. Some notes: When doing this, i have to load in: xr.map\_blocks().load().to\_zarr() or the code fails. My additional observations about this are very anecdotal, but they suggest that the gap might be in arranging the output along the coordinates: the gap between processing the input in dask and writing to\_zarr slows down if I try to process more than a few output time chunks, ie if the subset I’m taking is long in time then I pay additional overhead. It also appears that if I add multiple dimensions to what I’m passing to map\_blocks(), the slow down is worse (that is when x\_slice and y\_slice are more than single chunks in those dimensions).

* * *

I’m happy to go any direction that makes sense with this and provide more details, I’m just not sure what the best direction is. Please let me know what you think and I’m happy to add more.

A few additional notes:  
I’ll note that I think the input is reasonably well chunked for the task at hand, but I could be wrong about that.

The exercise with to\_zarr(append\_dim, compute, region) in this thread

> [@Processing large (too large for memory) xarray datasets, and writing to netcdf](https://discourse.pangeo.io/t/processing-large-too-large-for-memory-xarray-datasets-and-writing-to-netcdf/1724/8):
>
> An alternative approach, if you know the total number of timesteps in advance, would be to initialize the whole Zarr store at the beginning with a lazy “template” dataset, i.e. ds\_template.to\_zarr(tempdir, compute=False) and then use the region argument to write, i.e. something like for j in range(ntimesteps): dd.to\_zarr(tempdir, region={'time': slice(j, j+1)} This might be faster because it won’t actually open the Zarr store with Xarray each time. That’s what I think is happening with a…

is extended somewhat. I’ve found that when doing to\_zarr(compute=False) on large output data sets is very slow and does not apparently scale well. It is much more efficient to slice the output in time and loop to\_zarr(compute=False, append\_dim=‘time’) to generate the full uncomputed zarr store. Then you can write to this by region later (and in parallel)

My final solution was to run #3 with another level of parallelization at the script level. I define “team\_members” 0-5 and pass a team member id to each script which identifies which temporal chunks to process for each member. This works nicely and I get the processing done in ~6hours. but that’s just brute force!

---

<div class="post-metadata">

**Author:** ![dcherian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dcherian/32/2235_2.png) [@dcherian](https://discourse.pangeo.io/u/dcherian)\
**Post date:** [December 14, 2021, 8:09pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/2 "2021-12-14T20:09:27Z")

</div>

Hi @Jamesmcc

I don’t know about the `to_zarr` bits but you should be able to avoid map\_blocks with  
[flox](https://flox.readthedocs.io) (please double check the output though)

```python
import flox.xarray

ds = xr.tutorial.open_dataset('air_temperature', chunks={"time": 24*3})
flox.xarray.resample_reduce(ds.resample(time="1D"), func="mean")

```

This will be equivalent to your map\_blocks thing but will avoid the per-chunk overhead that comes with map\_blocks (you have so many chunks!). I’m hoping the decrease in number of tasks is enough to make things work smoothly.

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dougiesquire/32/1323_2.png) [@dougiesquire](https://discourse.pangeo.io/u/dougiesquire)\
**Post date:** [December 15, 2021, 10:23pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/3 "2021-12-15T22:23:10Z")

</div>

> [@Jamesmcc](#):
>
> I’ve found that when doing to\_zarr(compute=False) on large output data sets is very slow and does not apparently scale well. It is much more efficient to slice the output in time and loop to\_zarr(compute=False, append\_dim=‘time’) to generate the full uncomputed zarr store.

My understanding is that `to_zarr(compute=False)` still computes the dask graph to write the zarr store, which can take a really long time. This is not actually necessary when all you’re wanting to do is initialize the empty zarr store. You could hack the code from `xarray.backends` to write the empty store without computing the dask graph. E.g. something like the following should write the empty store almost instantly even for very large datasets ( **note, I have not tested the following and have no idea how robust it is** ):

```python
def write_zarr_metadata(dataset, output_file):
    """ Write zarr metadata without computing the dask graph.
        Stolen and adapted from xarray.backends:
        https://github.com/pydata/xarray/blob/6a29380008dcd790f9adfbc290affcb767c913b2/xarray/backends/api.py#L1309
    """

    from xarray.backends.common import ArrayWriter, _normalize_path
    from xarray.backends.api import _validate_dataset_names, _validate_attrs, dump_to_store
    import xarray.backends as backends
    
    # expand str and Path arguments
    output_file = _normalize_path(output_file)
    
    mapper = output_file
    encoding = {}
    mode = "w-"

    _validate_dataset_names(dataset)
    _validate_attrs(dataset)
    
    zstore = backends.ZarrStore.open_group(
        store=mapper,
        mode=mode,
        consolidated=False,
        consolidate_on_close=True,
        stacklevel=4,
    )

    writer = ArrayWriter()
    dump_to_store(dataset, zstore, writer, encoding=encoding)

```

---

<div class="post-metadata">

**Author:** ![Jamesmcc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jamesmcc/32/1180_2.png) [@Jamesmcc](https://discourse.pangeo.io/u/Jamesmcc)\
**Post date:** [December 17, 2021, 5:03pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/4 "2021-12-17T17:03:03Z")

</div>

@dougiesquire Thanks for this suggestion. I had guessed that the wait was for the graph and it also seems that the time to calculate the graph grows faster than the size of the data. Honestly, this piece of the problem is easy, though it creates unnecessary code. I guess it’s a question if this should be a change or and additional option to write\_zarr? I might be tempted to try your proposed solution next time I’m dealing with this and see if that has a path back to xarray.  
Cheers, James

---

<div class="post-metadata">

**Author:** ![Jamesmcc](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jamesmcc/32/1180_2.png) [@Jamesmcc](https://discourse.pangeo.io/u/Jamesmcc)\
**Post date:** [December 17, 2021, 5:03pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/5 "2021-12-17T17:03:24Z")

</div>

Thanks @dcherian! I’ll let you know when I get around to trying this.

---

<div class="post-metadata">

**Author:** ![dougiesquire](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dougiesquire/32/1323_2.png) [@dougiesquire](https://discourse.pangeo.io/u/dougiesquire)\
**Post date:** [December 17, 2021, 9:24pm UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/6 "2021-12-17T21:24:10Z")

</div>

FYI, I’ve opened an issue with xarray to see if there is interest in adding an option/method to initialise zarr metadata without computing the dask graph: [Initialise zarr metadata without computing dask graph · Issue #6084 · pydata/xarray · GitHub](https://github.com/pydata/xarray/issues/6084)

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [May 26, 2022, 7:26am UTC](https://discourse.pangeo.io/t/map-blocks-and-to-zarr-region/2019/7 "2022-05-26T07:26:03Z")

</div>

The above seemed to work if I took out stacklevel - an older version thing maybe in my particular environment.

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/b/be6ecbb6aca3ba3de603c05473e0ec141a14e3da.png)

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/1/13e15c2d50fea58a184b2eddc37cc85dc882b643.png)