# Understanding optimal zarr chunking scheme for a climatology

**URL:** <https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335>\
**Category:** Science\
**Created:** [March 22, 2022, 2:53pm UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335 "2022-03-22T14:53:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rwegener2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rwegener2/32/887_2.png) [@rwegener2](https://discourse.pangeo.io/u/rwegener2)\
**Post date:** [March 22, 2022, 2:53pm UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/1 "2022-03-22T14:53:48Z")

</div>

Hi Pangeo people 👋

I am fairly new to working with zarr and I am trying to understand how to pick an optimal chunking scheme for my use case of calculating a climatology.

I have been watching [this fantastic talk](https://youtu.be/qo0qqE7WGLQ?t=235) by Deepak Cherian from the Pangeo Showcase and I understand that the recommendation is to create chunks which are large in space and small in time (ex. 1 chunk = 180 latitude, 360 longitude, 1 time).

To me, it seems like the opposite shape (ex. 1 chunk = 1 latitude, 1 longitude, all of time) would be faster because for a single latitude/longitude location you can calculate the climatology for every day of the year by accessing just a single chunk. I think Deepak’s talk explains this by saying that anything where you access multiple values per chunk (multiple groups per block, in the talk’s phrasing) is slow, but I am having a hard time understanding why that is.

Is anyone willing to 1) confirm that I am understanding how to chunk for a climatology correctly or 2) point out where I am misunderstanding optimal chunk shape? Thanks so much in advance!

* * *

_Examples, if helpful, in the form of two cloud SST datasets:_

**OISST** - large in space, small in time. What I understand to be the recommended chunking scheme for a climatology  
`https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa_oisst/v2.1-avhrr.zarr`

 ![OISST](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/e/ee244accf973d399c05e9a1f7da9102bc03abf73.png)

**MUR** - small in space, large in time. What I would have thought was the best chunking scheme for a climatology  
`s3://mur-sst/zarr`

 ![MUR](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/3/350c813b4dad86b446c56cb6e0d9d30d53e589d4.png)

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [March 22, 2022, 3:17pm UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/2 "2022-03-22T15:17:11Z")

</div>

> [@rwegener2](#):
>
> To me, it seems like the opposite shape (ex. 1 chunk = 1 latitude, 1 longitude, all of time) would be faster because for a single latitude/longitude location you can calculate the climatology for every day of the year by accessing just a single chunk.

Rachel, I agree with this intuition. However, the catch is that the standard way of coding the climatology operation in xarray, i.e.

```python
ds.groupby('time.dayofyear').mean(dim='time')

```

may not lead to the optimal dask graph. You can verify this yourself by looking at the chunk structure and number of tasks.

An alternative is to use `apply_ufunc` or `map_blocks`. I explored this approach in the following notebook: [Xarray Anomaly Calculations · GitHub](https://gist.github.com/rabernat/30e7b747f0e3583b5b776e4093266114)

Curious what @dcherian has to say.

---

<div class="post-metadata">

**Author:** ![dcherian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dcherian/32/2235_2.png) [@dcherian](https://discourse.pangeo.io/u/dcherian)\
**Post date:** [March 23, 2022, 11:49am UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/3 "2022-03-23T11:49:39Z")

</div>

The point of [flox](https://flox.readthedocs.io/) is to use strategies that work for the chunking that is present. But you are correct in general, it’s better to have bigger chunk sizes in time.

For MUR-type data ( a single chunk in time), use [`method="blockwise"`](https://flox.readthedocs.io/en/latest/implementation.html) with [flox.xarray.xarray\_reduce](https://flox.readthedocs.io/en/latest/generated/flox.xarray.xarray_reduce.html). It is identical to ryan’s map\_blocks suggestion.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [March 23, 2022, 1:40pm UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/4 "2022-03-23T13:40:16Z")

</div>

Thanks for the reply Deepak!

Rachel, it would be really interesting to see some benchmarks of the different approaches in terms how many dask tasks they produce and how long they take to run on a medium-sized dataset (not MUR!).

---

<div class="post-metadata">

**Author:** ![dcherian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dcherian/32/2235_2.png) [@dcherian](https://discourse.pangeo.io/u/dcherian)\
**Post date:** [March 23, 2022, 3:16pm UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/5 "2022-03-23T15:16:15Z")

</div>

I agree. I’ve been putting it off for a few months now 😉

I did type up some quick notes on trying to do the daily climatology for the OISST dataset: [Strategies for climatology calculations - flox](https://flox.readthedocs.io/en/latest/user-stories/climatology.html)

I’m thinking we can build up a collection of such notes with some crowdsourcing.

---

<div class="post-metadata">

**Author:** ![rwegener2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rwegener2/32/887_2.png) [@rwegener2](https://discourse.pangeo.io/u/rwegener2)\
**Post date:** [March 23, 2022, 9:14pm UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/6 "2022-03-23T21:14:36Z")

</div>

Wow, thanks so much to you both! @rabernat I went through your whole notebook this morning and I feel much more comfortable with `.map_blocks()` and @dcherian that new flox docs page looks super informative.

I’ll read over the new flox page next, then it sounds like some benchmarking will be my next step. One of my takeaways here is that instead of trying to logic through how each chunking/compute approach would work, I should instead look at the chunk size and number of tasks in the dask graph. Through your examples I’m starting to get comfortable doing that.

I’ll be sure to re-post my benchmarking in case it becomes helpful for a set of crowsourced notes. Thanks again!

---

<div class="post-metadata">

**Author:** ![josephyang](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/josephyang/32/541_2.png) [@josephyang](https://discourse.pangeo.io/u/josephyang)\
**Post date:** [April 4, 2022, 1:56am UTC](https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335/7 "2022-04-04T01:56:54Z")

</div>

FYI, this is how we ([https://oikolab.com](https://oikolab.com)) chunk data (large in time, small in space) to store ERA5 data. Most of our users are looking for multiple years of data for a specific location or small region (e.g. a state in US or India) so this allows us to only fetch a small data volume for each API request.

A drawback with this is that constantly appending new data or replacing some data becomes a hassle and if the ‘time’ chunk is too large, fetching even a single day worth of data for a location is not terribly efficient. We’ve iterated quite a bit to find the chunk size that worked reasonably for the average use case.