# How to efficiently overwrite existing zarr archive with reordered time axis? (Updated Question)

**URL:** <https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714>\
**Category:** Data\
**Created:** [September 7, 2022, 10:08am UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714 "2022-09-07T10:08:59Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [September 7, 2022, 10:08am UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/1 "2022-09-07T10:08:59Z")

</div>

I am having the following problem:

I am using `xarray.open_mfdataset(['archive1', 'archive2'], engine='zarr')` to open multiple zarr archives at once. During creation of the archive2, data was append to the archive in a non chronological way. This means my time axis is neither monotonically increasing nor decreasing. This yields to an error when using `open_mfdataset`.

Actually it was possible to fix this for small to medium size archives by opening the archive with xarray.open\_dataset, apply sort() and store the archive as a copy. Afterwards I removed the old one and renamed the new fixed version.

But now I am having this issue with a huge zarr archive and the machine runs out of memory (and it has 128 GB RAM ) .  
So is there another way I can re-order a time axis of an zarr archive?

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [September 7, 2022, 3:39pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/2 "2022-09-07T15:39:12Z")

</div>

Can you please clarify: are the whole chunks out of time order, or are individual coordinates within each chunk also out of order?

---

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [September 7, 2022, 3:59pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/3 "2022-09-07T15:59:53Z")

</div>

The dataset is chunked along the time axis. So e.g. we have dimension (96, 1000, 1000), and chunk\_size(1, 100, 100) for 96 timesteps.  
Does this help?

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [September 7, 2022, 4:12pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/4 "2022-09-07T16:12:38Z")

</div>

If you have exactly one timestamp per chunk, then kerchunk can process this for you without any need for sorting or otherwise rewriting the data.

[https://fsspec.github.io/kerchunk/](https://fsspec.github.io/kerchunk/)

---

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [September 7, 2022, 6:50pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/5 "2022-09-07T18:50:22Z")

</div>

Thanks for this hint, I will check it out and give feedback to you asap.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 7, 2022, 8:20pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/6 "2022-09-07T20:20:53Z")

</div>

I like Martin’s suggestion. Kerchunk allows you to create a translation layer between the chunks seen by the user and the chunks on disk.

You might also be able to Xarray’s lazy indexing capabilities to accomplish the same thing. It would work something like this (untested pseudocode)

```python
ds = xr.open_dataset(zarr_store, engine="zarr") # by default no dask
correct_time_order = ... # figure this out somehow
ds_sorted = ds.isel(time=correct_time_order)
ds_sorted_chunked = ds_sorted.chunk({'time': desired_time_chunks})

```

I’m not 100% sure that zarr indexing in xarray supports this, but it’s worth a try.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 7, 2022, 10:18pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/7 "2022-09-07T22:18:39Z")

</div>

I just confirmed that the idea above works. Here is some code to reproduce the solution I proposed.

```python
import xarray as xr
import numpy as np

ds = xr.tutorial.open_dataset("air_temperature")
ds.chunk({"time": 1}).to_zarr("air_temperature.zarr", mode="w")
ds2 = xr.open_dataset("air_temperature.zarr", engine="zarr")
assert not ds2.air.variable._in_memory # data have not been loaded yet
time_order = np.arange(ds2.dims['time'])
np.random.shuffle(time_order)
ds_sorted = ds2.isel(time=time_order)
assert not ds_sorted.air.variable._in_memory # data still have not been loaded

# this triggers evaluation
xr.testing.assert_equal(
    ds.air.isel(time=time_order[0]),
    ds_sorted.air.isel(time=0)
)

```

---

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [September 8, 2022, 5:37am UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/8 "2022-09-08T05:37:27Z")

</div>

In general, using `ds.sortby('time')` is working as well. This is my actual solution. But I was not able to write this dataset back to the disk or overwrite the existing one.

Using isel in the correct order is a really nice alternative but it would mean that I will not be able to use `open_mfdataset()` or am I wrong?

Actually I recognize that my question should be adapted to:

**How to efficiently overwrite existing zarr archive with reordered time axis?**

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 8, 2022, 12:45pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/9 "2022-09-08T12:45:14Z")

</div>

I would probably avoid trying to overwrite the dataset in place. Just write a new one, delete the old one, and rename it. Does that work?

Can you call `ds.to_zarr` on your reordered dataset?

---

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [September 9, 2022, 6:40am UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/10 "2022-09-09T06:40:23Z")

</div>

Thats the way I do , but I am getting out of memory during this process because my dataset is very large.

It is working for smaller datasets but not for large ones.

I thought it would be possible to adapt the mapping in the json files which should be really fast and simple.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 9, 2022, 11:40am UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/11 "2022-09-09T11:40:31Z")

</div>

> [@meteoDaniel](#):
>
> Thats the way I do , but I am getting out of memory during this process because my dataset is very large.

I’m just confused why that is happening. Are you sure you are following this sequence precisely:

1. Open the original dataset _without any chunks_, i.e. `chunks=None`
2. Do the reordering
3. _Then_ call `.chunk` with the desired time chunks
4. Finally call `.to_zarr`

The kerchunk solution is a fine one, but at the same time it should be possible to make the xarray-based one work. If not, there is a bug somewhere that needs to be tracked.

---

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [January 19, 2023, 7:51pm UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/12 "2023-01-19T19:51:06Z")

</div>

Dear @rabernat , today I had the pleasure to use rechunker the first time, and it works really well at the moment.

Do you think it is possible to expand the functionality of rechunker to reorder e.g. as time axis ?

This would solve the issue I mentioned above.

Best regards  
Daniel

---

<div class="post-metadata">

**Author:** ![Jiaming\_Tan](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jiaming_tan/32/2946_2.png) [@Jiaming\_Tan](https://discourse.pangeo.io/u/Jiaming_Tan)\
**Post date:** [September 19, 2024, 9:02am UTC](https://discourse.pangeo.io/t/how-to-efficiently-overwrite-existing-zarr-archive-with-reordered-time-axis-updated-question/2714/13 "2024-09-19T09:02:41Z")

</div>

I have a question, I have a combined zarr named bigzarrno5.zarr，it has dimensions valid\_time, longitude, latitude, pressure\_level，and it contain day 1,2,3,4,6 and the valid\_time format is datatime64 like ‘2020-11-01T00:00:00.000000000’,each day have 24 hours.Now I have a day5.zarr，it has the same dimensions with bigzarrno5.zarr except valid time becase its time is from 2020-11-05T00:00:00.000000000 to 2020-11-05T23:00:00.000000000, i want to insert the day4.zarr to the bigzarrno5.zarr without rewrite the whole zarr, because i think the zarr file store data in chunk, i just need to store the day4.zarr’s chunk in a proper location and update the metadata, that’s means I didn’t need to change the original chunk.  
Thank you so much