# Slow append to existing s3 zarr store using xarray

**URL:** <https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117>\
**Category:** Data\
**Tags:** zarr\
**Created:** [May 6, 2025, 12:54pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117 "2025-05-06T12:54:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![markmuetz](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/markmuetz/32/1805_2.png) [@markmuetz](https://discourse.pangeo.io/u/markmuetz)\
**Post date:** [May 6, 2025, 12:54pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/1 "2025-05-06T12:54:32Z")

</div>

I have a large dataset that has already been written, publicly available here: [https://hackathon-o.s3-ext.jc.rl.ac.uk/sim-data/dev/v5/glm.n2560\_RAL3p3/um.PT1H.hp\_z10.zarr/](https://hackathon-o.s3-ext.jc.rl.ac.uk/sim-data/dev/v5/glm.n2560_RAL3p3/um.PT1H.hp_z10.zarr/).

I would like to append two new fields to this with:

```python
ds_static = xr.Dataset()
ds_static['orog'] = hporog.copy()
ds_static['sftlf'] = hpland.copy()
ds_static.to_zarr(zarr_store, mode='a')

```

`zarr_store` is an s3-like datastore, and I’m writing by setting up an `s3fs.S3FileSystem` object - see here: [wcrp\_hackathon/scripts/process\_um\_data/um\_process\_tasks.py at 4838df8a93ba8dde17ecd6bacd8ef394bd7ddb50 · markmuetz/wcrp\_hackathon · GitHub](https://github.com/markmuetz/wcrp_hackathon/blob/4838df8a93ba8dde17ecd6bacd8ef394bd7ddb50/scripts/process_um_data/um_process_tasks.py#L510)

However, this is absurdly slow (30 min+), and times out in my testing before any writes happen. Writing to a new store takes ~30s. I suspect it has to scan the large dataset first, which taken a very long time. Any help speeding this up would be much appreciated.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [May 6, 2025, 5:40pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/2 "2025-05-06T17:40:13Z")

</div>

Try wrapping `zarr_store` in a [`LoggingStore`](https://zarr.readthedocs.io/en/stable/api/zarr/storage/index.html#zarr.storage.LoggingStore) to understand what sort of calls to the store are happening.

---

<div class="post-metadata">

**Author:** ![markmuetz](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/markmuetz/32/1805_2.png) [@markmuetz](https://discourse.pangeo.io/u/markmuetz)\
**Post date:** [May 6, 2025, 8:51pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/3 "2025-05-06T20:51:28Z")

</div>

Thanks @rabernat - I’ll give this a shot. Presumably I’m looking for lots of accesses of existing data…

---

<div class="post-metadata">

**Author:** ![josephyang](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/josephyang/32/541_2.png) [@josephyang](https://discourse.pangeo.io/u/josephyang)\
**Post date:** [May 9, 2025, 4:41pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/4 "2025-05-09T16:41:10Z")

</div>

One method that we’ve used is for cases such as this (when all other dimensions align and we’re simply adding a new parameter) is to just upload the data to the cloud bucket using [Rclone](https://rclone.org/) to an appropriate location and then use the [zarr.convenience.consolidate\_metadata](https://zarr.readthedocs.io/en/stable/api/zarr/convenience/index.html#zarr.convenience.consolidate_metadata) function, which works fairly quick even for remote S3 buckets.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [May 9, 2025, 9:49pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/5 "2025-05-09T21:49:21Z")

</div>

Just to clarify…this behavior:

> this is absurdly slow (30 min+), and times out in my testing before any writes happen. Writing to a new store takes ~30s.

is not normal. It should take ~30s to do the append. Something is wrong. There should be no need to resort to Rclone or other such workarounds.

---

<div class="post-metadata">

**Author:** ![gunbra32](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/gunbra32/32/2626_2.png) [@gunbra32](https://discourse.pangeo.io/u/gunbra32)\
**Post date:** [May 13, 2025, 3:15pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/6 "2025-05-13T15:15:29Z")

</div>

You may try [zappend](https://bcdev.github.io/zappend/), a robust tool for appending to large zarrs. It has logging and profiling support and allows for safe roll backs.

---

<div class="post-metadata">

**Author:** ![markmuetz](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/markmuetz/32/1805_2.png) [@markmuetz](https://discourse.pangeo.io/u/markmuetz)\
**Post date:** [May 13, 2025, 8:41pm UTC](https://discourse.pangeo.io/t/slow-append-to-existing-s3-zarr-store-using-xarray/5117/7 "2025-05-13T20:41:43Z")

</div>

Thanks all. I agree with @rabernat - I’d prefer to stick to off-the-shelf tools. I have been completely focused on getting things ready for a hackathon this week so have not had time to explore the options. When I’ve got more time next week, I’ll try to explore/diagnose the problem (using the `LoggingStore` option in the first instance). For now, I went with expedience - creating a new store and combining this with the existing as part of our `intake` catalog.