# Upload zarr data directly to S3

**URL:** <https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040>\
**Category:** Data\
**Created:** [December 29, 2021, 5:13pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040 "2021-12-29T17:13:50Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Murugesh](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/murugesh/32/1329_2.png) [@Murugesh](https://discourse.pangeo.io/u/Murugesh)\
**Post date:** [December 29, 2021, 5:13pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/1 "2021-12-29T17:13:50Z")

</div>

Hi all,

I have been trying to upload zarr data directly to S3. But I cannot achieve it yet.

I can read the data from S3 using fsspec filestream. There is no problem with that. But when I try to upload, I can’t use the same method.

```auto
data.to_zarr("s3:/sample-storage/new.zarr", 
                 consolidated=True, mode='w',
                storage_options = {
                "key": access_key,
                "secret": secret,
                "endpoint": end_point
                })

```

Also there is no much information or resource available on the internet, other than documentation of xarray, zarr and pangeo discourse forum.

Can someone help with this?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [December 30, 2021, 8:37pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/2 "2021-12-30T20:37:20Z")

</div>

For some reason, `to_zarr` never got the storage\_options treatment. What you can instead, is create the mapper by hand

```auto
mapper = fsspec.get_mapper("s3://sample_storage/new.zarr",
    key=, secret=, endpoint=)
data.to_zarr(mapper)

```

---

<div class="post-metadata">

**Author:** ![Murugesh](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/murugesh/32/1329_2.png) [@Murugesh](https://discourse.pangeo.io/u/Murugesh)\
**Post date:** [December 31, 2021, 8:00am UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/3 "2021-12-31T08:00:26Z")

</div>

Thanks for the help @martindurant

I tried the same. But unfortunately it is throwing the following error.

`TypeError: __init__ () got an unexpected keyword argument 'endpoint'`

I tried multiple naming conventions for each of the three variables like key, access\_key, access, secret\_key, endpoint\_url etc. But all returns the same error.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [December 31, 2021, 4:46pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/4 "2021-12-31T16:46:08Z")

</div>

Ah, I didn’t actually check that the arguments you provided for storage\_options were reasonable. I believe it should be

```auto
mapper = fsspec.get_mapper("s3://sample_storage/new.zarr",
    key=, secret=, client_kwargs=dict(endpoint_url= ))
data.to_zarr(mapper)

```

---

<div class="post-metadata">

**Author:** ![rsignell](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rsignell/32/447_2.png) [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Post date:** [December 31, 2021, 4:58pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/5 "2021-12-31T16:58:04Z")

</div>

I just tried a small test case on our USGS HPC system, reading a 1.2GB uncompressed NetCDF4 file and then writing zarr directly to s3, but it took a super long time – 23 minutes.  
I then tried it with my usual method of writing locally and then copying to s3 using the AWS CLI, and it took 6 s to write, and then 20 s to transfer, so 26s total.

This dataset has 13Mb chunks, but only 1 chunk per variable, with 188 data variables.

@martindurant, do you have an explanation for this behavior?  
(the fsspec version I used was 2021.10.1, in case that’s relevant)

 ![zarr_to_s3](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/1/1d7db1a99a3610169b5451a2fb42a9a8149dac12.png)

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [December 31, 2021, 5:13pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/6 "2021-12-31T17:13:30Z")

</div>

The dask local client and dashboard could be used to profile, or even just `%%snakeviz`. I don’t have an immediate suggestion. It would be useful to know how many total chunks there is the data.

---

<div class="post-metadata">

**Author:** ![Murugesh](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/murugesh/32/1329_2.png) [@Murugesh](https://discourse.pangeo.io/u/Murugesh)\
**Post date:** [January 3, 2022, 8:26am UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/7 "2022-01-03T08:26:36Z")

</div>

It worked perfectly fine for me. I needed to add “https://” before the endpoint\_url as well.

Thank you so much for the help!!

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [January 3, 2022, 5:26pm UTC](https://discourse.pangeo.io/t/upload-zarr-data-directly-to-s3/2040/8 "2022-01-03T17:26:12Z")

</div>

23 minutes vs 26 seconds is indeed a huge difference! Rich could you post the profile information from your experiments? That would be super helpful.
