# Copy \> 5GB zarr file to S3

**URL:** <https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859>\
**Category:** Data\
**Created:** [November 27, 2023, 9:58pm UTC](https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859 "2023-11-27T21:58:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yves\_Moisan](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/yves_moisan/32/2070_2.png) [@Yves\_Moisan](https://discourse.pangeo.io/u/Yves_Moisan)\
**Post date:** [November 27, 2023, 9:58pm UTC](https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859/1 "2023-11-27T21:58:51Z")

</div>

My head is spinning when I look at the potential mix of xarray, zarr, fsspec, s3fs, boto3/glob for copying a large zarr file from a local FS to S3. By “large” I mean files \> 5 GB, which need to be split up to partial uploads otherwise they won’t get through.

Since I’m just copying a Zarr file, it seems redundant to open the local Zarr as ‘data’ with xarray and then “save as” Zarr over on S3, whichever way that can be done. Before delving into my tests, can anyone tell me the high level view for

`cp big_local_zarr to my_s3_bucket` ?

TIA

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 27, 2023, 10:14pm UTC](https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859/2 "2023-11-27T22:14:05Z")

</div>

If the file already exists in Zarr format on a local disk, you don’t have to use Xarray, Zarr, or Python at all. Just use your cloud provider’s transfer utility. With the [AWS CLI](https://aws.amazon.com/cli/), it’s something like this

```bash
aws s3 cp --recursive local_zarr_dir s3://myBucket/remote/path

```

There are many ways to tweak this to try to make it go faster. You could use rsync, [s3p](https://github.com/generalui/s3p), etc.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [November 28, 2023, 3:05pm UTC](https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859/3 "2023-11-28T15:05:07Z")

</div>

The python equivalent would be

```python
fs = fsspec.filesystem("s3")
fs.put(local_zarr_dir, "myBucket/remote/path", recursive=True)

```

which does use multi-part uploads for files that are big enough. Logging/callbacks for feedback are optional.

The AWS CLI might be faster, but for very large files if may make no difference, you’ll just max out the bandwidth.

fsspec’s generic module also has an “rsync” with a few of the optional of the namesake utility, and it will figure out which filesystem to pick for each URL. You don’t really need it here.

---

<div class="post-metadata">

**Author:** ![Yves\_Moisan](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/yves_moisan/32/2070_2.png) [@Yves\_Moisan](https://discourse.pangeo.io/u/Yves_Moisan)\
**Post date:** [November 28, 2023, 9:51pm UTC](https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859/4 "2023-11-28T21:51:03Z")

</div>

Thank you both. I try to work in notebooks and not worry about setting up CLI’s. (I’m using a CEPH S3 bucket)

For the record, here are 3 ways I uploaded the same 14 GB Zarr file, with

```auto
base_dir = "my/full/path/"
zarr_directory = "my.zarr"
endpoint_url='http://my_url:my_port'
os.environ['AWS_ACCESS_KEY_ID'] = access_key
os.environ['AWS_SECRET_ACCESS_KEY'] = secret_access_key
s3 = s3fs.S3FileSystem(key=access_key,
                       secret=secret_access_key,
                       endpoint_url=endpoint_url
                      )
fs = fsspec.filesystem('s3', endpoint_url=endpoint_url)
local_zarr_dir = base_dir + zarr_directory

```

# fsspec way

`fs.put(local_zarr_dir, "zarr-fsspec", recursive=True)`

zarr-fsspec bucket gets :  
Total Objects: 3091  
Total Size: 13.3 GiB

# Zarr way

```auto
store1 = zarr.DirectoryStore(base_dir + zarr_directory) # Source store
store2 = s3fs.S3Map(root='zarr-tt', s3=s3, check=False)
zarr.copy_store(store1, store2,if_exists='replace')

```

zarr-tt bucket gets :  
Total Objects: 3091  
Total Size: 13.3 GiB

# Fuse mount way

In a fresh directory mounted to the `zarr-fuse` bucket and copying the file into that mounted directory.

zarr-fuse bucket gets:  
Total Objects: 3095  
Total Size: 13.2 GiB

I end up with buckets without the Zarr filename, which I can of course fix before uploading contents, but it makes the copying trickier than for example copying a PMTile or a COG.

How much of an access penalty would I get if I were to use a gzipped Zarr, which would get me a “filename with the contents inside” ?

Cheers

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 29, 2023, 1:35am UTC](https://discourse.pangeo.io/t/copy-5gb-zarr-file-to-s3/3859/5 "2023-11-29T01:35:38Z")

</div>

Do not use gzip with Zarr. Instead, use a Zip file. It is supported natively by the format (e.g. [ZipStore](https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore)).

Similar discussion happening at [Any tips on avoiding high AWS S3 cost when storing Zarr's with lots of objects - #13 by rabernat](https://discourse.pangeo.io/t/any-tips-on-avoiding-high-aws-s3-cost-when-storing-zarrs-with-lots-of-objects/1977/13)