# How to read multiple zarr archives at once from s3?

**URL:** <https://discourse.pangeo.io/t/how-to-read-multiple-zarr-archives-at-once-from-s3/2564>\
**Category:** Uncategorized\
**Created:** [July 5, 2022, 1:37pm UTC](https://discourse.pangeo.io/t/how-to-read-multiple-zarr-archives-at-once-from-s3/2564 "2022-07-05T13:37:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [July 5, 2022, 1:37pm UTC](https://discourse.pangeo.io/t/how-to-read-multiple-zarr-archives-at-once-from-s3/2564/1 "2022-07-05T13:37:54Z")

</div>

Hello,  
I am succesfully using zarr+s3 but now I want to improve my solution.

```python
def get_xarray_from_s3(bucket_name: str, dataset_name: str) -> xarray.Dataset:
    """
    Basic function to take xarray data from s3 bucket

    Args:
        bucket_name: name of the bucket
        dataset_name: refined name of the dataset, can be a path too

    Returns:
        xarray.Dataset object with the data from s3

    """
    check_aws_env_vars()

    s3_out = s3fs.S3FileSystem(anon=False)
    return xarray.open_zarr(
        store=s3fs.S3Map(
            root=f"s3:///{bucket_name}/{dataset_name}.zarr", s3=s3_out, check=False
        )
    )

```

This is my latest running version.

Now I expect something like that, where I am able to pass a list of datasets in the bucket:

```python
def get_xarray_from_s3_multiple(bucket_name: str, dataset_names: List[str]) -> xarray.Dataset:
    """
    Basic function to take xarray data from s3 bucket

    Args:
        bucket_name: name of the bucket
        dataset_names: refined name of the dataset, can be a path too

    Returns:
        xarray.Dataset object with the data from s3

    """
    check_aws_env_vars()

    s3_out = s3fs.S3FileSystem(anon=False)
    fileset = [s3_out.open(f"s3:///{bucket_name}/{dataset_name}.zarr") for dataset_name in dataset_names]
    return xarray.open_mfdataset(fileset, engine='zarr', consolidated=True)

```

But this is not working due to this issue:

```python
ValueError: Starting with Zarr 2.11.0, stores must be subclasses of BaseStore, if your store exposes the MutableMapping interface wrap it in Zarr.storage.KVStore. Got ()

```

I tried to wrap the s3\_out.open() object by using Zarr.storage.KVStore but then I am running into TypeError.

So I hope anyone of you will know how to access multiple zarr archives at once.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [July 5, 2022, 6:20pm UTC](https://discourse.pangeo.io/t/how-to-read-multiple-zarr-archives-at-once-from-s3/2564/2 "2022-07-05T18:20:46Z")

</div>

Would you mind trying with an older version of zarr? It sounds like xarray may need to be updated for the multi-dataset case with newer zarr.

Note that kerchunk allows you to build a single virtual dataset out of many datasets, specifying how to merge them. For the case of zarr inputs, this is not particularly useful, but you could consider it a workaround in this case.

---

<div class="post-metadata">

**Author:** ![andersy005](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/andersy005/32/31_2.png) [@andersy005](https://discourse.pangeo.io/u/andersy005)\
**Post date:** [July 5, 2022, 7:28pm UTC](https://discourse.pangeo.io/t/how-to-read-multiple-zarr-archives-at-once-from-s3/2564/3 "2022-07-05T19:28:58Z")

</div>

> [@meteoDaniel](#):
>
> `fileset = [s3_out.open(f"s3:///{bucket_name}/{dataset_name}.zarr") for dataset_name in dataset_names]`

Is the `.open()` call intentional here? You may want to create a mapper object instead.

```python
...
fileset = [s3fs.S3Map(
            root=f"s3:///{bucket_name}/{dataset_name}.zarr", s3=s3_out, check=False
        ) for dataset_name in dataset_names]
return xarray.open_mfdataset(fileset, engine='zarr', consolidated=True)

```

Another option is to pass the list of stores to `xr.open_mfdataset()` directly:

```python
....
fileset = [f"s3:///{bucket_name}/{dataset_name}.zarr" for dataset_name in dataset_names]
return xarray.open_mfdataset(fileset, engine='zarr', consolidated=True)

```

---

<div class="post-metadata">

**Author:** ![meteoDaniel](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/meteodaniel/32/1584_2.png) [@meteoDaniel](https://discourse.pangeo.io/u/meteoDaniel)\
**Post date:** [July 5, 2022, 9:13pm UTC](https://discourse.pangeo.io/t/how-to-read-multiple-zarr-archives-at-once-from-s3/2564/4 "2022-07-05T21:13:27Z")

</div>

Thanks @andersy005 , this works for me so far. Thanks a lot