# Struglin with Zarr files reading in Azure Blob Storage using Databricks

**URL:** <https://discourse.pangeo.io/t/struglin-with-zarr-files-reading-in-azure-blob-storage-using-databricks/2940>\
**Category:** Cloud\
**Created:** [November 14, 2022, 12:28pm UTC](https://discourse.pangeo.io/t/struglin-with-zarr-files-reading-in-azure-blob-storage-using-databricks/2940 "2022-11-14T12:28:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![INAVAS](https://avatars.discourse-cdn.com/v4/letter/i/c5a1d2/32.png) [@INAVAS](https://discourse.pangeo.io/u/INAVAS)\
**Post date:** [November 14, 2022, 12:28pm UTC](https://discourse.pangeo.io/t/struglin-with-zarr-files-reading-in-azure-blob-storage-using-databricks/2940/1 "2022-11-14T12:28:36Z")

</div>

**After experiencing very long times in the process of reading and slicing monthly grib files of my own grib database, I have investigated in the internet and found that Zarr might be my solution.**

**I have managed to generate a zarr file but i am strugling to read it. I get the error KeyError: '.zmetadata" even tough the file is in there. Please see picture attached**

**I would appreciate any clue on why the libary cannot find the .zmetadata.**

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/8/8d9b40052aa81b099d5bd1c980df2b1bc874d1ae.png)

_ **Code** :_  
_import zarr_  
_from azure.storage.blob import BlobServiceClient_

_store = zarr.ABSStore( prefix=‘zarr\_example’,client=client, blob\_service\_kwargs={‘is\_emulated’: False})_

_compressor = zarr.Blosc(cname=‘zstd’, clevel=3)_  
_encoding = {vname: {‘compressor’: compressor} for vname in ds.data\_vars}_  
_ds.to\_zarr(store=store, encoding=encoding, consolidated=True)_

_zarr\_ds = xr.open\_zarr(store=‘zarr\_example’, consolidated=True)_  
_zarr\_ds_

_ **Error** :_

_KeyError Traceback (most recent call last)_  
 _in_   
_----\> 1 zarr\_ds = xr.open\_zarr(store=‘zarr\_example’, consolidated=True)_

- 

```
 2 zarr_ds*

```

\*/databricks/python/lib/python3.8/site-packages/xarray/backends/zarr.py in open\_zarr(store, group, synchronizer, chunks, decode\_cf, mask\_and\_scale, decode\_times, concat\_characters, decode\_coords, drop\_variables, consolidated, overwrite\_encoded\_chunks, chunk\_store, storage\_options, decode\_timedelta, use\_cftime, \*_kwargs)_

- 785 }\*
- 786 \*  
 _→ 787 ds = open\_dataset(_
- 788 filename\_or\_obj=store,\*
- 789 group=group,\*

\*/databricks/python/lib/python3.8/site-packages/xarray/backends/api.py in open\_dataset(filename\_or\_obj, engine, chunks, cache, decode\_cf, mask\_and\_scale, decode\_times, decode\_timedelta, use\_cftime, concat\_characters, decode\_coords, drop\_variables, inline\_array, backend\_kwargs, \*_kwargs)_

- 537 \*
- 538 overwrite\_encoded\_chunks = kwargs.pop(“overwrite\_encoded\_chunks”, None)\*  
 _→ 539 backend\_ds = backend.open\_dataset(_
- 540 filename\_or\_obj,\*
- 541 drop\_variables=drop\_variables,\*

_/databricks/python/lib/python3.8/site-packages/xarray/backends/zarr.py in open\_dataset(self, filename\_or\_obj, mask\_and\_scale, decode\_times, concat\_characters, decode\_coords, drop\_variables, use\_cftime, decode\_timedelta, group, mode, synchronizer, consolidated, chunk\_store, storage\_options, stacklevel)_

- 846 \*
- 847 filename\_or\_obj = \_normalize\_path(filename\_or\_obj)\*  
 _→ 848 store = ZarrStore.open\_group(_
- 849 filename\_or\_obj,\*
- 850 group=group,\*

_/databricks/python/lib/python3.8/site-packages/xarray/backends/zarr.py in open\_group(cls, store, mode, synchronizer, group, consolidated, consolidate\_on\_close, chunk\_store, storage\_options, append\_dim, write\_region, safe\_chunks, stacklevel)_

- 398 elif consolidated:\*
- 399 # TODO: an option to pass the metadata\_key keyword\*  
\*–\> 400 zarr\_group = zarr.open\_consolidated(store, \*_open\_kwargs)_
- 401 else:\*
- 402 zarr\_group = zarr.open\_group(store, \*_open\_kwargs)_

\*/databricks/python/lib/python3.8/site-packages/zarr/convenience.py in open\_consolidated(store, metadata\_key, mode, \*_kwargs)_

- 1298 \*
- 1299 # setup metadata store\*  
 _→ 1300 meta\_store = ConsolidatedStoreClass(store, metadata\_key=metadata\_key)_
- 1301 \*
- 1302 # pass through\*

_/databricks/python/lib/python3.8/site-packages/zarr/storage.py in **init** (self, store, metadata\_key)_

- 2859 \*
- 2860 # retrieve consolidated metadata\*  
 _→ 2861 meta = json\_loads(self.store[metadata\_key])_
- 2862 \*
- 2863 # check format of consolidated metadata\*

_/databricks/python/lib/python3.8/site-packages/zarr/storage.py in **getitem** (self, key)_

- 1072 return self.\_fromfile(filepath)\*
- 1073 else:\*  
 _→ 1074 raise KeyError(key)_
- 1075 \*
- 1076 def **setitem** (self, key, value):\*

_KeyError: ‘.zmetadata’_

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [November 14, 2022, 12:39pm UTC](https://discourse.pangeo.io/t/struglin-with-zarr-files-reading-in-azure-blob-storage-using-databricks/2940/2 "2022-11-14T12:39:55Z")

</div>

I find it helpful to ensure that you can access the file using the blob storage client directly, to ensure that you get the storage account, container name, and prefix sorted out properly.

Using the zarr store at “[https://daymeteuwest.blob.core.windows.net/daymet-zarr/annual/hi.zarr](https://daymeteuwest.blob.core.windows.net/daymet-zarr/annual/hi.zarr)” as an example, the storage account is `daymeteuwest`, the container is `daymet-zarr`, and the prefix is `annual/hi.zarr`.

```auto
In [12]: import zarr, azure.storage.blob, xarray as xr

In [13]: client = azure.storage.blob.ContainerClient("https://daymeteuwest.blob.core.windows.net", "daymet-zarr")

In [14]: assert client.get_blob_client("annual/hi.zarr/.zmetadata").exists()

In [15]: store = zarr.ABSStore(client=client, prefix="annual/hi.zarr")

In [16]: ds = xr.open_dataset(store, engine="zarr")

```

---

<div class="post-metadata">

**Author:** ![INAVAS](https://avatars.discourse-cdn.com/v4/letter/i/c5a1d2/32.png) [@INAVAS](https://discourse.pangeo.io/u/INAVAS)\
**Post date:** [November 14, 2022, 12:49pm UTC](https://discourse.pangeo.io/t/struglin-with-zarr-files-reading-in-azure-blob-storage-using-databricks/2940/3 "2022-11-14T12:49:48Z")

</div>

> [@TomAugspurger](#):
>
> `assert client.get_blob_client("annual/hi.zarr/.zmetadata").exists()`

Thank you very much for the answer.

I got True in get\_blob\_client…

> client.get\_blob\_client(“zarr\_example/.zmetadata”).exists()  
> Out[24]: True