# RioXarray & Dask in a cloud env

**URL:** <https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979>\
**Category:** Cloud\
**Created:** [December 1, 2021, 4:38pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979 "2021-12-01T16:38:53Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![pl.marasco](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/pl.marasco/32/258_2.png) [@pl.marasco](https://discourse.pangeo.io/u/pl.marasco)\
**Post date:** [December 1, 2021, 4:38pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/1 "2021-12-01T16:38:54Z")

</div>

I’m working with some Modis data, the scenes have been downloaded on a local folder so that I can read through rioxarray. Once read the second step is to scale, thanks to rioxarray, the 500 m product to 250 m. Over our HPC I have no issue achieving the reprojection but the folder where files are is visible from all the nodes. If I try to achieve the same over the planetary computer I’m getting an error of readability of files. (I’ve tested the same on the Pangeo AWS Deployment but the situation is a little bit more complicated as seems to me that Rioxarray isn’t updated and provoke other errors)

The code looks like

> ds\_250 = rxr.open\_rasterio(filename\_250, chunks= ‘auto’)  
> ds\_500 = rxr.open\_rasterio(filename\_500, chunks= ‘auto’)
> 
> ds\_500\_match = ds\_500.rio.reproject\_match(ds\_250)

and the error looks like :

> `/tmp/modis/MOD09A1_22_07_2021321_MOD09A1.A2021321.h22v07.006.2021331124727.hdf:MOD_Grid_500m_Surface_Reflectance:sur_refl_b01: No such file or directory`

If I don’t specify any Dask cluster, the code runs smoothly but on the contrary when workers are involved isn’t possible to get the reprojection done.  
I’m prone to think that problem is related to the folder visibility from workers. (I would assume that workers are on different pods and can’t see the original location)  
Am I far from the truth? How I can solve this?

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [December 1, 2021, 6:00pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/2 "2021-12-01T18:00:24Z")

</div>

Can you share how you’re making the cluster? Is it a `GateayCluster()` or a `LocalCluster()`?

If you’re using a `GateayCluster`, then yes, your guess is correct. The workers won’t have access to notebook server’s filesystem, so they can’t read those files. This is touched on a bit at [Pangeo Cloud — Pangeo documentation](https://pangeo.io/cloud#cloud-object-storage).

So right now your options are to do things as you are with a LocalCluster (the Python environment on the Planetary Computer has 4 cores, but you might get I/O bound quickly), or shift to using [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/), and pass around a credential so that all your workers can read / write to that blob filesystem.

Finally, we (Microsoft / AI For Earth) do have Modis available through Blob Storage, but it’s under some flux right now. [AIforEarthDataSets/modis.md at main · microsoft/AIforEarthDataSets · GitHub](https://github.com/microsoft/AIforEarthDataSets/blob/main/data/modis.md) has the details, but right now it’s accessible from East US. We should have it available through the STAC API some time in the next 3-6 months. You might be able to read from there if that’s the right products (might be a bit slow if you’re using the Planetary Computer Hub, which is in West Europe).

---

<div class="post-metadata">

**Author:** ![pl.marasco](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/pl.marasco/32/258_2.png) [@pl.marasco](https://discourse.pangeo.io/u/pl.marasco)\
**Post date:** [December 1, 2021, 6:24pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/3 "2021-12-01T18:24:41Z")

</div>

Is through a GatewayCluster and what I’m trying to do is to avoid the local cluster to scale up the process.

About the Azure Blob I would assume that to let rioxarray able to read from a blob I would need fsspec. Am I right?

Up to now I was just grabbing ideas from this [example](https://nbviewer.org/github/microsoft/AIforEarthDataSets/blob/main/data/modis.ipynb) and definitely Microsoft Azure Blobs are the origin of my data.

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [December 1, 2021, 6:39pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/4 "2021-12-01T18:39:40Z")

</div>

> [@pl.marasco](#):
>
> About the Azure Blob I would assume that to let rioxarray able to read from a blob I would need fsspec. Am I right?

You would probably want to use adlfs or azure.storage.blob to list the files in directories. But once you have a URL you can just pass that to rioxarray. It uses GDAL’s HTTP handling to fetch the data.

So if you’re already getting the data from Blob Storage, I’d recommend streaming it straight into a DataArray rather than writing to a local filesystem ([Skip the download! Stream NASA data directly into Python objects | by Scott Henderson | pangeo | Medium](https://medium.com/pangeo/intake-stac-nasa-4cd78d6246b7) explains this nicely)

---

<div class="post-metadata">

**Author:** ![pl.marasco](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/pl.marasco/32/258_2.png) [@pl.marasco](https://discourse.pangeo.io/u/pl.marasco)\
**Post date:** [December 3, 2021, 2:45pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/5 "2021-12-03T14:45:54Z")

</div>

I’ve already the URL but seems that something goes wrong at GDAL level. Here below what I’ve been testing and the error:

```auto
rxr.open_rasterio('https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf', chunks= 'auto')

```

return an error like this:

```auto
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/file_manager.py in _acquire_with_cache_info(self, needs_lock)
    198 try:
--> 199 file = self._cache[self._key]
    200 except KeyError:

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/lru_cache.py in __getitem__ (self, key)
     52 with self._lock:
---> 53 value = self._cache[key]
     54 self._cache.move_to_end(key)

KeyError: [<function open at 0x7fe52d837700>, ('https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf',), 'r', (('sharing', False),)]

During handling of the above exception, another exception occurred:

CPLE_OpenFailedError Traceback (most recent call last)
rasterio/_base.pyx in rasterio._base.DatasetBase. __init__ ()

rasterio/_shim.pyx in rasterio._shim.open_dataset()

rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsicurl/https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

RasterioIOError Traceback (most recent call last)
/tmp/ipykernel_539/3845122745.py in <module>
----> 1 rxr.open_rasterio('https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf' class="ansi-blue-fg">, chunks= 'auto')

/srv/conda/envs/notebook/lib/python3.8/site-packages/rioxarray/_io.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, decode_times, decode_timedelta, **open_kwargs)
    831 else:
    832 manager = URIManager(rasterio.open, filename, mode="r", kwargs=open_kwargs)
--> 833 riods = manager.acquire()
    834 captured_warnings = rio_warnings.copy()
    835 

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/file_manager.py in acquire(self, needs_lock)
    179 An open file object, as returned by ``opener(*args, **kwargs)``.
    180 """
--> 181 file, _ = self._acquire_with_cache_info(needs_lock)
    182 return file
    183 

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/file_manager.py in _acquire_with_cache_info(self, needs_lock)
    203 kwargs = kwargs.copy()
    204 kwargs["mode"] = self._mode
--> 205 file = self._opener(*self._args, **kwargs)
    206 if self._mode == "w":
    207 # ensure file doesn't get overriden when opened again

/srv/conda/envs/notebook/lib/python3.8/site-packages/rasterio/env.py in wrapper(*args, **kwds)
    435 
    436 with env_ctor(session=session):
--> 437 return f(*args, **kwds)
    438 
    439 return wrapper

/srv/conda/envs/notebook/lib/python3.8/site-packages/rasterio/ __init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    218 # None.
    219 if mode == 'r':
--> 220 s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    221 elif mode == "r+":
    222 s = get_writer_for_path(path, driver=driver)(

rasterio/_base.pyx in rasterio._base.DatasetBase. __init__ ()

RasterioIOError: '/vsicurl/https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf' not recognized as a supported file format.

```

by the way, HDF is supported.  
Configuration is really scarce on [rasterio.env module](https://rasterio.readthedocs.io/en/latest/api/rasterio.env.html) and mostly focus on AWS but I’ve test as well this approach with the same result.

```auto
import rasterio
with rasterio.env.Env(AZURE_STORAGE_ACCOUNT='modissa', AZURE_STORAGE_SAS_TOKEN='sv=2019-12-12&si=modis-ro&sr=c&sig=jauBgV7FNXyhbjO60XSMr6AnFFd5sl%2BwXsNazkOOFjs%3D'):
    ds_250 = rxr.open_rasterio('https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf', chunks= 'auto')

```

At the end I’ve been testing at the real base through GDAL following [GDAL Virtual File Systems](https://gdal.org/user/virtual_file_systems.html#vsiaz-microsoft-azure-blob-files) documentation (not pretty sure about the account name but I didn’t had any other idea) but it fails as well with a timeout:

```auto
export $AZURE_STORAGE_ACCOUNT='modissa'
export AZURE_STORAGE_SAS_TOKEN='sv=2019-12-12&si=modis-ro&sr=c&sig=jauBgV7FNXyhbjO60XSMr6AnFFd5sl%2BwXsNazkOOFjs%3D'

gdalinfo /vsiaz/https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf

```

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [December 3, 2021, 3:50pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/6 "2021-12-03T15:50:27Z")

</div>

Your first example wouldn’t work

```auto
rxr.open_rasterio('https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf', chunks= 'auto')

```

since it doesn’t include the SAS token as a query parameter. I would have hoped that `rxr.open_rasterio('https://modissa.blob.core.windows.net/modis-006/MOD09Q1/22/07/2021289/MOD09Q1.A2021289.h22v07.006.2021299111058.hdf?sv=2019-12-12&si=modis-ro&sr=c&sig=jauBgV7FNXyhbjO60XSMr6AnFFd5sl%2BwXsNazkOOFjs%3D', chunks= 'auto')` would work, but it fails as well.

Looking at [GDAL Virtual File Systems (compressed, network hosted, etc...): /vsimem, /vsizip, /vsitar, /vsicurl, ... — GDAL documentation](https://gdal.org/user/virtual_file_systems.html#drivers-supporting-virtual-file-systems), it seems like GDAL doesn’t support opening netCDF files with a virtual filesystem:

> Notable exceptions are the netCDF, HDF4 and HDF5 drivers.

So unfortunately it seems like downloading the file to disk might be unavoidable (I’m not sure if one of xarray’s `open_dataset` engines would be able to read this file over the network).

---

<div class="post-metadata">

**Author:** ![pl.marasco](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/pl.marasco/32/258_2.png) [@pl.marasco](https://discourse.pangeo.io/u/pl.marasco)\
**Post date:** [December 3, 2021, 5:30pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/7 "2021-12-03T17:30:22Z")

</div>

The real problem isn’t downloading files locally but more how to make them lazy.  
As workers can’t get access to local files scalability is gone as tiles must be processed on the notebook pod.  
Most probably I’ve to rethink the entire approach.  
Thanks for all your inputs

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [December 5, 2021, 8:39pm UTC](https://discourse.pangeo.io/t/rioxarray-dask-in-a-cloud-env/1979/8 "2021-12-05T20:39:17Z")

</div>

Yes, each worker does part of the job in memory and writes that part back to cloud storage, as far as approach goes. Then you do something at the end…combine etc.