Pangeo Cloud Data Cookbook

Cloud
#1

At today’s telecon we discussed the idea of putting together a “Cloud Data Cookbook” to synthesize everything we have learned about how to work with geoscience data in the cloud. For example:

  • Reading / writing zarr in s3 / gcsfs / azure
  • Credentials management
  • COG stuff
  • Serializability

@paigem suggested this could be modeled off the cosima-cookbook

Pangeo Gallery could be a good way to publish this cookbook.

Please chime in or leave a :heart: if you would be willing to devote 1 hour to working on this over the next few weeks.

15 Likes
#2

Sure. I have benefited from some work here, so happy to see if I can try and explain some things. e.g. dask-era5/PANGEO-Sentinel-Intake-SA-OZLocation.ipynb at main · RichardScottOZ/dask-era5 · GitHub

#3

Here’s the Reading and Writing Data to S3 notebook I made for folks using the ESIP Qhub.

#4

I like this idea and would be happy to contribute some examples, e.g. for reading and writing to Zarr in gcsfs. If I have time later today, I’ll see if I can get this started by creating a new gallery - looks like there are some good instructions here.

Also, it’s the cosima-recipes (GitHub - COSIMA/cosima-recipes: Example recipes using the cosima-cookbook infrastructure) portion of the cosima-cookbook that is most relevant here, and is quite similar to Pangeo Gallery.

1 Like
#5

Progress update: I’ve started the process of making a new repo in the pangeo gallery for the cloud data cookbook. I’ll add an example notebook and aim to push the repo from my local directory in the next few days. Sorry I’ve been slow getting to this!

2 Likes
#6

It is a great idea! One suggestion would be providing a suggested template for people who want to contribute. This could make it easier for people to contribute. Also, it provides consistent user experiences for others who are looking for the cookbook/tutorials.

3 Likes