New to Pangeo? A Quickstart Guide for Data Analysts and Engineers

[rabernat]

I’m involved in several projects that are hiring new data engineers to help on technical aspects of research projects. These folks need to quickly spin up on the main tools, technologies, and architectures used in our community. Rather than putting this information in private emails, I’ve decided to share it here on the forum.

Disclaimer: this is a very biased list that comes from my personal experience and perspective! I’m actively seeking feedback to include more resources. Leave your comments and suggestions below, and I’ll update the post on a rolling basis.

Audience

This is written for data analysts engineers who already have general experience but do not have experience specific to geospatial, weather, climate, ocean etc. data. In particular, I assume you already have the following skills:

• Can use the core scientific python packages numpy and pandas for data loading and processing and matplotlib for data visualization.
• Comfortable writing and running python code in Jupyter notebooks and standalone scripts.
• Comfortable using git and github.
• Basic understanding of how to package and share python code.
• Solid foundation in the fundamentals of data: data types (e.g. float, int, text, etc), data volumes, throughput, latency, arrays vs. tables, schemas, binary files, json, csv, etc.

Learn about the Data Model and Data Formats

Here is a page I wrote for my course with a quick overview of data models and formats.

All About Data

Here are some other resources to dig deeper

• NetCDF: The NetCDF Data Model
• Data Structures

Learn to Use the Core Libraries

Go through these tutorials

Xarray

• Welcome to the Xarray Tutorial!
• Material from Ryan’s class:
  • Xarray Fundamentals — Earth and Environmental Data Science
  • Assignment: Xarray Fundamentals with Atmospheric Radiation Data — Earth and Environmental Data Science
  • Xarray Interpolation, Groupby, Resample, Rolling, and Coarsen — Earth and Environmental Data Science
  • Assignment: More Xarray with El Niño-Southern Oscillation (ENSO) Data — Earth and Environmental Data Science
• Xarray — Pythia Foundations

Dask

• Welcome to the Dask Tutorial — Dask Tutorial documentation

Zarr

• Tutorial — zarr 2.16.1 documentation
• Ryan’s Zarr tutorial for OGC Cloud Native Outreach Event
  • https://www.youtube.com/watch?v=unGL07trSjA
  • GitHub - zarr-developers/tutorials: Notebooks and other content for Zarr tutorials

RasterIO and RioXarray

• Python Quickstart — rasterio 1.4dev documentation
• Raster processing using Python Tools: Working with Raster Datasets
• Welcome to rioxarray’s documentation! — rioxarray 0.15.0 documentation

GeoPandas

• Introduction to GeoPandas — GeoPandas 0+untagged.50.g9a9f097.dirty documentation
• Geopandas Tutorial — Pangeo Gallery documentation

Understand Advanced Use Cases and Challenges

These are documented on our forum

Best practices to go from 1000s of netcdf files to analyses on a HPC cluster?

What? 8759 netcdf files totalling 17TB of HYCOM ocean model (u,v) velocity data at two depth levels (and bottom velocity), at hourly time steps. So the total data arrays of interest are [9000 (X) by 7055 (Y) by 8759 (time) by 2 (Depth)] for both u and v; Using nco tools, I can reduce these files as an example to 365 netcdf files, totaling 4.3TB, one file per day (24 hourly steps), for one depth level only, so data arrays are 9000 (X) by 7055 (Y) by 8759 (time) for the u,v components. How? Us…

Optimizing climatology calculation with Xarray and Dask

I just wanted to leave some experience I am having on a toy project I used for teaching xarray + dask (I teach only the basic premises, as I don’t feel very confident in understanding what’s going on under the hood). Here is the notebook. Anyways, the short summary is: averaging over multiple files on a large (30+ Gb) dataset on a local cluster works well, BUT dask “pollutes” my swap memory: it’s unclear to me as to why this happens, given that workers with limited memory are not supposed to …

Tables, (x)arrays, and rasters¶

Hi all, I’ve been playing around with some ideas for working with geospatial raster data. I’d be curious for any feedback you have. The core question: what’s the best data model for raster data in Python? Unsurprisingly, I think the answer is “it depends”. Let’s use work through a concrete task and evaluate the various options. Suppose we wanted to compute NDVI for all the scenes captured by Landsat 8 over a couple of hours. (Full notebook at Jupyter Notebook Viewer) We’ll use the Planetary C…

Explore the more Experimental Libraries and Projects

These are all recent projects that have emerged from this community in response to specific user needs.

GitHub - pangeo-data/rechunker: Disk-to-disk chunk transformation for chunked...

Pangeo-Forge

GitHub - xarray-contrib/xbatcher: Batch generation from xarray datasets

GitHub - google/xarray-beam: Distributed Xarray with Apache Beam

Learn about Cloud Storage

Cloud data storage is an area where we are really lacking documentation, tutorials, guides, etc.

Some high level material about “why cloud?” can be found here:

• Cloud-Native Repositories for Big Scientific Data | IEEE Journals & Magazine | IEEE Xplore
• https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2020AV000354

More technical material

• Pangeo and Data — Pangeo documentation (Note that this is pretty old and out of date)
• Big Arrays, Fast: Profiling Cloud Storage Read Throughput — Pangeo Gallery documentation
• The “Zarr in the Cloud” section of Ryan’s Zarr Tutorial has a bit of material about how cloud data access works.

Clearly this is an area where we have work to do in terms of documenting workflows. Does anyone have any more material they can suggest here?

[raybellwaves]

For intermediate/advanced cloud storage. Could put something on limits of PUT and GET requests e.g. to_zarr to s3 with asynchronous=False · Discussion #5869 · pydata/xarray · GitHub

[akpetty]

I smell the start of a bootcamp…

[geynard]

Thanks Ryan, that’s awesome!

This is really complete, and I agree we lack some documentation about Cloud storage and how to store data at scale. You already provide some content with your Zarr tutorials and other links, and you also have some interesting content in your Pangeo slides decks. For example this one.

We also tried to explain things like chunking in recent courses, maybe this is relevant?

An I hesitated to share this because this is a bit rough and hard to use, but I made some contents for courses I gave in the past years: GitHub - CNES/big-data-processing-course: Course on Big Data Processing and Cloud computing, for up to 4 days training.. There is for example things about Cloud and Big Data infrastructure or on Object storage (but this comes a lot from your slides ).