# New to Pangeo? A Quickstart Guide for Data Analysts and Engineers

**URL:** <https://discourse.pangeo.io/t/new-to-pangeo-a-quickstart-guide-for-data-analysts-and-engineers/2900>\
**Category:** Education\
**Created:** [November 4, 2022, 3:43pm UTC](https://discourse.pangeo.io/t/new-to-pangeo-a-quickstart-guide-for-data-analysts-and-engineers/2900 "2022-11-04T15:43:52Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 4, 2022, 3:43pm UTC](https://discourse.pangeo.io/t/new-to-pangeo-a-quickstart-guide-for-data-analysts-and-engineers/2900/1 "2022-11-04T15:43:52Z")

</div>

I’m involved in several projects that are hiring new data engineers to help on technical aspects of research projects. These folks need to quickly spin up on the main tools, technologies, and architectures used in our community. Rather than putting this information in private emails, I’ve decided to share it here on the forum.

_Disclaimer: this is a very biased list that comes from my personal experience and perspective!_ I’m actively seeking feedback to include more resources. Leave your comments and suggestions below, and I’ll update the post on a rolling basis.

# Audience

This is written for data analysts engineers who already have general experience but do not have experience specific to geospatial, weather, climate, ocean etc. data. In particular, I assume you already have the following skills:

- Can use the core scientific python packages **numpy** and **pandas** for data loading and processing and **matplotlib** for data visualization.
- Comfortable writing and running python code in Jupyter notebooks and standalone scripts.
- Comfortable using git and github.
- Basic understanding of how to package and share python code.
- Solid foundation in the fundamentals of data: data types (e.g. float, int, text, etc), data volumes, throughput, latency, arrays vs. tables, schemas, binary files, json, csv, etc.

# Learn about the Data Model and Data Formats

Here is a page I wrote for my course with a quick overview of data models and formats.

> **[All About Data](https://earth-env-data-science.github.io/lectures/data.html)**
>
> This lecture attempts to give a broad overview of Earth and Environment Science data formats, exchange protocols, and best practices. What is (are) data?: Ultimately, as we know, all digital data a...

Here are some other resources to dig deeper

- [NetCDF: The NetCDF Data Model](https://docs.unidata.ucar.edu/netcdf-c/current/netcdf_data_model.html)
- [Data Structures](https://docs.xarray.dev/en/stable/user-guide/data-structures.html)

# Learn to Use the Core Libraries

Go through these tutorials

## Xarray

- [Welcome to the Xarray Tutorial!](https://tutorial.xarray.dev/intro.html)
- Material from Ryan’s class:
  - [Xarray Fundamentals — Earth and Environmental Data Science](https://earth-env-data-science.github.io/lectures/xarray/xarray.html)
  - [Assignment: Xarray Fundamentals with Atmospheric Radiation Data — Earth and Environmental Data Science](https://earth-env-data-science.github.io/assignments/basic_xarray.html)
  - [Xarray Interpolation, Groupby, Resample, Rolling, and Coarsen — Earth and Environmental Data Science](https://earth-env-data-science.github.io/lectures/xarray/xarray-part2.html)
  - [Assignment: More Xarray with El Niño-Southern Oscillation (ENSO) Data — Earth and Environmental Data Science](https://earth-env-data-science.github.io/assignments/more_xarray.html)

- [https://foundations.projectpythia.org/core/xarray.html](https://foundations.projectpythia.org/core/xarray.html)

## Dask

- [Welcome to the Dask Tutorial — Dask Tutorial documentation](https://tutorial.dask.org/00_overview.html)

## Zarr

- [https://zarr.readthedocs.io/en/stable/tutorial.html](https://zarr.readthedocs.io/en/stable/tutorial.html)
- Ryan’s Zarr tutorial for OGC Cloud Native Outreach Event
  - [https://www.youtube.com/watch?v=unGL07trSjA](https://www.youtube.com/watch?v=unGL07trSjA)
  - [GitHub - zarr-developers/tutorials: Notebooks and other content for Zarr tutorials](https://github.com/zarr-developers/tutorials)

## RasterIO and RioXarray

- [Python Quickstart — rasterio 1.5.0.dev documentation](https://rasterio.readthedocs.io/en/latest/quickstart.html)
- [Raster processing using Python Tools: Working with Raster Datasets](https://geohackweek.github.io/raster/04-workingwithrasters/)
- [Welcome to rioxarray’s documentation! — rioxarray 0.20.0 documentation](https://corteva.github.io/rioxarray/stable/)

## GeoPandas

- [Introduction to GeoPandas — GeoPandas 1.1.1+0.ge9b58ce.dirty documentation](https://geopandas.org/en/stable/getting_started/introduction.html)
- [Geopandas Tutorial — Pangeo Gallery documentation](http://gallery.pangeo.io/repos/pangeo-data/pangeo-tutorial-gallery/geopandas.html)

# Understand Advanced Use Cases and Challenges

These are documented on our forum

> [@Best practices to go from 1000s of netcdf files to analyses on a HPC cluster?](https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster/588):
>
> What? 8759 netcdf files totalling 17TB of HYCOM ocean model (u,v) velocity data at two depth levels (and bottom velocity), at hourly time steps. So the total data arrays of interest are [9000 (X) by 7055 (Y) by 8759 (time) by 2 (Depth)] for both u and v; Using nco tools, I can reduce these files as an example to 365 netcdf files, totaling 4.3TB, one file per day (24 hourly steps), for one depth level only, so data arrays are 9000 (X) by 7055 (Y) by 8759 (time) for the u,v components. How? Us…

> [@Optimizing climatology calculation with Xarray and Dask](https://discourse.pangeo.io/t/optimizing-climatology-calculation-with-xarray-and-dask/2453):
>
> I just wanted to leave some experience I am having on a toy project I used for teaching xarray + dask (I teach only the basic premises, as I don’t feel very confident in understanding what’s going on under the hood). Here is [the notebook](https://gist.github.com/fmaussion/48a33deae4fdee1f9433ec58c8956ac6). Anyways, the short summary is: averaging over multiple files on a large (30+ Gb) dataset on a local cluster works well, BUT dask “pollutes” my swap memory: it’s unclear to me as to why this happens, given that workers with limited memory are not supposed to …

> [@Tables, (x)arrays, and rasters¶](https://discourse.pangeo.io/t/tables-x-arrays-and-rasters/1945):
>
> Hi all, I’ve been playing around with some ideas for working with geospatial raster data. I’d be curious for any feedback you have. The core question: what’s the best data model for raster data in Python? Unsurprisingly, I think the answer is “it depends”. Let’s use work through a concrete task and evaluate the various options. Suppose we wanted to compute NDVI for all the scenes captured by Landsat 8 over a couple of hours. (Full notebook at [Jupyter Notebook Viewer](https://nbviewer.org/github/TomAugspurger/rasterpandas/blob/main/example.ipynb)) We’ll use the Planetary C…

# Explore the more Experimental Libraries and Projects

These are all recent projects that have emerged from this community in response to specific user needs.

> **[GitHub - pangeo-data/rechunker: Disk-to-disk chunk transformation for chunked...](https://github.com/pangeo-data/rechunker)**
>
> Disk-to-disk chunk transformation for chunked arrays.

> **[Pangeo-Forge](https://pangeo-forge.org/)**
>
> A cloud-native data repository for ocean, weather, and climate science.

> **[GitHub - xarray-contrib/xbatcher: Batch generation from xarray datasets](https://github.com/xarray-contrib/xbatcher)**
>
> Batch generation from xarray datasets

> **[GitHub - google/xarray-beam: Distributed Xarray with Apache Beam](https://github.com/google/xarray-beam)**
>
> Distributed Xarray with Apache Beam

# Learn about Cloud Storage

Cloud data storage is an area where we are really lacking documentation, tutorials, guides, etc.

Some high level material about “why cloud?” can be found here:

- [Cloud-Native Repositories for Big Scientific Data | IEEE Journals & Magazine | IEEE Xplore](https://ieeexplore.ieee.org/document/9354557)
- [https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2020AV000354](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2020AV000354)

More technical material

- [https://pangeo.io/data.html](https://pangeo.io/data.html) (Note that this is pretty old and out of date)
- [Big Arrays, Fast: Profiling Cloud Storage Read Throughput — Pangeo Gallery documentation](https://gallery.pangeo.io/repos/earthcube2020/ec20_abernathey_etal/cloud_storage.html)
- The “Zarr in the Cloud” section of Ryan’s [Zarr Tutorial](https://github.com/zarr-developers/tutorials/blob/main/zarr_cloud_native_geospatial_2022.ipynb) has a bit of material about how cloud data access works.

Clearly this is an area where we have work to do in terms of documenting workflows. Does anyone have any more material they can suggest here?

---

_[View the full topic](https://discourse.pangeo.io/t/new-to-pangeo-a-quickstart-guide-for-data-analysts-and-engineers/2900)._