The National Water Model Reanalysis Zarr dataset on AWS

The National Water Model Reanalysis v2.0 is a 26 year simulation of 2.7 million rivers in the US at hourly intervals. The data was delivered as part of the NOAA Big Data Program to AWS as 227,000+ hourly NetCDF files.

I downloaded (!) and then converted the streamflow files from the reanalysis to a single Zarr dataset with chunks that had a dimension of 100 in the time dimension to facilitate the extraction of time series data. I used rechunker, and to deal with potential input data problems, I looped through the data in month-long chunks, writing and then appending to Zarr at the end of every month. This way I could correct issues with the input data (missing data and bad time stamps), the try again, and on success, append the chunk. See the full notebook for details on the conversion.

The result is a single Zarr dataset on AWS that can be used for time series extraction as well as mapping.

Here’s proof: A sample analysis notebook using this new Zarr dataset.

In this notebook we use a cluster of 20 workers on a Qhub Dask Gateway cluster to both extract time series and compute the annual mean river discharge for a specific year in less than 2 minutes of wall clock.

5 Likes

Rich, Thanks for leading that effort on using ZARR.
If users want to extract and download specific rivers (e.g. certain variables from rivers within a specified time period and bounding box), what would be the recommended procedure?

@LloydBalfour-NOAA, welcome to the Pangeo community and that’s a great question!

Of course we advocate for people to work on Cloud next to the data (in this case that would be AWS us-west-2), but if folks need to download the data they could run a notebook like this, which extracts just the streamflow variable from rivers in the Gulf of Maine region from 2000-01-01 to the present.

If they want to run a script from the command line, they could do:

python Subset_GOM_local.py 

using this script.

To run the notebook or script using a local python environment:

  1. If conda is not installed, follow these instructions, but don’t create the IOOS environment.
  2. Create an custom conda environment using this environment file: conda env create -f nwm_subset_env.yml

To run the notebook with Docker:

  1. Create your own customized container using the environment above, or just use the pangeo/pangeo-notebook container: docker run --rm -i --tty pangeo/pangeo-notebook bash
1 Like

Rich,

Fantastic!!!

Appreciate this.

Regards,
Lloyd Balfour Sr.
Data Architect (TESA)
GID (OWP/NWC/NOAA).
205-960-4754.

So cool Rich! Thanks for sharing.

Would be great to get this into a catalog. However, our catalog is kind of broken now as we are transitioning to Pangeo Forge.

cc @cisaacstern

Incredible, @rsignell !

@rabernat, I forget who told me about this api (maybe @TomAugspurger?), but seems like GitHub - stac-utils/stac-fastapi: STAC API implementation with FastAPI. might be one accessible way of rebooting the catalog.

1 Like