WxData 1.6 is released on both conda-forge and pypi and has a lot of exciting new features.
In this post I will cover both the new features as well as provide an overview of existing features for those who have not heard of this Python package before.
Before we begin, here are some useful links:
-
WxData Documentation (including installation instructions) & Jupyter Lab Tutorials: wxdata 路 PyPI
WxData Overview
Q: What is WxData and what makes it unique from other packages people use to access and process meteorological data?
WxData is an open-source Python package that has a series of clients, post-processors and tools to help fellow meteorologists set up automated data pipelines for various types of weather data and make working with weather data as easy as possible. When writing this package, I strongly keep in mind people who are new to both Python, writing computer programs in general and working with weather data in Python. WxData also has both observational and forecast model data clients.
WxData also has built in support for those working behind a firewall (this is very common in workplaces). These types of users can pass in the address:port of a proxy server they want to use which tells the client the correct path from the user鈥檚 PC to the data server. The client鈥檚 default settings assume the user is not going through a proxy server, users that are going through a proxy server will have to define their proxy information and pass it into the client. Visit for more information on configuring a proxy server: wxdata 路 PyPI
Clients
There are two general types of clients in WxData:
-
End-To-End (E2E) Clients: These clients are very much geared for beginners who would like to set up an automated pipeline in Python for weather data retrieval and processing.
wxdata.gfs_0p25()is an example of an E2E client in this package. These clients work well in an operational setting as they:
i) Scan the data server for the latest available data.
ii) Download the latest available data and automatically deletes old files prior to downloading new ones (preserves system memory).
iii) Build the local directories and sorts the files into their proper directories upon downloading.
iv) a. Post-process GRIB data by decoding all the GRIB variable keys into a plain-language (i.e.'r2' --> '2m_relative_humidity') format as well as resolve dataset build errors where you need tofilter_by_keys={}.
b. Post-process observational RAWS data as station meta-data (i.e. latitude and longitude) is stored separately from the weather and fuels data in the FEMS database. -
Raw Clients: These clients are geared for more advanced users who wish to use pieces of WxData to set up their pipeline but may not want the complete E2E pipeline which allows for more customization in the workflow.
wxdata.client.get_gridded_data()is an example of a raw data client in this package. Raw clients still have some house keeping to build the local directory paths defined by the user and sort the files upon downloading them. However, the additional safety features present in the E2E clients do not exist in the raw clients.
Q: What also makes the E2E clients beginner friendly?
A: These clients have scanners that scan the latest available data on the data server as well as scanners for the local files. These scanners prevent users from attempting to download data at a rate faster than the data is updated on the server. This prevents users from 5-10 minute IP address bans from submitting too many requests in too short of a short-period of time. This also helps maximize performance by skipping unnecessary downloads and going straight to post-processing. This scanner also ensures the model runtime is consistent throughout the dataset for those who improperly time their data downloads. I may be a little biased but, I think WxData could be an excellent package for professors in academia to use in labs when introducing their students to working with weather data in Python given these beginner-friendly features.
When working with GRIB data:
- These clients decode all the GRIB variable keys into a plain-language format so users will not need to waste their time looking up what each GRIB key means.
- Resolve dataset build errors by creating a dataset for each key via
filter_by_keys={}and then combining the datasets at the end. - GRIB files are big. Users can also query by parameter and by lat/lon coordinates (-90 to 90 & -180 to 180) to subset the data prior to downloading to increase speed and decrease downloading/processing time.
When working with FEMS data:
- These clients combine the meta-data for a station in the Pandas.DataFrame that consists of the current weather and/or fuels data so users can plot current RAWS observations on a map.
For more information on WxData Clients, please visit:
E2E Clients: wxdata 路 PyPI
Raw Clients: wxdata 路 PyPI
Post-Processors
WxData has post-processing modules specific to each type of gridded dataset that is supported by the E2E clients. Post-processing is already included inside of the E2E clients, so if your intention is to stick to using the E2E clients, this section isn鈥檛 really for you.
If you are someone who is a more advanced user and already has a system in place to download and sort the data but would like to decode all GRIB keys into a plain language format, this section is for you. For example, wxdata.gfs_post_processing is a module that hosts different functions for decoding GFS data as well as resolving dataset build errors that are common with GRIB.
For more information on post-processors, please visit: wxdata 路 PyPI
Data Tools
WxData has several tools that assist users with querying, filling gaps in and transforming data:
-
wxdata.pixel_query()extracts a point (pixel) from a gridded dataset and return the data for that point to the user. -
wxdata.line_query()extracts a series of pixels from a gridded dataset in a straight line from point A to B. -
wxdata.linear_anti_aliasing()linearly interpolates n-amount of points between two data points. -
wxdata.shift_longitude()shifts longitude coordinates from a 0 to 360 into a -180 to 180 reference system. -
wxdata.cyclic_point()resolves the data gap along the 0 to 180 degree meridian in gridded datasets.
Python Automation Tools
WxData also has a tool that assists users in executing a Python automated workflow
wxdata.run_external_scripts()executes a sequence of Python scripts in the order listed by the user.
WxData New Features
Since the last post I made on here reflects WxData 1.4, this post will show the combination of new features in WxData 1.5 and WxData 1.6.
E2E Clients
wxdata.cfs_pressure()- An end-to-end client that downloads, pre-process and post-processes 6-hourly Climate Forecast System (CFS) data (for pressure-field variables).wxdata.cfs_flux()- An end-to-end client that downloads, pre-process and post-processes 6-hourly Climate Forecast System (CFS) data (for flux-field variables).wxdata.download_current_single_station_nexrad2_radar_data()- Downloads the latest NEXRAD II radar data for a user-specified radar site and returns a list of Py-ART radar objects corresponding to all the scans for a given radar.wxdata.download_current_multi_station_nexrad2_radar_data()- Downloads the latest NEXRAD II radar data for a user-specified list of radar sites and returns a list of Py-ART radar objects corresponding to all the scans for a given radar for all radars specified by the user.wxdata.get_single_raws_station_weather_observations()- Client that downloads and returns observational weather data from a single user-specified RAWS station for a user-specified period of time.wxdata.get_single_raws_station_fuels_observations()- Client that downloads and returns observational fuels data from a single user-specified RAWS station for a user-specified period of time.wxdata.get_multi_raws_station_weather_observations()- Client that downloads and returns observational weather data for a list of user-specified RAWS stations for a user-specified period of time.wxdata.get_multi_raws_station_fuels_observations()- Client that downloads and returns observational fuels data for a list of user-specified RAWS stations for a user-specified period of time.wxdata.get_current_multi_raws_station_weather_observations()- Client that downloads and returns current weather observations for a list of user-specified RAWS stations.wxdata.get_current_multi_raws_station_fuels_observations()- Client that downloads and returns current fuels observations for a list of user-specified RAWS stations.wxdata.get_current_all_raws_station_weather_observations()- Client that downloads and returns current weather observations for all RAWS stations in a given state.wxdata.get_current_all_raws_station_fuels_observations()- Client that downloads and returns current fuels observations for all RAWS stations in a given state.wxdata.get_single_raws_station_weather_forecast()- Client that downloads and returns the latest 7-Day weather forecast for a single user-specified RAWS station.wxdata.get_single_raws_station_nfdrs_forecast()- Client that downloads and returns the latest 7-Day NFDRS (fuels) forecast for a single user-specified RAWS station.wxdata.get_multi_raws_station_weather_forecast()- Client that downloads and returns the latest 7-Day weather forecast for a list of user-specified RAWS stations.wxdata.get_multi_raws_station_nfdrs_forecast()- Client that downloads and returns the latest 7-Day NFDRS (fuels) forecast for a list of user-specified RAWS stations.wxdata.get_single_raws_station_meta_data()- Client that downloads and returns the meta-data for a single user-specified RAWS station.wxdata.get_multi_raws_station_meta_data()- Client that downloads and returns the meta-data for a list of user-specified RAWS stations.
Raw Clients
-
wxdata.client.get_excel_data()- A new raw client to download and extract data from Excel files. -
wxdata.client.get_aws_open_data()- A raw data client that allows users to download any publicly available data (data not hidden behind an API key) from Amazon AWS servers.
Post-Processors
wxdata.cfs_post_processing- A new module consisting of post-processors for CFS Pressure and CFS Flux data.
