Feedback wanted: recipe-based tool for building aligned radar/satellite/model/obs datasets (CONUS)

Data
1

Hi all,

I’m designing an open-source Python tool for building analysis-ready datasets that combine US radar, satellite, NWP, surface observations, and severe-weather reports, and I’d really value this community’s input before I commit to the design.

The problem

Assembling something like “MRMS + GOES ABI + GLM + HRRR + ASOS on one grid at 10-minute steps” usually means weeks of one-off code: GRIB index parsing, grid-relative wind rotation, accumulation buckets that reset every few hours, MRMS missing-value codes, projection mismatches, and deciding how to align data that were never meant to be combined. Datasets like SEVIR and GridRad-Severe exist because someone did this by hand, but they’re fixed snapshots. If you want a different period, domain, grid, or extra variables (e.g., HRRR environment fields alongside radar), you start over.

The idea

You write a short recipe; the tool plans, fetches, aligns, QCs, and writes the dataset:

sources:
  mrms: [reflectivity_composite, mesh, qpe_1h]
  goes_abi: [brightness_temperature@C13]
  glm: [flash_extent_density]
  hrrr: [mlcape, shear_0_6km, wind@10m]
  asos: [air_temperature@2m, wind_gust]
  spc_reports: [hail, wind, tornado]
domain: {bbox: [-104, 30, -94, 40]}
time: {start: 2024-05-06T18Z, end: 2024-05-07T03Z, step: 10min}
grid: hrrr
policy: as_of
output: plains.icechunk

plan gives a dry run (objects, bytes, gaps); build writes Zarr/Icechunk for grids, Parquet for stations, and GeoParquet for events.

Design choices I’d most like opinions on

  • Alignment based on what each variable means. Accumulations are re-aggregated over target windows, never interpolated. Hazard fields like MESH take window maxima, and coarsening uses max-pooling rather than averaging. Point events (GLM flashes, storm reports) are binned. Regridding method is chosen per variable type (conservative / bilinear / mode / max).
  • As-of selection. For each target time, inputs only use the latest model cycle and observations that would actually have been available then, based on per-source latency, to prevent leakage in ML training data. Targets can still look ahead.
  • Lockfiles. Resolving a recipe pins every upstream object (URI, byte range, ETag) plus SDK, QC, and regridding-weight versions, so a dataset can be rebuilt bit-for-bit later even if the upstream archives change.
  • QC as flags, not deletion, with coverage masks that distinguish “no echo” from “no radar.”

How it relates to existing tools

I plan to build on Herbie, satpy, xESMF, and the IEM services rather than replace them. Anemoi-datasets, Earth2Studio, earthkit, WxData, and dynamical.org each cover parts of this well: recipe-built training datasets, multi-source fetching, per-source clients, and ready-made cloud-native stores. The gap I’m aiming at is user-defined alignment across radar, satellite, model, station, and event data at convective scales, with as-of correctness and reproducible builds. If something already does this and I’ve missed it, I’d genuinely like to know before building it.

Questions

  1. If you’ve built a dataset like this, roughly how long did it take, and what was the most painful part?
  2. Which alignment decisions should a tool make automatically, and which should always be explicit user choices?
  3. Would a recipe + lockfile workflow fit how you work, or would you want something different (e.g., pure Python API, lazy xarray access first)?
  4. Anything in this design that looks wrong, naive, or already solved?

Thanks for reading. Happy to share more detail on any part.

Yuchen Wei

1 Like