Xue: A binary format for animating weather forecasts in the browser

Visualization
1

I’ve been working on weather/climate data visualization for a few years.

GRIB/NetCDF are good file formats for data distribution. Zarr is good for large-scale datasets, especially for machine learning cases. But none of these can provide a smooth experience of weather data animation in the browser.

The traditional solution is to use XYZ map tiles and update them as the time frame changes, which introduces a significant image downloading and rendering cost, even with grayscale tiles. So I designed another solution. Data animation is like video, so by borrowing the design ideas and infrastructure of the video industry, a weather data animation problem reduces to video stream distribution and rendering. In practice that’s a Rust WASM decoder for the frames and a WebGL2 layer that does the inverse Mercator projection and palette lookup on the GPU; 10 m wind goes through a GPU particle layer.

The size difference is the main payoff. For one GFS run, the same two variables come to ~3,206 MB as precolored tile pyramids (zoom 0–4), against ~138 MB of source GRIB2 — and ~65 MB as .xue.

Now NOAA’s GFS 0.25° / GFS surface flux / HRRR and ECMWF IFS 0.25° have been deployed, running on GitHub Actions and uploading to Cloudflare R2 (no CDN cost).

Playback streams the bundle over HTTP range requests (206) as it goes, so a stable connection matters more than a fast one.

The format is specified in docs/format.md, and I’d be curious how this compares with how people here approach the same problem.

Precipitation rate forecast map · GFS 240H · Xue
Precipitation rate forecast map · GFS 240H · Xue1920×1154 731 KB

Note: the container is modeled on Zarr (chunked, one zstd frame per chunk, range-addressable), and v2 corresponds to a sharded Zarr v3 uint8 array with a temporal delta codec. The difference is layout: analysis-oriented Zarr chunking (long time chunks, whole-level spatial chunks, float32) makes browser animation costly. The next step is a Zarr layout convention aimed at direct visualization: small spatial tiles × a few consecutive steps per chunk, uint8 codes, codebook and GRIB2 identity in attrs. In that layout the frontend reads the store directly, and the single-file container is one packaging of the same bytes.

2

Cool animation!

So if I understand this correctly your new binary format is not necessary for good performance, all that’s necessary is choosing visualization-friendly Zarr chunk shapes, and doing browser-side decoding with WASM?

3

Yes.

The gains come from two things: uint8 quantization, which cuts the data 4× before compression and uploads directly as a GPU texture, and a chunk layout chosen for playback (a few consecutive steps × a small spatial tile, one shard per time group).

Both can be expressed as a Zarr v3 store with standard codecs. The temporal delta codec adds 8–13% on temperature and wind and costs 15% on precipitation, so it is optional.

For context on how it got here: the first version packed per-frame raster tiles into PMTiles, which gave the 3.2 GB figure in the post. The binary format started as a replacement for that, as whole quantized planes with one zstd frame each. The later revisions (spatial tiles, chunk-local residuals, a fixed physical order that keeps a time group contiguous) followed Zarr’s chunk and shard model. I plan to write the same bundles out as Zarr and post the comparison here.

I plan to write the same bundles out as Zarr the later version.

1 Like