Producing *a lot* of synoptic weather maps

Visualization
1

This post is adapted from a similar question I sent to PyAOS.

My dept is in the process of rewriting the webpage we use for teaching weather forecasting. This platform provides real-time synoptic maps of operational forecasts (for example with GFS).

These maps are static pngs generated by a mix of python + GRADS scripts. We produce about 80000 pngs per day. One of the questions that came about is whether we can (or should) move away from GRADS and go pure python (e.g. cartopy + matplotlib, or even something more interactive like bokeh?). Our concern is of course performance: without having looked into it a lot, my experience is that cartopy is slow, and saving pngs with matplotlib is also not the fastest.

I’d like to add that we want the platform to run on a standard web server, i.e. “brute force” solutions such as moving to python and then having to use massive parallel jobs to replace the speed of GRADS isn’t really an option.

I’d say that for us the priorities for these maps would be (in order): (1) performance (the plots need to be ready in time for the forecasters), (2) modern look and ease of reading (GRADS is ok, but…), (3) interactive exploration of map values (we have built in “some” interactivity by parsing the map color under the mouse pointer, but this is a bit hacky)

I’m sure that many institutions have similar websites around here.

I would be glad to hear your experience with such a use case: do you know of other tools that can produce such maps? Would you recommend cartopy, and do you know of hacks to make it more efficient? Any other recommendations or tool tips?

Thanks in advance!

Fabien

1 Like
2

Does someone actually look at every png?

Why do have to pre-render all the images? Have you considered a dynamic web app that renders the desired visualizations on the fly?

1 Like
3

Thanks!

No. The usual procedure is to click through the lead times in the forecasts, for the region and variable of interest. Not all maps are looked at every day, by far.

app

Yes we did consider this as well. In the past this has been discarded as too risky / too slow (the platform is around since years/decades), but since we are planning an overhaul this is definitely on the table yes.

4

So why not an interactive geo/holoviews / panel app? Given your expertise with panel, it seems like an obvious solution? Did you rule that out?

1 Like
5

Sounds like panel/holoviews is the right choice. but if you really want pngs see: Using Dask to parallelize plotting - #10 by dcherian

2 Likes
6

Thanks both for your answers!

Sounds like panel/holoviews is the right choice

It’s a long story, but I don’t think it’ll do - I will explore the possibility for some of the graphs, but on second thought I really don’t think we want to set-up a panel app for ~10s (100s?) of concurrent users all asking for map graphs requiring to open files, read, convert and plot the data. If we go for bokeh, it will have to be for static html, i.e. no python in the background.

I might be too pessimistic though - I’ll bring all your ideas to the next dept meeting!