# Large-scale data processing benchmarks for Xarray-Beam
**URL:** https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489
**Category:** Uncategorized
**Created:** [May 18, 2021, 9:13pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489 "2021-05-18T21:13:10Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@shoyer](https://discourse.pangeo.io/u/shoyer)
#### Post date: [May 18, 2021, 9:13pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/1 "2021-05-18T21:13:10Z")
I released a new “Xarray-Beam” project on GitHub last week: [GitHub - google/xarray-beam: Distributed Xarray with Apache Beam](https://github.com/google/xarray-beam)
The idea is to facilitate a different model for large-scale distributed analytics in the Cloud, building on Apache Beam as an alternative to Dask. I’m still working on documenting it, but hopefully the README gives a reasonable overview of the idea.
One thing I’d love to include are a handful of end-to-end examples of large scale data-processing that run out of the box on Google Cloud Dataflow. These should capture the flavor of important data processing workflows for working with weather/climate data.
I’m currently thinking of two demos, based on Pangeo’s publicly available ERA5-surface dataset (17 data-variables adding up to 25 TB total):
1. Rechunking, from “stack of images” to “time-series” format. (Xarray-Beam leverages [Rechunker](https://rechunker.readthedocs.io/) internally to figure out the optimal chunking scheme.)
2. Calculating climatological averages over time, per hour of the day and per calendar month.
I like ERA5 surface because it’s relatively high-resolution (and thus makes pretty pictures) and is also quite relatable – most humans have some understanding of surface weather!
My goal is to show how Xarray-Beam could be useful for solving problems the Pangeo community cares about. So towards that end, I would appreciate feedback and suggestions. For example, are there alternative benchmarking tasks and/or datasets that I should be considering instead? I would be particularly interested in cases where we could compare performance to Dask or another distributed computing engine.
---
### Author:  [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)
#### Post date: [May 19, 2021, 3:42am UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/2 "2021-05-19T03:42:52Z")
There’s the Sentinel 2 public dataset on AWS, as another obvious use case. Or Landsat.
Google mentioned Apache Beam to me as a possibility - so I’d be interested in time/cost comparisons between that and other workflows.
---
### Author:  [@darothen](https://discourse.pangeo.io/u/darothen)
#### Post date: [May 31, 2021, 3:46pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/3 "2021-05-31T15:46:20Z")
@shoyer do you have any sense of pricing/cost for similar workflows that might be executed via a “traditional” dask cluster versus Beam? I think this might be a really useful set of metrics that could feed into your comparisons with dask or other distributed computing engines.
I’m on paternity leave for the next 2-3 weeks and would be happy to sprint on this with you if you have time.
---
### Author:  [@shoyer](https://discourse.pangeo.io/u/shoyer)
#### Post date: [June 3, 2021, 5:51pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/4 "2021-06-03T17:51:15Z")
@darothen I would love to get some comparisons to Dask for the same workload! I don’t have a clear answer for you on cost, but my expectation is that it should be in roughly the same ballpark, depending on lots of little details. I have two examples (climatology calculation and rechunking) on this ERA5 dataset worked out for Xarray-Beam that might be interesting to port to Dask: [xarray-beam/examples at main · google/xarray-beam · GitHub](https://github.com/google/xarray-beam/tree/main/examples)
My general impression is that the Dask scheduler is much more “clever” than Beam scheduler (or rather, the scheduler behind Beam runners like the Cloud Dataflow). Dask has a lower-level representation of workflows (individual tasks), whereas Beam keeps things as higher-level (e.g., GroupByKey and Map). This means Dask has more opportunities for clever automatic optimizations, but also more opportunities for things to go wrong.
---
### Author:  [@rabernat](https://discourse.pangeo.io/u/rabernat)
#### Post date: [June 7, 2022, 12:53pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/5 "2022-06-07T12:53:27Z")
There is growing interested in implementing a Dask Runner for Beam. This would make it a lot easier for people with existing Dask infrastructure to try Beam.
In order to kickstart the discussion of implementing a Dask Beam runner, I propose we meet during the week of June 13-17. I have created a When2Meet Poll here - [Dask Beam Runner Discussion - When2meet](https://www.when2meet.com/?15861604-jLnA4) . If you are interested in attending, please give your availability. Hope to see many people there! 🚀
---
### Author:  [@dcherian](https://discourse.pangeo.io/u/dcherian)
#### Post date: [June 7, 2022, 2:05pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/6 "2022-06-07T14:05:08Z")
I can’t make that week (conferences) but would like to be involved in future meetings
---
### Author:  [@rabernat](https://discourse.pangeo.io/u/rabernat)
#### Post date: [June 13, 2022, 3:52pm UTC](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489/7 "2022-06-13T15:52:10Z")
Thanks to all who replied! We have scheduled the call for **Wed June 15, 1:30 pm ET**. The zoom link is [Launch Meeting - Zoom](https://columbiauniversity.zoom.us/j/94977204481?pwd=TVFVTGVXY25sVnMxTHROeTloWlZ5dz09)
Looking forward to the discussion!
Deepak, we will take notes and share them.