# New Working Group for Distributed Array Computing

**URL:** <https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734>\
**Category:** News & Announcements\
**Created:** [September 14, 2022, 7:21pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734 "2022-09-14T19:21:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [September 14, 2022, 7:21pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/1 "2022-09-14T19:21:05Z")

</div>

We’re forming a new working group to explore **alternative parallel computing frameworks to dask** , such as apache-beam/cubed/ramba.

I propose we try a **bi-weekly hour-long meeting, starting next week**. If you’re interested can you please [fill out this doodle poll](https://doodle.com/meeting/participate/id/bmQ0423b) for times that work for you.

See also:

- [Large-scale data processing benchmarks for Xarray-Beam](https://discourse.pangeo.io/t/large-scale-data-processing-benchmarks-for-xarray-beam/1489)
- [Cubed: Fixed-memory serverless distributed N-dimensional array processing](https://discourse.pangeo.io/t/cubed-fixed-memory-serverless-distributed-n-dimensional-array-processing/2484)
- [Applications and testers wanted! New Ramba distributed arrays through xarray](https://discourse.pangeo.io/t/applications-and-testers-wanted-new-ramba-distributed-arrays-through-xarray/1956)

---

<div class="post-metadata">

**Author:** ![shoyer](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/shoyer/32/1005_2.png) [@shoyer](https://discourse.pangeo.io/u/shoyer)\
**Post date:** [September 14, 2022, 10:43pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/2 "2022-09-14T22:43:44Z")

</div>

I don’t know if I’ll be able to make an hour every other week, but I’ll try. 30 minutes would be more doable.

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [September 15, 2022, 2:46pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/3 "2022-09-15T14:46:18Z")

</div>

Maybe an hour was too much - lets start by meeting at the best time from the poll and see what length we think is sufficient.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 16, 2022, 1:25pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/4 "2022-09-16T13:25:00Z")

</div>

The time selected is **every other Monday at 1pm ET**. The first meeting with be **Monday, Sept. 19**.

The working group has been added to the calendar and published on the Pangeo website: [https://pangeo.io/meeting-notes.html](https://pangeo.io/meeting-notes.html)

- [Calendar Event](https://calendar.google.com/calendar/u/0/r/week/2022/9/19?eid=MzBiYWI3OHZnNGxkZmhtajgzMGVxaGF0NWZfMjAyMjA5MTlUMTcwMDAwWiB1Y2FyLmVkdV9jMjNsbjQwMTRraHMzZjY1bzkzdnF2NWtxY0Bn)
- ~~[OLD Zoom Link](https://columbiauniversity.zoom.us/j/99521529645?pwd=YzhIbDlhZUE1dUNvR2R4SzV5VVBhQT09)~~ [NEW Pangeo Zoom Link](https://numfocus-org.zoom.us/j/83352967369?pwd=AidxVNb0Rlc6EjnWqG7MinrAZsUgQI.1)
- [Notes](https://docs.google.com/document/d/1_FkLZ3NjXlzlc7p4mr1GtWuPGKN3-E991dQEdPv6bNc/edit)

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [September 19, 2022, 5:33pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/5 "2022-09-19T17:33:05Z")

</div>

- [Notes](https://docs.google.com/document/d/1_FkLZ3NjXlzlc7p4mr1GtWuPGKN3-E991dQEdPv6bNc/edit?usp=sharing)

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [September 19, 2022, 6:02pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/6 "2022-09-19T18:02:11Z")

</div>

[GitHub - pangeo-data/distributed-array-examples](https://github.com/pangeo-data/distributed-array-examples) is a repository where we’re collecting examples of challenging distributed array computations.

---

<div class="post-metadata">

**Author:** ![dcherian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dcherian/32/2235_2.png) [@dcherian](https://discourse.pangeo.io/u/dcherian)\
**Post date:** [September 19, 2022, 10:01pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/7 "2022-09-19T22:01:01Z")

</div>

Ah sorry to have missed it! (the one time I don’t check notifications on a weekend!)

I added some brief comments in red to the notes.

---

<div class="post-metadata">

**Author:** ![mrocklin](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/mrocklin/32/2280_2.png) [@mrocklin](https://discourse.pangeo.io/u/mrocklin)\
**Post date:** [September 23, 2022, 9:33pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/8 "2022-09-23T21:33:16Z")

</div>

Good to keep us on our toes.

As a heads-up, the Dask folks are Coiled are collecting workloads for large scale benchmarking to help inform development. We’ve gotten significantly more data-driven in the last few months. I _think_ that James Bourbeau is aware of the array examples repo Tom points to above. If anything else arises that would help to guide things in Dask-land please speak up.

Thanks all

---

<div class="post-metadata">

**Author:** ![Todd\_Anderson](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/todd_anderson/32/1284_2.png) [@Todd\_Anderson](https://discourse.pangeo.io/u/Todd_Anderson)\
**Post date:** [September 28, 2022, 12:13am UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/9 "2022-09-28T00:13:55Z")

</div>

Ugh…I was expecting a notification of the selected time the same way I got the initial email so sorry I missed the first one. I’ll be there for the next one. Thanks for posting the notes.

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [October 3, 2022, 4:13pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/10 "2022-10-03T16:13:48Z")

</div>

Reminder that we have another meeting at 1pm EST TODAY (so in 45 minutes’ time)

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [October 3, 2022, 8:00pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/11 "2022-10-03T20:00:03Z")

</div>

Today we had a great intro to Ramba from Todd and Babu, and an overview of cubed from Tom White - thanks everyone!

On the 31st October we’re going to have a presentation on [Arkouda](https://github.com/Bears-R-Us/arkouda) from Scott Bachman!

---

<div class="post-metadata">

**Author:** ![Todd\_Anderson](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/todd_anderson/32/1284_2.png) [@Todd\_Anderson](https://discourse.pangeo.io/u/Todd_Anderson)\
**Post date:** [October 4, 2022, 12:42am UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/12 "2022-10-04T00:42:11Z")

</div>

Here is the link I promised in the meeting about standardizing a partitioning API.

> <https://github.com/data-apis/consortium-feedback/issues/7>
>
> In order for an ndarray/dataframe system to interact with a variety of framework…s in a distributed environment (such as clusters of workstations) a stable description of the distribution characteristics is needed.
> 
> The \[\\\_\\\_partitioned\\\_\\\_ protocol\](https://github.com/IntelPython/DPPY-Spec/blob/draft/partitioned/Partitioned.md) accomplishes this by defining a structure which provides the necessary meta-information. Implementations show that it allows data exchange on between different distributed frameworks without unnecessary data transmission or even copies.
> 
> The structure defines how the data is partitioned, where it is located and provides a function to access the local data. It does not define or provide any means of communication, messaging and/or resource management. It merely describes the current distribution state.
> 
> In a way, this is similar to the meta-data which \`dlpack\` provides for exchanging data within a single node (including the local GPU) - but it does it for data which lives on more than on process/node. It complements mechanism for intra-node exchange, such as \`dlpack\`, \`\_\_array\_interface\_\_\` and alike.
> 
> The current lack of such a structure typically leads to one of the following scenarios when connecting different frameworks:
> \*	a data consumer implements a dedicated import functionality for every distributed data container it sees important enough. As an example, xgboost\_ray implements a variety of data\_sources.
> This \[PR\](https://github.com/ray-project/xgboost\_ray/pull/153) lets xgboost\_ray automatically work with any new container that supports \`\_\_partitioned\_\_\` (the extra code for modin DF and Ray/MLDataSet are no longer needed once they support it, too)
> \*	the user needs to explicitly deal with the distributed nature of the data. This either leads to unnecessary data transfer and/or developers need to understand internals of the data-container. In the latter case they get exposed to explicit parallelism/distribution while often the original intend of the producer was to hide exactly that.
> The implementation \[here\](https://github.com/fschlimb/daal4py/tree/feature/partitioned\_interface) avoids that by entirely hiding the distribution features but still allowing zero-copy data exchange between \`\_\_partitioned\_\_\` (exemplified by modin (\[PR\](https://github.com/modin-project/modin/pull/3452)) and HeAT (\[PR\](https://github.com/helmholtz-analytics/heat/pull/788))) and scikit-learn-intelex/daal4py.
> \*	frameworks like dask and ray wrap established APIs (such as pytorch, xgboost etc) and ask developers to switch to the new framework/API and to adopt their programing model.
> 
> A structure like \`\_\_partitioned\_\_\` enables distributed data exchange between various frameworks which can be seamless to users while avoiding unnecessary data transfer.
> 
> The \[\\\_\\\_partitioned\\\_\\\_ protocol\](https://github.com/IntelPython/DPPY-Spec/blob/draft/partitioned/Partitioned.md) is an initial proposal, any feedback from this consortium will be highly appreciated. We would like to build a neutral, open governance that continues to oversee the evolution of the spec. For example, if some version of it receives positive response from the community we would be more than happy to donate the \\\_\\\_partitioned\\\_\\\_ protocol to the data-api consortium or host it in a clean github org.

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [October 17, 2022, 2:51pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/13 "2022-10-17T14:51:41Z")

</div>

Reminder that this will happen again today at 1pm EST!

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [November 14, 2022, 3:22pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/14 "2022-11-14T15:22:55Z")

</div>

We’re on again today at 1pm EST, when we will have a talk by Frank Schlimbach on the partitioning API!

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [November 28, 2022, 4:19pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/15 "2022-11-28T16:19:42Z")

</div>

Today (at 1pm EST) we have Deepak talking about his dask algorithms for groupby problems using [his package flox](https://github.com/xarray-contrib/flox)!

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [December 12, 2022, 3:20pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/16 "2022-12-12T15:20:11Z")

</div>

We don’t have a speaker today, so unless someone has something they would like to discuss with the group or present last minute I think we should cancel today and pick it back up after Christmas! That would make the next meeting 9th January.

---

<div class="post-metadata">

**Author:** ![dcherian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dcherian/32/2235_2.png) [@dcherian](https://discourse.pangeo.io/u/dcherian)\
**Post date:** [December 12, 2022, 5:31pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/17 "2022-12-12T17:31:14Z")

</div>

Sounds good to me.

Perhaps a good topic for next time would be “What’s needed for a full end-to-end workflow with cubed/Ramba” organized around your xarray+cubed PR?

---

<div class="post-metadata">

**Author:** ![Todd\_Anderson](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/todd_anderson/32/1284_2.png) [@Todd\_Anderson](https://discourse.pangeo.io/u/Todd_Anderson)\
**Post date:** [January 17, 2023, 5:05pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/18 "2023-01-17T17:05:50Z")

</div>

What are the plans for this working group in the new year? [@everyone](https://discourse.pangeo.io/groups/everyone)

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [January 17, 2023, 10:28pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/19 "2023-01-17T22:28:00Z")

</div>

Hey @Todd_Anderson ! Thanks for the reminder. I got completely distracted by conferences / workshops the last few weeks.

> “What’s needed for a full end-to-end workflow with cubed/Ramba” organized around your xarray+cubed PR?

I could talk about this if people are interested? Though it’s not finished, but would be useful information and should start some useful discussion as to API standards etc.

---

<div class="post-metadata">

**Author:** ![dcherian](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/dcherian/32/2235_2.png) [@dcherian](https://discourse.pangeo.io/u/dcherian)\
**Post date:** [January 18, 2023, 10:02pm UTC](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734/20 "2023-01-18T22:02:32Z")

</div>

> [@TomNicholas](#):
>
> “What’s needed for a full end-to-end workflow with cubed/Ramba” organized around your xarray+cubed PR?

IMO this should be our goal (and a great SciPy talk 😉 )

[Next page](https://discourse.pangeo.io/t/new-working-group-for-distributed-array-computing/2734.md?page=2)