New Working Group for Distributed Array Computing

We’re forming a new working group to explore alternative parallel computing frameworks to dask, such as apache-beam/cubed/ramba.

I propose we try a bi-weekly hour-long meeting, starting next week. If you’re interested can you please fill out this doodle poll for times that work for you.

See also:

1 Like

I don’t know if I’ll be able to make an hour every other week, but I’ll try. 30 minutes would be more doable.

Maybe an hour was too much - lets start by meeting at the best time from the poll and see what length we think is sufficient.

1 Like

The time selected is every other Monday at 1pm ET. The first meeting with be Monday, Sept. 19.

The working group has been added to the calendar and published on the Pangeo website: Meeting Schedule and Notes — Pangeo documentation

GitHub - pangeo-data/distributed-array-examples is a repository where we’re collecting examples of challenging distributed array computations.

1 Like

Ah sorry to have missed it! (the one time I don’t check notifications on a weekend!)

I added some brief comments in red to the notes.

1 Like

Good to keep us on our toes.

As a heads-up, the Dask folks are Coiled are collecting workloads for large scale benchmarking to help inform development. We’ve gotten significantly more data-driven in the last few months. I think that James Bourbeau is aware of the array examples repo Tom points to above. If anything else arises that would help to guide things in Dask-land please speak up.

Thanks all

3 Likes

Ugh…I was expecting a notification of the selected time the same way I got the initial email so sorry I missed the first one. I’ll be there for the next one. Thanks for posting the notes.

1 Like

Reminder that we have another meeting at 1pm EST TODAY (so in 45 minutes’ time)

Today we had a great intro to Ramba from Todd and Babu, and an overview of cubed from Tom White - thanks everyone!

On the 31st October we’re going to have a presentation on Arkouda from Scott Bachman!

Here is the link I promised in the meeting about standardizing a partitioning API.