While we look forward to next month’s ESIP summer meeting, we have this month’s knowledge sharing session to tide us over. This month we’re pleased to have Matt Rocklin, CEO of Coiled.io and early member of the Pangeo team, to talk about lessons learned scaling Dask clusters in the cloud.
Finally, in the last 15 minutes of the knowledge sharing session we’d like to have an open conversation about when it makes sense to allow presentations on behalf of a specific company and their solution.
Read the abstract and meeting agenda below.
Meeting Logistics!
Topic: Dask and Coiled: Python on Cloud Infrastructure
When: Monday June 26th, 10:00-11:00 am PT / 1:00-2:00 pm ET / 7pm CEST
Where: Find joining information on ESIP Community Calendar
Abstract:
In this talk we’ll discuss Dask, an open source library for parallel computing in Python, and Coiled, a service to deploy Dask efficiently on cloud infrastructure.
Dask makes it easy to parallelize Python code, either your own custom code, or through integrations with common libraries (pandas, numpy, xgboost, xarray, etc.). While Dask is used in a broad set of applications, in this talk we’ll mostly focus on applications in the Pangeo ecosystem, a set of tools (Xarray, Dask, Zarr, and others) that make it easy to perform earth science applications at scale.
Coiled makes it easy to manage cloud infrastructure. Coiled came out of many experiments running Dask clusters on cloud infrastructure. We found that users had trouble managing software package versions, cloud credentials, managing idle clusters, managing users, and predicting and constraining costs. The Coiled platform was developed as a centralized service to deploy Dask clusters in a way that addressed these issues.
We hope to show that, with these two technologies, we’ve found a good way to enable the vast majority of Python users to run their analyses at scale.
Agenda:
- 5-10 minutes - Announcements: Working sessions, ESIP Summer Meeting Announcements, Open Call for other announcements
- 20 minutes - Presentation
- 15 minutes - Q@A
- 15 minutes - Open Conversation: What sort of presentations would the cluster like to invite? Looking forward to seeing you there!
- ESIP Cloud Computing Cluster Committee
More about ESIP: https://www.esipfed.org/
More about the Cloud Computing Cluster: Cloud Computing - Earth Science Information Partners (ESIP)
The ESIP summer meeting: ESIP Meetings - Earth Science Data