# Serverless Pangeo

**URL:** <https://discourse.pangeo.io/t/serverless-pangeo/1224>\
**Category:** Cloud\
**Created:** [February 10, 2021, 3:46pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224 "2021-02-10T15:46:26Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![rsignell](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rsignell/32/447_2.png) [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Post date:** [February 10, 2021, 3:46pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/1 "2021-02-10T15:46:26Z")

</div>

We’ve discussed serverless compute here and there over the years, but I thought it might be nice to start a dedicated topic. Here’s a timeline of some relevant activities I know of:

November 2017: AWS Workshop on [Massive parallel data processing using pywren and AWS Lambda](https://github.com/aws-samples/pywren-workshops#massive-parallel-data-processing-using-pywren-and-aws-lambda), featuring a [Landsat8 NDVI processing example notebook](https://github.com/aws-samples/pywren-workshops/blob/master/Lab-4-Landsat-NDVI/Landsat_NDVI_Timeseries.ipynb)! This used the `pywren.default_executor()`. (It requires python 2.7 and [pywren](http://pywren.io) is been supplanted by [lithops](https://github.com/lithops-cloud/lithops/blob/master/README.md)). (Thanks to @RichardScottOZ for the discovering this)!

April 2018: Post by @jacobtomlinson in on [Exploring Dask and Distributed on AWS Lambda](https://medium.com/informatics-lab/exploring-dask-and-distributed-on-aws-lambda-55d81d9641d)  
where he discusses the 250MB memory limit for Lambda and other problems. He concludes:

> … the future of distributed computation will make use of highly abstracted function execution services like Lambda. However with the tools we have today there are too many issues to get this working without having major limitations.

May 2020: @tomwhite creates [Dask Executor Scheduler](https://github.com/tomwhite/dask-executor-scheduler), a Dask scheduler that uses `concurrent.futures.Executor` to run tasks. He then used this for a [zarr rechunking workflow](https://github.com/tomwhite/dask-executor-scheduler/blob/master/examples/rechunk_cloud_storage_cloud_compute.py) using [pywren and GCP](https://github.com/tomwhite/dask-executor-scheduler#configuring-pywren-for-google-cloud) (and also using [rechunker](https://rechunker.readthedocs.io/en/latest/)).

November 2020: The GitHub project pywren-ibm-cloud becomes [Lithops](https://github.com/lithops-cloud/lithops/blob/master/README.md)

January 2021: Blog post by IBM discussing integration of [Lithops](https://github.com/lithops-cloud/lithops/blob/master/README.md) with IBM Cloud: [Using Serverless to Run Your Python Code on 1000 Cores by Changing Two Lines of Code](https://www.ibm.com/cloud/blog/using-serverless-to-run-your-python-code-on-1000-cores-by-changing-two-lines-of-code).

It would be cool to have an updated workshop showing how this stuff can work for Pangeo workflows. Perhaps it would just take updating the old AWS 2017 workshop to use Python 3 and lithops, using the Dask Executor Scheduler?

I look forward to seeing what people have to say who actually understand this stuff. 🙂

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [February 10, 2021, 8:30pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/2 "2021-02-10T20:30:08Z")

</div>

There are things like [github.com/developmentseed/titiler](http://github.com/developmentseed/titiler) - for example [COG Talk — Part 2: Mosaics. This blog is the second in a series… | by Vincent Sarago | Development Seed | Medium](https://medium.com/devseed/cog-talk-part-2-mosaics-bbbf474e66df)

---

<div class="post-metadata">

**Author:** ![Gil\_Vernik](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/gil_vernik/32/799_2.png) [@Gil\_Vernik](https://discourse.pangeo.io/u/Gil_Vernik)\
**Post date:** [February 12, 2021, 6:13am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/3 "2021-02-12T06:13:32Z")

</div>

It always great to learn more use cases where Lithops can fit. I would love to hear more about your use cases and see if I can assist to leverage Lithops for your workloads. That would be very interesting to me.

One of our recent projects is with EMBL (European Molecular Biology Laboratory ), where Lithops used ([GitHub - metaspace2020/Lithops-METASPACE: Lithops-based Serverless implementation of the METASPACE spatial metabolomics annotation pipeline](https://github.com/metaspace2020/Lithops-METASPACE)) to deploy their workloads against serverless backend. Prior using Lithops, their main challenge was that amount of compute resources are only known in runtime. So it was almost impossible to setup a compute cluster and then run a workload, since in runtime it could be turned out that cluster too small or too large. As Lithops truly serverless, it can request needed resources dynamically in runtime, as much as needed so their problem was resolved.

Next week i am giving a talk at CNCF about Lithops ([CNCF Live Webinar: Toward Hybrid Cloud Serverless Transparency with Lithops Framework | Cloud Native Computing Foundation](https://www.cncf.io/webinars/cncf-live-webinar-toward-hybrid-cloud-serverless-transparency-with-lithops-framework/)) and I give plenty of demos with real usage examples. If you can’t attend, the talks are recorded and will appear on Youtube at some point. This might give you more background on the project and how it can be used.

---

<div class="post-metadata">

**Author:** ![nbren12](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/nbren12/32/386_2.png) [@nbren12](https://discourse.pangeo.io/u/nbren12)\
**Post date:** [February 17, 2021, 6:24am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/4 "2021-02-17T06:24:44Z")

</div>

I have some experience mixing xarray + dask with Google Cloud Dataflow for coarse-graining atmospheric model output. It was amazing seeing 100s of workers spin-up, but it requires some setup cost. Because the workers take so long to spin-up, it was not possible to iteratively develop these pipelines quickly in the typical pangeo style. It was more of a serious software engineering task requiring unit tests/clean-code etc or an extremely careful eye. I am not sure if this is intrinsic to all “serverless” platforms or specific to the APIs provided by apache beam.

For some reason, dataflow reminds of elaborate mail sorting machines like this: [High Speed Letter Sorter - The NPI Maxim for High Speed Letter Sorting - YouTube](https://www.youtube.com/watch?v=wG8ULky-NwA). It can process a high volume of items, but I don’t need one at my house

A good path forward might be writing general purpose transformation scripts backed by dataflow. For example, the “rechunker” tool recently developed by @rabernat et. al. also works with Apache Beam thanks to @shoyer : [rechunker/beam.py at master · pangeo-data/rechunker · GitHub](https://github.com/pangeo-data/rechunker/blob/master/rechunker/executors/beam.py).

---

<div class="post-metadata">

**Author:** ![rsignell](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rsignell/32/447_2.png) [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Post date:** [February 19, 2021, 3:03am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/5 "2021-02-19T03:03:33Z")

</div>

@Gil_Vernik, thanks so much for the update on Lithops. I wasn’t able to attend your webinar earlier this week but would be very interested in watching the recording when it’s available. Should I just keep searching every few days on [CNCF [Cloud Native Computing Foundation] - YouTube](https://www.youtube.com/c/cloudnativefdn/search?query=lithops) ?

---

<div class="post-metadata">

**Author:** ![jeffdlb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jeffdlb/32/835_2.png) [@jeffdlb](https://discourse.pangeo.io/u/jeffdlb)\
**Post date:** [March 3, 2021, 6:56pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/6 "2021-03-03T18:56:59Z")

</div>

The 250MB limit for Lambda is no longer applicable. Current limit is 10GB as of Dec 2020 (up from 3GB) and 6 vCPU cores. Surely this is enough for most tasks? Serverless seems ideally suited for some of these Notebook-style workflows that sit mostly idle until someone clicks a cell.

---

<div class="post-metadata">

**Author:** ![jeffdlb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jeffdlb/32/835_2.png) [@jeffdlb](https://discourse.pangeo.io/u/jeffdlb)\
**Post date:** [March 3, 2021, 6:57pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/7 "2021-03-03T18:57:35Z")

</div>

PS. Link: [AWS Lambda now supports up to 10 GB of memory and 6 vCPU cores for Lambda Functions](https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-supports-10gb-memory-6-vcpu-cores-lambda-functions/)

---

<div class="post-metadata">

**Author:** ![Gil\_Vernik](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/gil_vernik/32/799_2.png) [@Gil\_Vernik](https://discourse.pangeo.io/u/Gil_Vernik)\
**Post date:** [March 30, 2021, 5:04am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/8 "2021-03-30T05:04:34Z")

</div>

here is the link [https://youtu.be/-uS-wi8CxBo](https://youtu.be/-uS-wi8CxBo)

by the way, there are other cloud provides who also offer serverless without “limits”… more run time, more memory, etc.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [March 30, 2021, 2:26pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/9 "2021-03-30T14:26:56Z")

</div>

This is another cool new “serverless” thing:

> **[Introducing Amazon S3 Object Lambda – Use Your Code to Process Data as It Is...](https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/)**
>
> When you store data in Amazon Simple Storage Service (S3), you can easily share it for use by multiple applications. However, each application has its own requirements and may need a different view of the data. For example, a dataset created by an...

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [April 14, 2021, 4:53am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/10 "2021-04-14T04:53:33Z")

</div>

I just talked to Google about Apache Beam - there is apparently Dataflow geobeam.

[GoogleCloudPlatform/dataflow-geobeam (github.com)](https://github.com/GoogleCloudPlatform/dataflow-geobeam)

Which is all I know until I do some reading - but sounds interesting at least.

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [April 14, 2021, 4:58am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/11 "2021-04-14T04:58:21Z")

</div>

[https://medium.com/weareservian/gcp-dataflow-with-python-for-satellite-image-analysis-5b9b8a81ff74](https://medium.com/weareservian/gcp-dataflow-with-python-for-satellite-image-analysis-5b9b8a81ff74)

> **[Processing Landsat satellite images with GPUs  |  Cloud Dataflow](https://cloud.google.com/dataflow/docs/samples/satellite-images-gpus)**

---

<div class="post-metadata">

**Author:** ![Gil\_Vernik](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/gil_vernik/32/799_2.png) [@Gil\_Vernik](https://discourse.pangeo.io/u/Gil_Vernik)\
**Post date:** [May 3, 2021, 5:30am UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/12 "2021-05-03T05:30:34Z")

</div>

Can you share please your use case? out of curiosity, what is the problem / software you try to benefit from serverless? I am looking for nice use cases where serverless can fit, so if you have some ideas and it’s open source, etc. , can you share it here please?

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [October 13, 2021, 8:43pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/13 "2021-10-13T20:43:07Z")

</div>

50000 runs of a model or an inversion, return results and metadata for each to storage?

---

<div class="post-metadata">

**Author:** ![ShaneMill](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/shanemill/32/1778_2.png) [@ShaneMill](https://discourse.pangeo.io/u/ShaneMill)\
**Post date:** [August 11, 2022, 12:57pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/14 "2022-08-11T12:57:28Z")

</div>

Hi all,

In NOAA, we are prototyping an Serverless environment from which API Gateway invokes Lambda functions to access Zarr located in S3. We have prototyped a centralized Dask Cluster using ECS Fargate but there are some challenges with cost/performance when using Fargate. For example, it takes serveral minutes sometimes for a task to spin up. We are interested to see if Lithops could fulfill the task of loading the chunks of data from Zarr in S3 to an xarray dataset located within a Lambda function.

Any feedback on this possibility or any other ideas is greatly appreciated!

-Shane Mill

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [July 13, 2023, 5:31pm UTC](https://discourse.pangeo.io/t/serverless-pangeo/1224/15 "2023-07-13T17:31:30Z")

</div>

@rsignell just showed me this thread. @tomwhite has written [Cubed](https://github.com/tomwhite/cubed), which generalizes Rechunker to implement arbitrary distributed array operations in a serverless manner. You can read more in our [recent blog post](https://xarray.dev/blog/cubed-xarray).

@Gil_Vernik Lithops is one of the main serverless Executors which Cubed supports, and from what Rich told me it sounds like this is something you might be interested in! I would be happy to chat about this anytime.