We’ve discussed serverless compute here and there over the years, but I thought it might be nice to start a dedicated topic. Here’s a timeline of some relevant activities I know of:
… the future of distributed computation will make use of highly abstracted function execution services like Lambda. However with the tools we have today there are too many issues to get this working without having major limitations.
It would be cool to have an updated workshop showing how this stuff can work for Pangeo workflows. Perhaps it would just take updating the old AWS 2017 workshop to use Python 3 and lithops, using the Dask Executor Scheduler?
I look forward to seeing what people have to say who actually understand this stuff.
It always great to learn more use cases where Lithops can fit. I would love to hear more about your use cases and see if I can assist to leverage Lithops for your workloads. That would be very interesting to me.
One of our recent projects is with EMBL (European Molecular Biology Laboratory ), where Lithops used (GitHub - metaspace2020/Lithops-METASPACE: Lithops-based Serverless implementation of the METASPACE spatial metabolomics annotation pipeline) to deploy their workloads against serverless backend. Prior using Lithops, their main challenge was that amount of compute resources are only known in runtime. So it was almost impossible to setup a compute cluster and then run a workload, since in runtime it could be turned out that cluster too small or too large. As Lithops truly serverless, it can request needed resources dynamically in runtime, as much as needed so their problem was resolved.
I have some experience mixing xarray + dask with Google Cloud Dataflow for coarse-graining atmospheric model output. It was amazing seeing 100s of workers spin-up, but it requires some setup cost. Because the workers take so long to spin-up, it was not possible to iteratively develop these pipelines quickly in the typical pangeo style. It was more of a serious software engineering task requiring unit tests/clean-code etc or an extremely careful eye. I am not sure if this is intrinsic to all “serverless” platforms or specific to the APIs provided by apache beam.
@Gil_Vernik, thanks so much for the update on Lithops. I wasn’t able to attend your webinar earlier this week but would be very interested in watching the recording when it’s available. Should I just keep searching every few days on CNCF [Cloud Native Computing Foundation] - YouTube ?
The 250MB limit for Lambda is no longer applicable. Current limit is 10GB as of Dec 2020 (up from 3GB) and 6 vCPU cores. Surely this is enough for most tasks? Serverless seems ideally suited for some of these Notebook-style workflows that sit mostly idle until someone clicks a cell.
Can you share please your use case? out of curiosity, what is the problem / software you try to benefit from serverless? I am looking for nice use cases where serverless can fit, so if you have some ideas and it’s open source, etc. , can you share it here please?
In NOAA, we are prototyping an Serverless environment from which API Gateway invokes Lambda functions to access Zarr located in S3. We have prototyped a centralized Dask Cluster using ECS Fargate but there are some challenges with cost/performance when using Fargate. For example, it takes serveral minutes sometimes for a task to spin up. We are interested to see if Lithops could fulfill the task of loading the chunks of data from Zarr in S3 to an xarray dataset located within a Lambda function.
Any feedback on this possibility or any other ideas is greatly appreciated!
@rsignell just showed me this thread. @tomwhite has written Cubed, which generalizes Rechunker to implement arbitrary distributed array operations in a serverless manner. You can read more in our recent blog post.
@Gil_Vernik Lithops is one of the main serverless Executors which Cubed supports, and from what Rich told me it sounds like this is something you might be interested in! I would be happy to chat about this anytime.
