I rebuilt the Dask scheduler and distributed compute system in Rust. It’s called Frisky. It’s fast, but also very early and could use some adventurous users to hammer on it.
Have fun!
Oh, for xarray it also benefits from being used with dask-array, a reimplementation of dask.array with query optimization and graph generation also in Rust. This is supported by mainline xarray, but only in git main at the moment. Instructions in the example blogpost linked above.
from dask_array.xarray import register
register()
import frisky
client = frisky.hijack(client)
This seems very cool! On getfrisky.dev the menu links to a GitHub repo that seems to be private… are you intending to open source this at some point?
That’s right! My thoughts on this are recorded here: License - Frisky . Frisky is free to use and there aren’t any commercial ambitions around it, but I’m not currently building things in the open, mostly just out of personal perference.
Just wanted to give a big +1.
I’ve been using Frisky a bit and coupled with Zarr V3’s async, Icechunk and cloud-compute, it’s pretty incredible. Getting network speeds of over 300GB/s on a single VM.
Glad you’re having a good time! 300 GB/s on one VM sounds improbable from a pure hardware perspective (barring exotic hardware) but if you’re talking about compressible data then maybe? I It could also be that frisky’s internal metrics are lying to you by each worker reporting full-system bandwidth and they’re getting summed unhelpfully.
Also if you’re on a single machine I recommend updating to 0.7.2. Frisky just switched Localhost from TCP to UDS, which gives a nice speed boost (although the network bandwidth reporting will go to zero)
Ah good to know! I’m on a single m8gn.12xlarge, which lists network bandwidth at 150 Gbps.