Access dashboard for LocalCluster on pangeo deployments

Cloud
1

Yesterday I found myself in a situation where it was very nice to have a dask dashboard to see things like the task stream and progress bars. I ended up getting a dask_gateway cluster, but would love if I could get these things working for a LocalCluster too.

What I tried on the pangeo google deployment:

from dask.distributed import Client
client = Client()
client

image
image1860×328 37.9 KB

The link to the dashboard is not useful in this instance. I wonder if there is a way to display a working link (accessible from the outside) in some clever way.

Cheers

1 Like
2

I think that Hub’s configuration could be updated to set the environment variable DASK_DISTRIBUTED__DASHBOARD__LINK to be '/user/{JUPYTERHUB_USER}/proxy/{port}/status'. See planetary-computer-hub/values.yaml at 480d206e0dc49155847352c0e89f49ae64720639 · microsoft/planetary-computer-hub · GitHub.

Julius, you should be able to test this yourself by setting that config value.

In [13]: import dask, distributed

In [14]: dask.config.set(**{"distributed.dashboard.link": "/user/{JUPYTERHUB_USER}/proxy/{port}/status"})
Out[14]: <dask.config.set at 0x7f40e4fa7f40>

And then creating your LocalCluster. I don’t think that will affect clusters created by dask gateway, which set their own dashboard link, but it’s worth confirming.

2 Likes
3

Thanks Tom! So helpful!

Maybe we should make this the default via the dask config file…

4

Yayyyy! That worked. I would love if this is the default on the deployment!

5

I just tried that on the gcloud production environment and it does not seem to work anymore.

I am doing

import dask
from distributed import LocalCluster, Client

dask.config.set(**{"distributed.dashboard.link": "/user/{JUPYTERHUB_USER}/proxy/{port}/status"})
cluster = LocalCluster()
client = Client(cluster)
client

and get

image
image1774×312 37.4 KB

But the link does not work anymore, like it did before. Very grateful for any advice. Cheers

6

The first thing to check is whether it’s something that changed in dask / distributed, or in jupyter-server-proxy.

Can you try starting another server like python -m http.server and see if you can navigate to it?

1 Like
7

I just hit the same problem. I started an http server from the command line

$ jovyan@jupyter-rabernat:~$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

And tried to access it at https://leap.2i2c.cloud/user/rabernat/proxy/8000, but I got a 404. This is the version of jupyter-server-proxy installed in this image

jupyter-server-proxy      3.2.1              pyhd8ed1ab_0    conda-forge
1 Like
8

It seems there is an interplay of image package versions and infrastructure config here that would be good to test somehow for issues like this (see Add dask proxy to labextension.yaml · Issue #253 · pangeo-data/pangeo-docker-images · GitHub and Automatically test a dask-gateway notebook for daskhubs · Issue #152 · 2i2c-org/infrastructure · GitHub).

One thing we should do is get in the habit of noting which image is being used when things are working (echo $JUPYTER_IMAGE), that makes identifying changes easier. Otherwise, it’s a bit of reverse engineering and guess work:

  1. last image bump Bump pangeo image tags · 2i2c-org/infrastructure@36c854d · GitHub

  2. difference in package versions Comparing 2022.02.04..2022.04.20 · pangeo-data/pangeo-docker-images · GitHub

I see jupyter-server-proxy=3.2.1 in both but distributed bumped from 2022.1.1 to 2022.4.1 …

9

I just tried it again, and it seems to work again! :tada:

1 Like
10

I’ve verified that https://leap.2i2c.cloud now has a functional jupyter-server-proxy with the local webserver example, and fixed a bug in how the dashboard link is configured in Fix dask daskboard link configuration by consideRatio · Pull Request #1503 · 2i2c-org/infrastructure · GitHub.

Thanks for investigating this so clearly everyone and leaving great documentation while doing it!

2 Likes