Access dashboard for LocalCluster on pangeo deployments

Yesterday I found myself in a situation where it was very nice to have a dask dashboard to see things like the task stream and progress bars. I ended up getting a dask_gateway cluster, but would love if I could get these things working for a LocalCluster too.

What I tried on the pangeo google deployment:

from dask.distributed import Client
client = Client()
client

The link to the dashboard is not useful in this instance. I wonder if there is a way to display a working link (accessible from the outside) in some clever way.

Cheers

1 Like

I think that Hub’s configuration could be updated to set the environment variable DASK_DISTRIBUTED__DASHBOARD__LINK to be '/user/{JUPYTERHUB_USER}/proxy/{port}/status'. See planetary-computer-hub/values.yaml at 480d206e0dc49155847352c0e89f49ae64720639 · microsoft/planetary-computer-hub · GitHub.

Julius, you should be able to test this yourself by setting that config value.

In [13]: import dask, distributed

In [14]: dask.config.set(**{"distributed.dashboard.link": "/user/{JUPYTERHUB_USER}/proxy/{port}/status"})
Out[14]: <dask.config.set at 0x7f40e4fa7f40>

And then creating your LocalCluster. I don’t think that will affect clusters created by dask gateway, which set their own dashboard link, but it’s worth confirming.

2 Likes

Thanks Tom! So helpful!

Maybe we should make this the default via the dask config file…

Yayyyy! That worked. I would love if this is the default on the deployment!

I just tried that on the gcloud production environment and it does not seem to work anymore.

I am doing

import dask
from distributed import LocalCluster, Client

dask.config.set(**{"distributed.dashboard.link": "/user/{JUPYTERHUB_USER}/proxy/{port}/status"})
cluster = LocalCluster()
client = Client(cluster)
client

and get


But the link does not work anymore, like it did before. Very grateful for any advice. Cheers

The first thing to check is whether it’s something that changed in dask / distributed, or in jupyter-server-proxy.

Can you try starting another server like python -m http.server and see if you can navigate to it?

1 Like

I just hit the same problem. I started an http server from the command line

$ jovyan@jupyter-rabernat:~$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

And tried to access it at https://leap.2i2c.cloud/user/rabernat/proxy/8000, but I got a 404. This is the version of jupyter-server-proxy installed in this image

jupyter-server-proxy      3.2.1              pyhd8ed1ab_0    conda-forge
1 Like

It seems there is an interplay of image package versions and infrastructure config here that would be good to test somehow for issues like this (see Add dask proxy to labextension.yaml · Issue #253 · pangeo-data/pangeo-docker-images · GitHub and Automatically test a dask-gateway notebook for daskhubs · Issue #152 · 2i2c-org/infrastructure · GitHub).

One thing we should do is get in the habit of noting which image is being used when things are working (echo $JUPYTER_IMAGE), that makes identifying changes easier. Otherwise, it’s a bit of reverse engineering and guess work:

  1. last image bump Bump pangeo image tags · 2i2c-org/infrastructure@36c854d · GitHub

  2. difference in package versions Comparing 2022.02.04..2022.04.20 · pangeo-data/pangeo-docker-images · GitHub

I see jupyter-server-proxy=3.2.1 in both but distributed bumped from 2022.1.1 to 2022.4.1 …

I just tried it again, and it seems to work again! :tada:

1 Like

I’ve verified that https://leap.2i2c.cloud now has a functional jupyter-server-proxy with the local webserver example, and fixed a bug in how the dashboard link is configured in Fix dask daskboard link configuration by consideRatio · Pull Request #1503 · 2i2c-org/infrastructure · GitHub.

Thanks for investigating this so clearly everyone and leaving great documentation while doing it!

2 Likes