No longer able to set `dask_gateway.Gateway.cluster_options()` manually from user-end

Cloud Pangeo Cloud Support
1

Looking for a way around this.

Usually

from dask_gateway import Gateway
g = Gateway()
options = g.cluster_options()

followed by

options.worker_cores = 1

works from the local hub. (Really appreciate this).
This seems to be failing yesterday and today with No option 'worker_cores' available.

Traceback 
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 1
----> 1 options.worker_cores = 1

File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:114, in Options.__setattr__(self, key, value)
    113 def __setattr__(self, key, value):
--> 114     return self._set(key, value, AttributeError)

File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:108, in Options._set(self, key, value, exc_cls)
    106     self._fields[key].set(value)
    107 except KeyError:
--> 108     raise exc_cls("No option %r available" % key) from None

AttributeError: No option 'worker_cores' available

Does someone know what could be the cause, or, where one should look/what code to run, to check currently available options?

Best,

2

If I’m not mistaken, those options come from a server-side configuration file. Are you using a LocalCluster or a distributed cluster? if it’s local I don’t think you need the GateWay. Or maybe the API changed? I wonder if @jrbourbeau can chime in and give us better clues of what may be happening here.

3

Hi @betolink, a distributed cluster.

The last week I have used cluster = GatewayCluster(), which as default sets worker_cores to 1,
instead of cluster = g.new_cluster(options). If it is a change of API then I should change this in our code as well.

For reference I moved this post from the dask discourse (link)

4

@betolink is correct. You, or whoever deployed your Dask Gateway service, needs to configure an option handler that accepts and uses worker_cores for this to work. See Exposing Cluster Options — Dask Gateway 2023.9.0 documentation for an example.

5

Hi @TomAugspurger, I am using the us-central1 deployment, so I dont have the access mentioned in the example. (administrator, dask_gateway_server).

cluster = GatewayCluster() solves this “issue” for me though but other users might want to be able to configure it(?). Let me know how as a user I could help out/test things if anything.

6

Just to sum up:

Which cluster are you using
us-central1-b.gcp.pangeo.io

If your problem involves code, please do your best to include an MRE

An example from https://pangeo.io/cloud.html that fails

https://pangeo.io/cloud.html#:~…

from dask_gateway import Gateway
g = Gateway()
options = g.cluster_options()
options.worker_memory = 10 

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 4
      2 g = Gateway()
      3 options = g.cluster_options()
----> 4 options.worker_memory = 10  # 10 GB of memory per worker.

File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:114, in Options.__setattr__(self, key, value)
    113 def __setattr__(self, key, value):
--> 114     return self._set(key, value, AttributeError)

File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:108, in Options._set(self, key, value, exc_cls)
    106     self._fields[key].set(value)
    107 except KeyError:
--> 108     raise exc_cls("No option %r available" % key) from None

AttributeError: No option 'worker_memory' available

A concise summary of your problem
No longer able to set dask_gateway.Gateway.cluster_options() manually.

From @betolink and @guillaumeeb I understand the options come from a server-configuration-file.

  • Maybe Dask Gateway backend has been modified/API changed?
  • If options like worker_cores and worker_memory are still meant to be configurable, what is the recommended way?