ofk123
November 22, 2023, 5:27pm
1
Looking for a way around this.
Usually
from dask_gateway import Gateway
g = Gateway()
options = g.cluster_options()
followed by
options.worker_cores = 1
works from the local hub. (Really appreciate this).No option 'worker_cores' available.
Traceback
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[4], line 1
----> 1 options.worker_cores = 1
File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:114, in Options.__setattr__(self, key, value)
113 def __setattr__(self, key, value):
--> 114 return self._set(key, value, AttributeError)
File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:108, in Options._set(self, key, value, exc_cls)
106 self._fields[key].set(value)
107 except KeyError:
--> 108 raise exc_cls("No option %r available" % key) from None
AttributeError: No option 'worker_cores' available
Does someone know what could be the cause, or, where one should look/what code to run, to check currently available options?
Best,
If I’m not mistaken, those options come from a server-side configuration file . Are you using a LocalCluster or a distributed cluster? if it’s local I don’t think you need the GateWay. Or maybe the API changed? I wonder if @jrbourbeau can chime in and give us better clues of what may be happening here.
ofk123
November 28, 2023, 10:01am
3
Hi @betolink , a distributed cluster.
The last week I have used cluster = GatewayCluster(), which as default sets worker_cores to 1,cluster = g.new_cluster(options). If it is a change of API then I should change this in our code as well.
For reference I moved this post from the dask discourse (link )
@betolink is correct. You, or whoever deployed your Dask Gateway service, needs to configure an option handler that accepts and uses worker_cores for this to work. See Exposing Cluster Options — Dask Gateway 2023.9.0 documentation for an example.
ofk123
November 28, 2023, 2:02pm
5
Hi @TomAugspurger , I am using the us-central1 deployment, so I dont have the access mentioned in the example. (administrator, dask_gateway_server).
cluster = GatewayCluster() solves this “issue” for me though but other users might want to be able to configure it(?). Let me know how as a user I could help out/test things if anything.
ofk123
February 6, 2024, 1:53pm
6
Just to sum up:
Which cluster are you using us-central1-b.gcp.pangeo.io
If your problem involves code, please do your best to include an MRE
An example from https://pangeo.io/cloud.html that fails
https://pangeo.io/cloud.html#:~…
from dask_gateway import Gateway
g = Gateway()
options = g.cluster_options()
options.worker_memory = 10
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[1], line 4
2 g = Gateway()
3 options = g.cluster_options()
----> 4 options.worker_memory = 10 # 10 GB of memory per worker.
File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:114, in Options.__setattr__(self, key, value)
113 def __setattr__(self, key, value):
--> 114 return self._set(key, value, AttributeError)
File /srv/conda/envs/notebook/lib/python3.10/site-packages/dask_gateway/options.py:108, in Options._set(self, key, value, exc_cls)
106 self._fields[key].set(value)
107 except KeyError:
--> 108 raise exc_cls("No option %r available" % key) from None
AttributeError: No option 'worker_memory' available
A concise summary of your problem dask_gateway.Gateway.cluster_options() manually.
From @betolink and @guillaumeeb I understand the options come from a server-configuration-file.
Maybe Dask Gateway backend has been modified/API changed?
If options like worker_cores and worker_memory are still meant to be configurable, what is the recommended way?