AttributeError: 'EntryPoint' object has no attribute '_key', stops computation after a process has been running

Hi,

I am testing computations on the Google Cloud Deployment (Server-options: 16 CPUs), and want to ask for help with interpreting the error that shows up on several of my delayed-dask-processes, so that I can understand more of the underlying process with dask on Pangeo, and can troubleshoot this further.

Running a “results = dask.compute(*list_of_delayed_functions)”-cell works, with the dask-processes fully visible in the dashboard. After a while the computation stops with the message AttributeError: 'EntryPoint' object has no attribute '_key.

Not familiar with this errormessage, and hope to receive directions or general advice.

  • Rerunning the same computation-cell makes it running for a while again, but stops at a different function, so it is not clear that one specific function is triggering this error.
  • The default notebook-environment is updated in the terminal with conda update -n notebook --all

I could not fit the errormessage in one photo, but have copied the full message preformatted below.

Best,
Ola

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [14], line 1
----> 1 results = dask.compute(*functions_at_a_gridpoint)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/base.py:600, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    597     keys.append(x.__dask_keys__())
    598     postcomputes.append(x.__dask_postcompute__())
--> 600 results = schedule(dsk, keys, **kwargs)
    601 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/client.py:3057, in Client.get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   3055         should_rejoin = False
   3056 try:
-> 3057     results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   3058 finally:
   3059     for f in futures.values():

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/client.py:2226, in Client.gather(self, futures, errors, direct, asynchronous)
   2224 else:
   2225     local_worker = None
-> 2226 return self.sync(
   2227     self._gather,
   2228     futures,
   2229     errors=errors,
   2230     direct=direct,
   2231     local_worker=local_worker,
   2232     asynchronous=asynchronous,
   2233 )

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/utils.py:339, in SyncMethodMixin.sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    337     return future
    338 else:
--> 339     return sync(
    340         self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    341     )

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/utils.py:406, in sync(loop, func, callback_timeout, *args, **kwargs)
    404 if error:
    405     typ, exc, tb = error
--> 406     raise exc.with_traceback(tb)
    407 else:
    408     return result

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/utils.py:379, in sync.<locals>.f()
    377         future = asyncio.wait_for(future, callback_timeout)
    378     future = asyncio.ensure_future(future)
--> 379     result = yield future
    380 except Exception:
    381     error = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.9/site-packages/tornado/gen.py:762, in Runner.run(self)
    759 exc_info = None
    761 try:
--> 762     value = future.result()
    763 except Exception:
    764     exc_info = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/client.py:2089, in Client._gather(self, futures, errors, direct, local_worker)
   2087         exc = CancelledError(key)
   2088     else:
-> 2089         raise exception.with_traceback(traceback)
   2090     raise exc
   2091 if errors == "skip":

Cell In [9], line 7, in optimize()
      1 def optimize(lat1,lon1):
      3     """
      4     Computes gridded coefficients of fit based on basisfunctions and dynamic height data within a local window
      5     """
----> 7     ds = xr.open_zarr(mapper, consolidated=True, chunks=None, decode_times=False)
      8     distance = xr.apply_ufunc(great_circle_distance, lat1, lon1, ds.latitude.load(), ds.longitude.load())
      9     ds_radius = ds.where(distance < window_size, drop=True)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/zarr.py:789, in open_zarr()
    778 if kwargs:
    779     raise TypeError(
    780         "open_zarr() got unexpected keyword arguments " + ",".join(kwargs.keys())
    781     )
    783 backend_kwargs = {
    784     "synchronizer": synchronizer,
    785     "consolidated": consolidated,
    786     "overwrite_encoded_chunks": overwrite_encoded_chunks,
    787     "chunk_store": chunk_store,
    788     "storage_options": storage_options,
--> 789     "stacklevel": 4,
    790 }
    792 ds = open_dataset(
    793     filename_or_obj=store,
    794     group=group,
   (...)
    805     use_cftime=use_cftime,
    806 )
    807 return ds

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py:517, in open_dataset()
    514 if engine is None:
    515     engine = plugins.guess_engine(filename_or_obj)
--> 517 backend = plugins.get_backend(engine)
    519 decoders = _resolve_decoders_kwargs(
    520     decode_cf,
    521     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)
    527     decode_coords=decode_coords,
    528 )
    530 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/plugins.py:161, in get_backend()
    159 """Select open_dataset method based on current engine."""
    160 if isinstance(engine, str):
--> 161     engines = list_engines()
    162     if engine not in engines:
    163         raise ValueError(
    164             f"unrecognized engine {engine} must be one of: {list(engines)}"
    165         )

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/plugins.py:106, in list_engines()
    104 else:
    105     entrypoints = entry_points().get("xarray.backends", ())
--> 106 return build_engines(entrypoints)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/plugins.py:91, in build_engines()
     89     if backend.available:
     90         backend_entrypoints[backend_name] = backend
---> 91 entrypoints = remove_duplicates(entrypoints)
     92 external_backend_entrypoints = backends_dict_from_pkg(entrypoints)
     93 backend_entrypoints.update(external_backend_entrypoints)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/plugins.py:23, in remove_duplicates()
     20 unique_entrypoints = []
     21 for name, matches in entrypoints_grouped:
     22     # remove equal entrypoints
---> 23     matches = list(set(matches))
     24     unique_entrypoints.append(matches[0])
     25     matches_len = len(matches)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py:239, in __eq__()
    238 def __eq__(self, other):
--> 239     return self._key() == other._key()

AttributeError: 'EntryPoint' object has no attribute '_key'

This sounds like a version clash between your environments. Can you please check your version numbers, and specify how you set up your environments?

Hi, thanks,

I am new to using Pangeo Cloud, and use the existing environment ‘notebook’. I do not install other packages. The notebook-environment has the versions pasted below, but I see some version-numbers are slightly lower than in pangeo-docker-images/packages.txt. If this is the problem, what is the correct way to update the environments?

I use data stored in Google cloud storage, with

with open('token.json') as f:
    token = json.load(f)
gcs = gcsfs.GCSFileSystem(token=token)

mapper = gcs.get_mapper('path to .zarr-file')

and start computing the delayed functions with

def function(x,y,p):
    ds = xr.open_zarr(mapper, consolidated=True, chunks=None)
    ...
    return b

from dask_gateway import GatewayCluster
cluster = GatewayCluster()
cluster.adapt()

from distributed import Client
client = Client(cluster)

from dask import delayed
import dask
function_delayed = delayed(function)

functions = []

for x, y in zip(xs, ys):
    functions.append(function_delayed(x,y,p))

results = dask.compute(*functions)

Sometimes I see rerunning the cell results = dask.compute(*functions) works, as in the picture attached, where I ran the computations on the same 500 points twice, and it completed the second time after failing with the AttributeError first.

Name Version Build Channel

* _libgcc_mutex             0.1                 conda_forge    conda-forge
* _openmp_mutex             4.5                       2_gnu    conda-forge
* adal                      1.2.7              pyhd8ed1ab_0    conda-forge
* adlfs                     2022.9.1           pyhd8ed1ab_0    conda-forge
* affine                    2.3.1              pyhd8ed1ab_0    conda-forge
* aiobotocore               2.4.0              pyhd8ed1ab_0    conda-forge
* aiohttp                   3.8.1            py39hb9d737c_1    conda-forge
* aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
* aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
* alembic                   1.8.1              pyhd8ed1ab_0    conda-forge
* anyio                     3.6.1              pyhd8ed1ab_1    conda-forge
* aom                       3.4.0                h27087fc_1    conda-forge
* appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
* argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
* argon2-cffi-bindings      21.2.0           py39hb9d737c_2    conda-forge
* argopy                    0.1.12             pyhd8ed1ab_0    conda-forge
* arrow-cpp                 9.0.0           py39hd75b5e9_6_cpu    conda-forge
* asciitree                 0.3.3                      py_2    conda-forge
* asn1crypto                1.5.1              pyhd8ed1ab_0    conda-forge
* astropy                   5.1              py39hd257fcd_0    conda-forge
* asttokens                 2.0.8              pyhd8ed1ab_0    conda-forge
* async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
* async_generator           1.10                       py_0    conda-forge
* atk-1.0                   2.36.0               h3371d22_4    conda-forge
* attrs                     22.1.0             pyh71513ae_1    conda-forge
* av                        9.2.0            py39hbd8a108_2    conda-forge
* aws-c-cal                 0.5.11               h95a6274_0    conda-forge
* aws-c-common              0.6.2                h7f98852_0    conda-forge
* aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
* aws-c-io                  0.10.5               hfb6a706_0    conda-forge
* aws-checksums             0.1.11               ha31a3da_7    conda-forge
* aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
* awscli                    1.25.60          py39hf3d152e_0    conda-forge
* azure-core                1.25.1             pyhd8ed1ab_0    conda-forge
* azure-datalake-store      0.0.51             pyh9f0ad1d_0    conda-forge
* azure-identity            1.11.0             pyhd8ed1ab_0    conda-forge
* azure-storage-blob        12.13.1            pyhd8ed1ab_0    conda-forge
* babel                     2.10.3             pyhd8ed1ab_0    conda-forge
* backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
* backports                 1.0                        py_2    conda-forge
* backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
* beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
* binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
* binutils_linux-64         2.36                hf3e587d_10    conda-forge
* bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
* blinker                   1.4                        py_1    conda-forge
* blosc                     1.21.1               h83bc5f7_3    conda-forge
* bokeh                     2.4.3              pyhd8ed1ab_3    conda-forge
* boost-cpp                 1.78.0               h75c5d50_1    conda-forge
* boto3                     1.24.59            pyhd8ed1ab_0    conda-forge
* botocore                  1.27.59            pyhd8ed1ab_0    conda-forge
* bottleneck                1.3.5            py39hd257fcd_0    conda-forge
* bounded-pool-executor     0.0.3              pyhd8ed1ab_0    conda-forge
* branca                    0.5.0              pyhd8ed1ab_0    conda-forge
* brotli                    1.0.9                h166bdaf_7    conda-forge
* brotli-bin                1.0.9                h166bdaf_7    conda-forge
* brotlipy                  0.7.0           py39hb9d737c_1004    conda-forge
* brunsli                   0.1                  h9c3ff4c_0    conda-forge
* bzip2                     1.0.8                h7f98852_4    conda-forge
* c-ares                    1.18.1               h7f98852_0    conda-forge
* c-blosc2                  2.4.1                h7a311fb_0    conda-forge
* ca-certificates           2022.9.14            ha878542_0    conda-forge
* cached-property           1.5.2                hd8ed1ab_1    conda-forge
* cached_property           1.5.2              pyha770c72_1    conda-forge
* cachetools                5.2.0              pyhd8ed1ab_0    conda-forge
* cachey                    0.2.1              pyh9f0ad1d_0    conda-forge
* cairo                     1.16.0            ha61ee94_1014    conda-forge
* cartopy                   0.21.0           py39hf5d525c_0    conda-forge
* certifi                   2022.9.14          pyhd8ed1ab_0    conda-forge
* certipy                   0.1.3                      py_0    conda-forge
* cf_xarray                 0.7.4              pyhd8ed1ab_0    conda-forge
* cffi                      1.15.1           py39he91dace_0    conda-forge
* cfgrib                    0.9.10.1           pyhd8ed1ab_0    conda-forge
* cfitsio                   4.1.0                hd9d235c_0    conda-forge
* cftime                    1.6.2            py39h2ae25f5_0    conda-forge
* cgen                      2020.1                     py_0    conda-forge
* charls                    2.3.4                h9c3ff4c_0    conda-forge
* charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
* ciso                      0.1.0            py39hd257fcd_4    conda-forge
* click                     8.1.3            py39hf3d152e_0    conda-forge
* click-plugins             1.1.1                      py_0    conda-forge
* cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
* cloudpickle               2.2.0              pyhd8ed1ab_0    conda-forge
* colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
* colorcet                  3.0.0              pyhd8ed1ab_0    conda-forge
* configobj                 5.0.6                      py_0    conda-forge
* contourpy                 1.0.5            py39hf939315_0    conda-forge
* croniter                  0.3.36             pyhd8ed1ab_0    conda-forge
* cryptography              37.0.4           py39hd97740a_0    conda-forge
* curl                      7.83.1               h7bff187_0    conda-forge
* cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
* cytoolz                   0.12.0           py39hb9d737c_0    conda-forge
* dask                      2022.9.1           pyhd8ed1ab_0    conda-forge
* dask-core                 2022.9.1           pyhd8ed1ab_0    conda-forge
* dask-gateway              2022.6.1           pyh8af1aa0_0    conda-forge
* dask-glm                  0.2.0                      py_1    conda-forge
* dask-labextension         5.3.0              pyhd8ed1ab_1    conda-forge
* dask-ml                   2022.5.27          pyhd8ed1ab_0    conda-forge
* datashader                0.14.2             pyh6c4a22f_0    conda-forge
* datashape                 0.5.4                      py_1    conda-forge
* dav1d                     1.0.0                h166bdaf_1    conda-forge
* debugpy                   1.6.3            py39h5a03fae_0    conda-forge
* decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
* defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
* descartes                 1.1.0                      py_4    conda-forge
* distributed               2022.9.1           pyhd8ed1ab_0    conda-forge
* docker-py                 5.0.3            py39hf3d152e_2    conda-forge
* docker-pycreds            0.4.0                      py_0    conda-forge
* docopt                    0.6.2                      py_1    conda-forge
* docrep                    0.3.2              pyh44b312d_0    conda-forge
* docutils                  0.16             py39hf3d152e_3    conda-forge
* donfig                    0.7.0              pyhd8ed1ab_1    conda-forge
* earthdata                 0.2.2              pyhd8ed1ab_0    conda-forge
* eccodes                   2.27.0               h164a9dd_0    conda-forge
* entrypoints               0.4                pyhd8ed1ab_0    conda-forge
* eofs                      1.4.0                      py_0    conda-forge
* erddapy                   1.2.1              pyhd8ed1ab_0    conda-forge
* esmf                      8.2.0           mpi_mpich_h5a1934d_102    conda-forge
* esmpy                     8.2.0           mpi_mpich_py39h8bb458d_101    conda-forge
* executing                 1.0.0              pyhd8ed1ab_0    conda-forge
* expat                     2.4.9                h27087fc_0    conda-forge
* fastapi                   0.85.0             pyhd8ed1ab_0    conda-forge
* fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
* fastjmd95                 0.2.1              pyh44b312d_0    conda-forge
* fastprogress              1.0.3              pyhd8ed1ab_0    conda-forge
* ffmpeg                    4.4.2           gpl_hfe78399_107    conda-forge
* findlibs                  0.0.2              pyhd8ed1ab_0    conda-forge
* fiona                     1.8.21           py39h54775ec_2    conda-forge
* flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
* flox                      0.5.9              pyhd8ed1ab_1    conda-forge
* folium                    0.12.1.post1       pyhd8ed1ab_1    conda-forge
* font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
* font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
* font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
* font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
* fontconfig                2.14.0               hc2a2eb6_1    conda-forge
* fonts-conda-ecosystem     1                             0    conda-forge
* fonts-conda-forge         1                             0    conda-forge
* fonttools                 4.37.3           py39hb9d737c_0    conda-forge
* freeglut                  3.2.2                h9c3ff4c_1    conda-forge
* freetype                  2.12.1               hca18f0e_0    conda-forge
* freexl                    1.0.6                h166bdaf_1    conda-forge
* fribidi                   1.0.10               h36c2ea0_0    conda-forge
* frozenlist                1.3.1            py39hb9d737c_0    conda-forge
* fsspec                    2022.8.2           pyhd8ed1ab_0    conda-forge
* ftfy                      6.1.1              pyhd8ed1ab_0    conda-forge
* future                    0.18.2           py39hf3d152e_5    conda-forge
* gcc_impl_linux-64         12.1.0              hea43390_16    conda-forge
* gcc_linux-64              12.1.0              h3bb4806_10    conda-forge
* gcm_filters               0.3.0              pyhd8ed1ab_0    conda-forge
* gcsfs                     2022.8.2           pyhd8ed1ab_0    conda-forge
* gdal                      3.5.2            py39h92c1d47_1    conda-forge
* gdk-pixbuf                2.42.8               hff1cb4f_1    conda-forge
* geocube                   0.3.2              pyhd8ed1ab_0    conda-forge
* geogif                    0.1.3              pyhd8ed1ab_0    conda-forge
* geographiclib             1.52               pyhd8ed1ab_0    conda-forge
* geopandas                 0.11.1             pyhd8ed1ab_0    conda-forge
* geopandas-base            0.11.1             pyha770c72_0    conda-forge
* geopy                     2.2.0              pyhd8ed1ab_0    conda-forge
* geos                      3.11.0               h27087fc_0    conda-forge
* geotiff                   1.7.1                h4fc65e6_3    conda-forge
* geoviews-core             1.9.5              pyha770c72_0    conda-forge
* gettext                   0.19.8.1          h73d1719_1008    conda-forge
* gflags                    2.2.2             he1b5a44_1004    conda-forge
* gh                        2.15.0               ha8f183a_0    conda-forge
* gh-scoped-creds           4.1                pyhd8ed1ab_0    conda-forge
* giflib                    5.2.1                h36c2ea0_2    conda-forge
* git-lfs                   3.2.0                ha770c72_0    conda-forge
* gitdb                     4.0.9              pyhd8ed1ab_0    conda-forge
* gitpython                 3.1.27             pyhd8ed1ab_0    conda-forge
* glog                      0.6.0                h6f12383_0    conda-forge
* gmp                       6.2.1                h58526e2_0    conda-forge
* gnutls                    3.7.7                hf3e180e_0    conda-forge
* google-api-core           2.10.1             pyhd8ed1ab_0    conda-forge
* google-auth               2.11.0             pyh6c4a22f_0    conda-forge
* google-auth-oauthlib      0.5.3              pyhd8ed1ab_0    conda-forge
* google-cloud-core         2.3.2              pyhd8ed1ab_0    conda-forge
* google-cloud-storage      2.5.0              pyh6c4a22f_0    conda-forge
* google-crc32c             1.1.2            py39h8db88ab_3    conda-forge
* google-resumable-media    2.3.3              pyhd8ed1ab_0    conda-forge
* googleapis-common-protos  1.56.4           py39hf3d152e_0    conda-forge
* graphite2                 1.3.13            h58526e2_1001    conda-forge
* graphviz                  6.0.1                h5abf519_0    conda-forge
* greenlet                  1.1.3            py39h5a03fae_0    conda-forge
* grpc-cpp                  1.47.1               h05bd8bd_6    conda-forge
* grpcio                    1.47.1           py39h712372c_6    conda-forge
* gsw                       3.4.0            py39hd257fcd_3    conda-forge
* gtk2                      2.24.33              h90689f9_2    conda-forge
* gts                       0.7.6                h64030ff_2    conda-forge
* h11                       0.13.0             pyhd8ed1ab_1    conda-forge
* h3-py                     3.7.4            py39h5a03fae_0    conda-forge
* h5netcdf                  1.0.2              pyhd8ed1ab_0    conda-forge
* h5py                      3.7.0           nompi_py39hd51670d_101    conda-forge
* harfbuzz                  5.2.0                hf9f4e7c_0    conda-forge
* hdf4                      4.2.15               h9772cbc_4    conda-forge
* hdf5                      1.12.2          mpi_mpich_h08b82f9_0    conda-forge
* heapdict                  1.0.1                      py_0    conda-forge
* holoviews                 1.15.0             pyhd8ed1ab_0    conda-forge
* hvplot                    0.8.1              pyhd8ed1ab_0    conda-forge
* icu                       70.1                 h27087fc_0    conda-forge
* idna                      3.4                pyhd8ed1ab_0    conda-forge
* imagecodecs               2022.8.8         py39h090f8d4_5    conda-forge
* imageio                   2.22.0             pyhfa7a67d_0    conda-forge
* importlib-metadata        4.11.4           py39hf3d152e_0    conda-forge
* importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
* importlib_resources       5.9.0              pyhd8ed1ab_0    conda-forge
* iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
* intake                    0.6.6              pyhd8ed1ab_0    conda-forge
* intake-esm                2022.9.18          pyhd8ed1ab_0    conda-forge
* intake-geopandas          0.4.0              pyhd8ed1ab_0    conda-forge
* intake-stac               0.4.0              pyhd8ed1ab_0    conda-forge
* intake-xarray             0.6.1              pyhd8ed1ab_0    conda-forge
* ipykernel                 6.15.3             pyh210e3f2_0    conda-forge
* ipyleaflet                0.17.1             pyhd8ed1ab_0    conda-forge
* ipyspin                   0.1.2              pyh9f0ad1d_0    conda-forge
* ipython                   8.5.0              pyh41d4057_1    conda-forge
* ipython_genutils          0.2.0                      py_1    conda-forge
* ipytree                   0.2.2              pyhd8ed1ab_0    conda-forge
* ipyurl                    0.1.2              pyh3684270_1    conda-forge
* ipywidgets                8.0.2              pyhd8ed1ab_1    conda-forge
* isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
* jasper                    2.0.33               ha77e612_0    conda-forge
* jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
* jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
* jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
* joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
* jpeg                      9e                   h166bdaf_2    conda-forge
* json-c                    0.16                 hc379101_0    conda-forge
* json5                     0.9.5              pyh9f0ad1d_0    conda-forge
* jsonschema                4.16.0             pyhd8ed1ab_0    conda-forge
* jupyter-panel-proxy       0.1.0                      py_0    conda-forge
* jupyter-resource-usage    0.6.2              pyhd8ed1ab_0    conda-forge
* jupyter-server-mathjax    0.2.6              pyhc268e32_0    conda-forge
* jupyter-server-proxy      3.2.2              pyhd8ed1ab_0    conda-forge
* jupyter_client            7.3.4              pyhd8ed1ab_0    conda-forge
* jupyter_core              4.11.1           py39hf3d152e_0    conda-forge
* jupyter_server            1.18.1             pyhd8ed1ab_0    conda-forge
* jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
* jupyterhub-base           3.0.0              pyhd8ed1ab_0    conda-forge
* jupyterhub-singleuser     3.0.0              pyhd8ed1ab_0    conda-forge
* jupyterlab                3.4.7              pyhd8ed1ab_0    conda-forge
* jupyterlab-git            0.39.2             pyhd8ed1ab_0    conda-forge
* jupyterlab-s3-browser     0.12.0                   pypi_0    pypi
* jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
* jupyterlab_server         2.15.1             pyhd8ed1ab_0    conda-forge
* jupyterlab_widgets        3.0.3              pyhd8ed1ab_0    conda-forge
* jxrlib                    1.1                  h7f98852_2    conda-forge
* kealib                    1.4.15               ha7026e8_1    conda-forge
* kerchunk                  0.0.8              pyhd8ed1ab_0    conda-forge
* kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
* keyutils                  1.6.1                h166bdaf_0    conda-forge
* kiwisolver                1.4.4            py39hf939315_0    conda-forge
* krb5                      1.19.3               h3790be6_0    conda-forge
* lame                      3.100             h7f98852_1001    conda-forge
* lcms2                     2.12                 hddcbb42_0    conda-forge
* ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
* lerc                      4.0.0                h27087fc_0    conda-forge
* libabseil                 20220623.0      cxx17_h48a1fff_4    conda-forge
* libaec                    1.0.6                h9c3ff4c_0    conda-forge
* libavif                   0.10.1               h166bdaf_1    conda-forge
* libblas                   3.9.0           16_linux64_openblas    conda-forge
* libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
* libbrotlidec              1.0.9                h166bdaf_7    conda-forge
* libbrotlienc              1.0.9                h166bdaf_7    conda-forge
* libcblas                  3.9.0           16_linux64_openblas    conda-forge
* libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
* libcurl                   7.83.1               h7bff187_0    conda-forge
* libdap4                   3.20.6               hd7c4107_2    conda-forge
* libdeflate                1.14                 h166bdaf_0    conda-forge
* libdrm                    2.4.113              h166bdaf_0    conda-forge
* libedit                   3.1.20191231         he28a2e2_2    conda-forge
* libev                     4.33                 h516909a_1    conda-forge
* libevent                  2.1.10               h9b69904_4    conda-forge
* libffi                    3.4.2                h7f98852_5    conda-forge
* libgcc-devel_linux-64     12.1.0              h1ec3361_16    conda-forge
* libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
* libgd                     2.3.3                h18fbbfe_3    conda-forge
* libgdal                   3.5.2                hc23bfc3_1    conda-forge
* libgfortran-ng            12.1.0              h69a702a_16    conda-forge
* libgfortran5              12.1.0              hdcd56e2_16    conda-forge
* libglib                   2.72.1               h2d90d5f_0    conda-forge
* libglu                    9.0.0             he1b5a44_1001    conda-forge
* libgomp                   12.1.0              h8d9b700_16    conda-forge
* libgoogle-cloud           2.2.0                h838d150_0    conda-forge
* libiconv                  1.16                 h516909a_0    conda-forge
* libidn2                   2.3.3                h166bdaf_0    conda-forge
* libkml                    1.3.0             h37653c0_1015    conda-forge
* liblapack                 3.9.0           16_linux64_openblas    conda-forge
* libllvm10                 10.0.1               he513fc3_3    conda-forge
* libnetcdf                 4.8.1           mpi_mpich_h06c54e2_4    conda-forge
* libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
* libnsl                    2.0.0                h7f98852_0    conda-forge
* libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
* libpciaccess              0.16                 h516909a_0    conda-forge
* libpng                    1.6.38               h753d276_0    conda-forge
* libpq                     14.5                 hd77ab85_0    conda-forge
* libprotobuf               3.21.5               h6239696_3    conda-forge
* librsvg                   2.54.4               h7abd40a_0    conda-forge
* librttopo                 1.1.0               hf730bdb_11    conda-forge
* libsanitizer              12.1.0              ha89aaad_16    conda-forge
* libsodium                 1.0.18               h36c2ea0_1    conda-forge
* libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
* libspatialite             5.0.1               h38b5f51_18    conda-forge
* libsqlite                 3.39.3               h753d276_0    conda-forge
* libssh2                   1.10.0               haa6b8db_3    conda-forge
* libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
* libtasn1                  4.19.0               h166bdaf_0    conda-forge
* libthrift                 0.16.0               h491838f_2    conda-forge
* libtiff                   4.4.0                h55922b4_4    conda-forge
* libtool                   2.4.6             h9c3ff4c_1008    conda-forge
* libunistring              0.9.10               h7f98852_0    conda-forge
* libutf8proc               2.7.0                h7f98852_0    conda-forge
* libuuid                   2.32.1            h7f98852_1000    conda-forge
* libva                     2.15.0               h166bdaf_0    conda-forge
* libvpx                    1.11.0               h9c3ff4c_3    conda-forge
* libwebp                   1.2.4                h522a892_0    conda-forge
* libwebp-base              1.2.4                h166bdaf_0    conda-forge
* libxcb                    1.13              h7f98852_1004    conda-forge
* libxml2                   2.9.14               h22db469_4    conda-forge
* libxslt                   1.1.35               h8affb1d_0    conda-forge
* libzip                    1.9.2                hc869a4a_1    conda-forge
* libzlib                   1.2.12               h166bdaf_3    conda-forge
* libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
* llvmlite                  0.36.0           py39h1bbdace_0    conda-forge
* locket                    1.0.0              pyhd8ed1ab_0    conda-forge
* lxml                      4.9.1            py39hb9d737c_0    conda-forge
* lz4                       4.0.0            py39h029007f_2    conda-forge
* lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
* mailchecker               5.0.1              pyhd8ed1ab_0    conda-forge
* mako                      1.2.2              pyhd8ed1ab_0    conda-forge
* mapclassify               2.4.3              pyhd8ed1ab_0    conda-forge
* markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
* markupsafe                2.1.1            py39hb9d737c_1    conda-forge
* marshmallow               3.18.0             pyhd8ed1ab_0    conda-forge
* marshmallow-oneofschema   3.0.1              pyhd8ed1ab_0    conda-forge
* matplotlib-base           3.6.0            py39hf9fd14e_0    conda-forge
* matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
* mercantile                1.2.1              pyhd8ed1ab_0    conda-forge
* metpy                     1.3.1              pyhd8ed1ab_0    conda-forge
* mistune                   2.0.4              pyhd8ed1ab_0    conda-forge
* morecantile               3.1.2              pyhd8ed1ab_0    conda-forge
* mpi                       1.0                       mpich    conda-forge
* mpi4py                    3.1.3            py39h32b9844_2    conda-forge
* mpich                     4.0.2              h846660c_100    conda-forge
* msal                      1.19.0             pyhd8ed1ab_0    conda-forge
* msal_extensions           1.0.0              pyhd8ed1ab_0    conda-forge
* msgpack-python            1.0.4            py39hf939315_0    conda-forge
* msrest                    0.7.1              pyhd8ed1ab_0    conda-forge
* multidict                 6.0.2            py39hb9d737c_1    conda-forge
* multipledispatch          0.6.0                      py_0    conda-forge
* munch                     2.5.0                      py_0    conda-forge
* munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
* mypy_extensions           0.4.3            py39hf3d152e_5    conda-forge
* natsort                   8.2.0              pyhd8ed1ab_0    conda-forge
* nb_conda_kernels          2.3.1            py39hf3d152e_1    conda-forge
* nbclassic                 0.4.3              pyhd8ed1ab_0    conda-forge
* nbclient                  0.6.8              pyhd8ed1ab_0    conda-forge
* nbconvert                 7.0.0              pyhd8ed1ab_0    conda-forge
* nbconvert-core            7.0.0              pyhd8ed1ab_0    conda-forge
* nbconvert-pandoc          7.0.0              pyhd8ed1ab_0    conda-forge
* nbdime                    3.1.1              pyhd8ed1ab_0    conda-forge
* nbformat                  5.6.0              pyhd8ed1ab_0    conda-forge
* nbgitpuller               1.1.0              pyhd8ed1ab_0    conda-forge
* nbstripout                0.6.0              pyhd8ed1ab_0    conda-forge
* nc-time-axis              1.4.1              pyhd8ed1ab_0    conda-forge
* ncurses                   6.3                  h27087fc_1    conda-forge
* nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
* netcdf-fortran            4.6.0           mpi_mpich_hd09bd1e_0    conda-forge
* netcdf4                   1.6.1           nompi_py39hfaa66c4_100    conda-forge
* nettle                    3.8.1                hc379101_1    conda-forge
* networkx                  2.8.6              pyhd8ed1ab_0    conda-forge
* noise                     1.2.2           py39hb9d737c_1003    conda-forge
* notebook                  6.4.12             pyha770c72_0    conda-forge
* notebook-shim             0.1.0              pyhd8ed1ab_0    conda-forge
* nspr                      4.32                 h9c3ff4c_1    conda-forge
* nss                       3.78                 h2350873_0    conda-forge
* numba                     0.53.1           py39h56b8d98_1    conda-forge
* numcodecs                 0.10.2           py39h5a03fae_0    conda-forge
* numpy                     1.23.3           py39hba7629e_0    conda-forge
* numpy_groupies            0.9.19             pyhd8ed1ab_0    conda-forge
* oauthlib                  3.2.1              pyhd8ed1ab_0    conda-forge
* odc-geo                   0.3.0              pyhd8ed1ab_0    conda-forge
* openh264                  2.3.0                h27087fc_0    conda-forge
* openjpeg                  2.5.0                h7d73246_1    conda-forge
* openssl                   1.1.1q               h166bdaf_0    conda-forge
* orc                       1.7.6                h09e0d61_1    conda-forge

* p11-kit                   0.24.1               hc5aa10d_0    conda-forge
* packaging                 21.3               pyhd8ed1ab_0    conda-forge
* pamela                    1.0.0                      py_0    conda-forge
* pandas                    1.5.0            py39h4661b88_0    conda-forge
* pandoc                    2.19.2               ha770c72_0    conda-forge
* pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
* panel                     0.13.1             pyhd8ed1ab_0    conda-forge
* pangeo-dask               2022.09.19           hd8ed1ab_0    conda-forge
* pangeo-notebook           2022.09.21           hd8ed1ab_0    conda-forge
* pango                     1.50.10              hc4f8a73_0    conda-forge
* param                     1.12.2             pyh6c4a22f_0    conda-forge
* parcels                   2.3.2            py39hf3d152e_0    conda-forge
* parquet-cpp               1.5.1                         2    conda-forge
* parso                     0.8.3              pyhd8ed1ab_0    conda-forge
* partd                     1.3.0              pyhd8ed1ab_0    conda-forge
* patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
* pcre                      8.45                 h9c3ff4c_0    conda-forge
* pendulum                  2.1.2            py39hb9d737c_4    conda-forge
* pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
* phonenumbers              8.12.55            pyhd8ed1ab_0    conda-forge
* pickleshare               0.7.5                   py_1003    conda-forge
* pillow                    9.2.0            py39hd5dbb17_2    conda-forge
* pint                      0.19.2             pyhd8ed1ab_0    conda-forge
* pint-xarray               0.3                pyhd8ed1ab_0    conda-forge
* pip                       22.2.2             pyhd8ed1ab_0    conda-forge
* pixman                    0.40.0               h36c2ea0_0    conda-forge
* pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
* platformdirs              2.5.2              pyhd8ed1ab_1    conda-forge
* pluggy                    1.0.0            py39hf3d152e_3    conda-forge
* pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
* pop-tools                 2021.5.28          pyhd8ed1ab_0    conda-forge
* poppler                   22.04.0              h0733791_3    conda-forge
* poppler-data              0.4.11               hd8ed1ab_0    conda-forge
* portalocker               2.5.1            py39hf3d152e_0    conda-forge
* postgresql                14.5                 hfdbbde3_0    conda-forge
* pqdm                      0.1.0              pyhd8ed1ab_0    conda-forge
* prefect                   0.15.13            pyhd8ed1ab_0    conda-forge
* proj                      9.0.1                h93bde94_1    conda-forge
* prometheus_client         0.14.1             pyhd8ed1ab_0    conda-forge
* prompt-toolkit            3.0.31             pyha770c72_0    conda-forge
* properscoring             0.1                        py_0    conda-forge
* protobuf                  4.21.5           py39h5a03fae_1    conda-forge
* psutil                    5.9.2            py39hb9d737c_0    conda-forge
* pthread-stubs             0.4               h36c2ea0_1001    conda-forge
* ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
* pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
* py                        1.11.0             pyh6c4a22f_0    conda-forge
* pyarrow                   9.0.0           py39h6267de0_6_cpu    conda-forge
* pyasn1                    0.4.8                      py_0    conda-forge
* pyasn1-modules            0.2.7                      py_0    conda-forge
* pycamhd                   0.7.0                      py_0    conda-forge
* pycparser                 2.21               pyhd8ed1ab_0    conda-forge
* pyct                      0.4.6                      py_0    conda-forge
* pyct-core                 0.4.6                      py_0    conda-forge
* pycurl                    7.45.1           py39hd73adbb_2    conda-forge
* pydantic                  1.10.2           py39hb9d737c_0    conda-forge
* pydap                     3.3.0              pyhd8ed1ab_0    conda-forge
* pyerfa                    2.0.0.1          py39hd257fcd_2    conda-forge
* pygeos                    0.13             py39h68ae834_0    conda-forge
* pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
* pyjwt                     2.5.0              pyhd8ed1ab_0    conda-forge
* pykdtree                  1.3.5            py39hd257fcd_0    conda-forge
* pymbolic                  2022.1             pyhd8ed1ab_0    conda-forge
* pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
* pyorbital                 1.7.3              pyhd8ed1ab_0    conda-forge
* pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
* pyproj                    3.4.0            py39hdcf6798_0    conda-forge
* pyresample                1.25.1           py39h1832856_0    conda-forge
* pyrsistent                0.18.1           py39hb9d737c_1    conda-forge
* pyshp                     2.3.1              pyhd8ed1ab_0    conda-forge
* pysocks                   1.7.1              pyha2e5f31_6    conda-forge
* pyspectral                0.11.0             pyhd8ed1ab_0    conda-forge
* pystac                    1.6.1              pyhd8ed1ab_1    conda-forge
* pystac-client             0.5.0              pyhd8ed1ab_0    conda-forge
* pytest                    7.1.3            py39hf3d152e_0    conda-forge
* python                    3.9.13          h9a8a25e_0_cpython    conda-forge
* python-benedict           0.24.3             pyhd8ed1ab_0    conda-forge
* python-blosc              1.10.6           py39h1832856_0    conda-forge
* python-box                6.0.2            py39hb9d737c_3    conda-forge
* python-cmr                0.4.1              pyhd8ed1ab_0    conda-forge
* python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
* python-eccodes            1.4.2            py39hd257fcd_0    conda-forge
* python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
* python-fsutil             0.7.0              pyhd8ed1ab_0    conda-forge
* python-geotiepoints       1.4.1            py39hd257fcd_0    conda-forge
* python-gist               0.10.6             pyhd8ed1ab_0    conda-forge
* python-gnupg              0.4.9              pyhd8ed1ab_0    conda-forge
* python-graphviz           0.20.1             pyh22cad53_0    conda-forge
* python-json-logger        2.0.1              pyh9f0ad1d_0    conda-forge
* python-slugify            6.1.2              pyhd8ed1ab_0    conda-forge
* python_abi                3.9                      2_cp39    conda-forge
* pytools                   2022.1.12          pyhd8ed1ab_0    conda-forge
* pytz                      2022.2.1           pyhd8ed1ab_0    conda-forge
* pytzdata                  2020.1             pyh9f0ad1d_0    conda-forge
* pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
* pyviz_comms               2.2.1              pyhd8ed1ab_1    conda-forge
* pywavelets                1.3.0            py39hd257fcd_1    conda-forge
* pyyaml                    5.4.1            py39hb9d737c_3    conda-forge
* pyzmq                     24.0.0           py39headdf64_0    conda-forge
* rasterio                  1.3.2            py39h082961e_0    conda-forge
* re2                       2022.06.01           h27087fc_0    conda-forge
* readline                  8.1.2                h0f457ee_0    conda-forge
* rechunker                 0.5.0              pyhd8ed1ab_0    conda-forge
* requests                  2.28.1             pyhd8ed1ab_1    conda-forge
* requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
* rio-cogeo                 3.4.1              pyhd8ed1ab_0    conda-forge
* rioxarray                 0.12.2             pyhd8ed1ab_0    conda-forge
* rsa                       4.7.2              pyh44b312d_0    conda-forge
* rtree                     1.0.0            py39hb102c33_1    conda-forge
* ruamel.yaml               0.17.21          py39hb9d737c_1    conda-forge
* ruamel.yaml.clib          0.2.6            py39hb9d737c_1    conda-forge
* s2n                       1.0.10               h9b69904_0    conda-forge
* s3fs                      2022.8.2           pyhd8ed1ab_0    conda-forge
* s3transfer                0.6.0              pyhd8ed1ab_0    conda-forge
* satpy                     0.37.1             pyhd8ed1ab_0    conda-forge
* scikit-image              0.19.3           py39h1832856_1    conda-forge
* scikit-learn              1.1.2            py39he5e8d7e_0    conda-forge
* scipy                     1.9.1            py39h8ba3f38_0    conda-forge
* seaborn                   0.12.0               hd8ed1ab_0    conda-forge
* seaborn-base              0.12.0             pyhd8ed1ab_0    conda-forge
* send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
* setuptools                65.3.0             pyhd8ed1ab_1    conda-forge
* shapely                   1.8.4            py39h68ae834_0    conda-forge
* simpervisor               0.4                pyhd8ed1ab_0    conda-forge
* singleton-decorator       1.0.0                    pypi_0    pypi
* six                       1.16.0             pyh6c4a22f_0    conda-forge
* smmap                     3.0.5              pyh44b312d_0    conda-forge
* snappy                    1.1.9                hbd366e4_1    conda-forge
* sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
* snuggs                    1.4.7                      py_0    conda-forge
* sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
* soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
* sparse                    0.13.0             pyhd8ed1ab_0    conda-forge
* sqlalchemy                1.4.41           py39hb9d737c_0    conda-forge
* sqlite                    3.39.3               h4ff8645_0    conda-forge
* stack_data                0.5.0              pyhd8ed1ab_0    conda-forge
* stackstac                 0.4.3              pyhd8ed1ab_0    conda-forge
* starlette                 0.20.4             pyhd8ed1ab_0    conda-forge
* statsmodels               0.13.2           py39hd257fcd_0    conda-forge
* svt-av1                   1.2.1                h27087fc_0    conda-forge
* sysroot_linux-64          2.12                he073ed8_15    conda-forge
* tabulate                  0.8.10             pyhd8ed1ab_0    conda-forge
* tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
* terminado                 0.15.0           py39hf3d152e_0    conda-forge
* text-unidecode            1.3                        py_0    conda-forge
* threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
* tifffile                  2022.8.12          pyhd8ed1ab_0    conda-forge
* tiledb                    2.11.2               h1e4a385_0    conda-forge
* tiledb-py                 0.17.3           py39h47b85db_0    conda-forge
* timezonefinder            6.0.2              pyhd8ed1ab_0    conda-forge
* tinycss2                  1.1.1              pyhd8ed1ab_0    conda-forge
* tk                        8.6.12               h27826a3_0    conda-forge
* toml                      0.10.2             pyhd8ed1ab_0    conda-forge
* tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
* toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
* tornado                   6.1              py39hb9d737c_3    conda-forge
* tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
* traitlets                 5.4.0              pyhd8ed1ab_0    conda-forge
* traittypes                0.2.1              pyh9f0ad1d_2    conda-forge
* trollimage                1.18.3             pyhd8ed1ab_0    conda-forge
* trollsift                 0.4.0              pyhd8ed1ab_0    conda-forge
* typing-extensions         4.3.0                hd8ed1ab_0    conda-forge
* typing_extensions         4.3.0              pyha770c72_0    conda-forge
* tzcode                    2022c                h166bdaf_0    conda-forge
* tzdata                    2022c                h191b570_0    conda-forge
* ujson                     5.5.0            py39h5a03fae_0    conda-forge
* unicodedata2              14.0.0           py39hb9d737c_1    conda-forge
* unidecode                 1.3.4              pyhd8ed1ab_0    conda-forge
* urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
* uvicorn                   0.18.3           py39hf3d152e_0    conda-forge
* wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
* webencodings              0.5.1                      py_1    conda-forge
* webob                     1.8.7              pyhd8ed1ab_0    conda-forge
* websocket-client          1.4.1              pyhd8ed1ab_0    conda-forge
* wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
* widgetsnbextension        4.0.3              pyhd8ed1ab_0    conda-forge
* wrapt                     1.14.1           py39hb9d737c_0    conda-forge
* x264                      1!164.3095           h166bdaf_2    conda-forge
* x265                      3.5                  h924138e_3    conda-forge
* xarray                    2022.6.0           pyhd8ed1ab_1    conda-forge
* xarray-spatial            0.3.5              pyhd8ed1ab_0    conda-forge
* xarray_leaflet            0.1.16             pyhd8ed1ab_0    conda-forge
* xarrayutils               1.1.1              pyhd8ed1ab_0    conda-forge
* xcape                     0.1.4            py39h825f5bc_2    conda-forge
* xerces-c                  3.2.3                h55805fa_5    conda-forge
* xesmf                     0.6.3              pyhd8ed1ab_1    conda-forge
* xgcm                      0.8.0              pyhd8ed1ab_0    conda-forge
* xhistogram                0.3.2              pyhd8ed1ab_0    conda-forge
* xmip                      0.6.1rc0           pyhd8ed1ab_0    conda-forge
* xmitgcm                   0.5.2              pyhd8ed1ab_0    conda-forge
* xmltodict                 0.13.0             pyhd8ed1ab_0    conda-forge
* xorg-fixesproto           5.0               h7f98852_1002    conda-forge
* xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
* xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
* xorg-libice               1.0.10               h7f98852_0    conda-forge
* xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
* xorg-libx11               1.7.2                h7f98852_0    conda-forge
* xorg-libxau               1.0.9                h7f98852_0    conda-forge
* xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
* xorg-libxext              1.3.4                h7f98852_1    conda-forge
* xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
* xorg-libxi                1.7.10               h7f98852_0    conda-forge
* xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
* xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
* xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
* xorg-xproto               7.0.31            h7f98852_1007    conda-forge
* xpublish                  0.2.0              pyhd8ed1ab_0    conda-forge
* xrft                      1.0.0              pyhd8ed1ab_0    conda-forge
* xskillscore               0.0.24             pyhd8ed1ab_0    conda-forge
* xyzservices               2022.9.0           pyhd8ed1ab_0    conda-forge
* xz                        5.2.6                h166bdaf_0    conda-forge
* yaml                      0.2.5                h7f98852_2    conda-forge
* yarl                      1.7.2            py39hb9d737c_2    conda-forge
* zarr                      2.12.0             pyhd8ed1ab_0    conda-forge
* zeromq                    4.3.4                h9c3ff4c_1    conda-forge
* zfp                       1.0.0                h27087fc_1    conda-forge
* zict                      2.2.0              pyhd8ed1ab_0    conda-forge
* zipp                      3.8.1              pyhd8ed1ab_0    conda-forge
* zlib                      1.2.12               h166bdaf_3    conda-forge
* zlib-ng                   2.0.6                h166bdaf_0    conda-forge
* zstd                      1.5.2                h6239696_4    conda-forge

The environment on the dask workers needs to be the same as in your main session.

  • The default notebook-environment is updated in the terminal with conda update -n notebook --all

The key is that this updating of the notebook environment is not propagated to dask workers, have a close read of these docs Pangeo Cloud — Pangeo documentation

Hi @scottyhq and @martindurant, thanks. From what I read, the main session image, in jupyter hub, matches the image used on the dask cluster workers. I am guessing my mismatch comes from during troubleshooting of this error I updated packages in the main session environment. I previously thought that each time one logs into the deployments main jupyter-hub-session, the notebook environment is back to the default image, but I very possibly could be wrong. What would be the correct way for me to set the main session environment back to its default?

Best,

Currently when you log in to https://us-central1-b.gcp.pangeo.io you’ll see a message like "2022-10-17T19:56:12Z [Normal] Pulling image “pangeo/pangeo-notebook:2022.09.21” (you can also check the default image from a terminal with echo JUPYTER_IMAGE)

You can check the list of packages in that default environment here pangeo-docker-images/packages.txt at 2022.09.21 · pangeo-data/pangeo-docker-images · GitHub

Only files under /home/jovyan persist across sessions, so if you have installed things or changed configuration files there with pip or conda there try deleting those files (for example /home/jovyan/.local/lib/PYTHON/site-packages/).

Hi,

  • The notebook-environment has the same packages and version-numbers as in the list of default environment packages, using conda list. In addition it has singleton-decorator and jupyterlab-s3-browser, the only extra packages I can see.

  • I removed my repositories and removed all files under home/jovyan/. I log out and back in to the us-central gc-deployment to let the image be pulled, and upload the code again, and the error shows up on the first run.

  • It happens both when running on the GatewayCluster as displayed above, and when using the LocalCluster on the local cloud-server.

I wonder if you have advice for what I possibly am doing wrong and should check?

These warnings show during log-in. Could they help explain something about the error, that I am not picking up? But since the image is successfully pulled I am not sure this is relevant:

Event log
Server requested
2022-11-01T19:22:23Z [Warning] 0/39 nodes are available: 11 node(s) didn't match Pod's node affinity/selector, 25 node(s) had taint {k8s.dask.org_dedicated: worker}, that the pod didn't tolerate, 3 Insufficient memory.
2022-11-01T19:22:30Z [Normal] pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/pangeo-integration-te-3eea/zones/us-central1-b/instanceGroups/gke-pangeo-hubs-cluster-nb-huge-795c064d-grp 3->4 (max: 100)}]
2022-11-01T19:22:57Z [Warning] 0/40 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 11 node(s) didn't match Pod's node affinity/selector, 25 node(s) had taint {k8s.dask.org_dedicated: worker}, that the pod didn't tolerate, 3 Insufficient memory.
2022-11-01T19:23:07Z [Normal] Successfully assigned prod/jupyter-ofk123 to gke-pangeo-hubs-cluster-nb-huge-795c064d-d6ws
2022-11-01T19:23:11Z [Normal] Pulling image "busybox"
2022-11-01T19:23:12Z [Normal] Successfully pulled image "busybox" in 1.079886367s
2022-11-01T19:23:12Z [Normal] Created container volume-mount-ownership-fix
2022-11-01T19:23:12Z [Normal] Started container volume-mount-ownership-fix
2022-11-01T19:23:16Z [Normal] Pulling image "pangeo/pangeo-notebook:2022.09.21"
2022-11-01T19:24:15Z [Normal] Successfully pulled image "pangeo/pangeo-notebook:2022.09.21" in 59.343933498s
2022-11-01T19:24:15Z [Normal] Created container notebook
2022-11-01T19:24:15Z [Normal] Started container notebook

@ofk123 it seems possible we got a bit sidetracked with the versions mismatches. A way forward to troubleshoot your 'EntryPoint' object has no attribute _key message would be to share the data and code since you’re using the pangeo jupyterhub, that way others could possibly dig into it further.

  1. To share a zarr dataset you can use the object storage scratch space: Cloud Object Storage — Hub Service Guide

  2. To share a notebook two great options are:

Hi @scottyhq, thanks alot for the help in the office-hours and for the suggestions!
Here is a notebook showing the issue, (one where I use GatewayCluster is similar)
I am new to using pangeo and assume the error stems from either some kind of setting or configuration I have missed or accidentally changed, and/or the code is not set up correctly.
The packages and versionnumbers should be the same(at least from checking manually), and I have tried different alternatives of file-loading-code and cluster-type and some cluster-settings.
I will doublecheck the packages again (Edit: packages are equal, in addition it has singleton-decorator and jupyterlab-s3-browser). I appreciate any advice on how to possibly narrow down the issue more.

  • After the error-message occurs, running the cell dask.compute(run()) a second time always seems to work.
  • One thing that is hard about troubleshooting this is that sometimes the errormessage
    does not show up and it works perfectly. I test this with: running all cells in the notebook, changing the zarr-filename, then “Restart Kernel and Run All Cells”. The errormessage occurs after ~6 out of 10 of these runs.

Best,