How about creating a pangeo-hpc github repo?

The idea (that originaly comes from @odakat from Ifremer) would be to share some configuration files, script or some documentation on how Pangeo has been deployed on various supercomputers.

Some thing I could personaly share from CNES Pangeo instance:

  • jupyterhub_config.py config file, obfuscated from any IP, but that still contains things like ProfileSpawner or BatchSpawner configuration,
  • Systemctl compliant juyterhub.service file,
  • Not sure about the complete ansible script that deploys the hub, maybe too tightly linked with our infrastructure,
  • Environment.yml files to deploy a complete Pangeo conda environement,
  • Environment.yml file that can be used to launch a notebook server with useful widgets.
  • Some docs on how to setup or configure this or that…
3 Likes

Great idea! This was discussed at the Seattle meeting, and Will Barnes got a start on it already!

We also tried doing it in the past. What is left of that is http://pangeo.io/setup_guides/hpc.html.

We used to host some specific configurations in the main pangeo repo, but we ended up removing them. Matt encouraged us to migrate the configurations to https://jobqueue.dask.org/en/latest/examples.html.

I personally would love to see hpc.pangeo.io, full of detailed configurations for deploying on HPC. The challenge of course is keeping if fresh in a rapidly changing software environment. If you and Will wanted to take this one, many people would benefit.

Great! I was not aware of this github repo.

I guess I now just have to fill it with our stuff! I’ll try to share some of the things I listed above soon.

1 Like

Sorry I’m a bit late to the party. I was out of the office last week and have not used discourse before!

As Ryan mentioned, David Shean and I created the pangeo-for-hpc repo at the Seattle meeting. However, as you can probably see, not much has been done there outside of the PR I opened during the meeting. This is a good home for configuration files, but not necessarily the most ideal place for hosting docs, i.e. we probably want something more than a README.

It would be great to migrate all HPC-related docs to something like hpc.pangeo.io. Could we also store the source for such a webpage in the same repo and deploy to that URL?

1 Like

That sounds like a great plan @wtbarnes!

We have a pangeo sphinx theme which makes it easy to deploy a pangeo-style sphinx site (example).

I recommend you use the existing pangeo-for-hpc repo for this. Once you have a github pages site working there, we can create the DNS entry for hpc.pangeo.io.

Great! Thanks for the links to the examples. I’ve just created an issue on the repo for this.