How about creating a pangeo-hpc github repo?

HPC
1

The idea (that originaly comes from @odakat from Ifremer) would be to share some configuration files, script or some documentation on how Pangeo has been deployed on various supercomputers.

Some thing I could personaly share from CNES Pangeo instance:

  • jupyterhub_config.py config file, obfuscated from any IP, but that still contains things like ProfileSpawner or BatchSpawner configuration,
  • Systemctl compliant juyterhub.service file,
  • Not sure about the complete ansible script that deploys the hub, maybe too tightly linked with our infrastructure,
  • Environment.yml files to deploy a complete Pangeo conda environement,
  • Environment.yml file that can be used to launch a notebook server with useful widgets.
  • Some docs on how to setup or configure this or that…
3 Likes
2

Great idea! This was discussed at the Seattle meeting, and Will Barnes got a start on it already!

We also tried doing it in the past. What is left of that is http://pangeo.io/setup_guides/hpc.html.

We used to host some specific configurations in the main pangeo repo, but we ended up removing them. Matt encouraged us to migrate the configurations to https://jobqueue.dask.org/en/latest/examples.html.

I personally would love to see hpc.pangeo.io, full of detailed configurations for deploying on HPC. The challenge of course is keeping if fresh in a rapidly changing software environment. If you and Will wanted to take this one, many people would benefit.

3

Great! I was not aware of this github repo.

I guess I now just have to fill it with our stuff! I’ll try to share some of the things I listed above soon.

1 Like
4

Sorry I’m a bit late to the party. I was out of the office last week and have not used discourse before!

As Ryan mentioned, David Shean and I created the pangeo-for-hpc repo at the Seattle meeting. However, as you can probably see, not much has been done there outside of the PR I opened during the meeting. This is a good home for configuration files, but not necessarily the most ideal place for hosting docs, i.e. we probably want something more than a README.

It would be great to migrate all HPC-related docs to something like hpc.pangeo.io. Could we also store the source for such a webpage in the same repo and deploy to that URL?

1 Like
5

That sounds like a great plan @wtbarnes!

We have a pangeo sphinx theme which makes it easy to deploy a pangeo-style sphinx site (example).

I recommend you use the existing pangeo-for-hpc repo for this. Once you have a github pages site working there, we can create the DNS entry for hpc.pangeo.io.

6

Great! Thanks for the links to the examples. I’ve just created an issue on the repo for this.