# Needing help and clarification about Jupyter on HPC Cheyenne

**URL:** <https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263>\
**Category:** HPC\
**Created:** [October 10, 2019, 3:28pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263 "2019-10-10T15:28:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![chiaral](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chiaral/32/430_2.png) [@chiaral](https://discourse.pangeo.io/u/chiaral)\
**Post date:** [October 10, 2019, 3:28pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/1 "2019-10-10T15:28:21Z")

</div>

I will start with a new post here - let me know if I should open a new issue on github.

Today I accessed Cheyenne and started setting up everything (I had already created a pangeo environment a year ago, so I updated it). Then I configured dask (I did not configured the password for Jupyter, I rather use the token) as explained [here](http://pangeo.io/setup_guides/hpc.html) . I successfully launched a qsub using the account number I have access to, and then launched jupyter with:

`jupyter lab --no-browser --ip=`hostname` --port=8888`

and then tried to forward it to my browser by opening another terminal page and using

`ssh -N -L 8888:r8i4n0:8888 username@cheyenne.ucar.edu`

and then it hangs for a bit and then I get

`channel 2: open failed: connect failed: Connection refused`

indeed if I try to open localhost:8888 I cannot connect to the notebook.

I am sure is some silly thing I am overlooking, But what is it?

Also my second question is, should I just try the “Deploy Option 2” with launch\_dask.sh? What are the differences/drawbacks/limitations of one option over the other?

Thanks!

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [October 10, 2019, 4:24pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/2 "2019-10-10T16:24:50Z")

</div>

I think the problem is that you are ssh’ing to the wrong node. `r8i4n0` the hostname of the compute node where your notebook is running. It will vary depending on where the scheduler places your job. There are a couple of ways to get the hostname. If you already have an interactive terminal running, you just type `hostname` on the command line. These lines from the docs your referenced give you a command to run to get the full ssh command printed out.

```auto
echo "ssh -N -L 8888:`hostname`:8888 $USER@cheyenne.ucar.edu"
ssh -N -L 8888:r8i4n0:8888 username@cheyenne.ucar.edu

```

The fact that you are using the same hostname as the docs tells me that you probably just copied that line exactly, rather than trying to get the actual host name where you job is running.

---

<div class="post-metadata">

**Author:** ![chiaral](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chiaral/32/430_2.png) [@chiaral](https://discourse.pangeo.io/u/chiaral)\
**Post date:** [October 10, 2019, 4:56pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/3 "2019-10-10T16:56:33Z")

</div>

Thanks. That worked.

---

<div class="post-metadata">

**Author:** ![jhamman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhamman/32/34_2.png) [@jhamman](https://discourse.pangeo.io/u/jhamman)\
**Post date:** [October 10, 2019, 5:08pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/4 "2019-10-10T17:08:03Z")

</div>

@chiaral- I think you know about this already but, just in case, and for others who come along here, we now have a JupyterHub setup on Cheyenne which means you **don’t** need to use the ssh-tunneling options anymore: [https://jupyterhub.ucar.edu](https://jupyterhub.ucar.edu)

@jukent posted this nice walk through describing how to access this hub:

> [@JupyterHub access on Cheyenne](https://discourse.pangeo.io/t/jupyterhub-access-on-cheyenne/253/1):
>
> JupyterHub access on Cheyenne Step 1: Navigate to [https://jupyterhub.ucar.edu/](https://jupyterhub.ucar.edu/). Step 2: Launch your server, select from: - Cheyenne Supercomputer (i.e., the batch nodes) - Casper DAV (Analysis machine) [image] You may have to click “Launch Server” after selecting your machine: [image] Step 3: Authenticate with either your yubikey or your DUO mobile app Step 4: Specify: - Queue (reservation or Slurm QoS) - Project account number - Wall time Step 5: Click “Spawn.” What to …

---

<div class="post-metadata">

**Author:** ![chiaral](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chiaral/32/430_2.png) [@chiaral](https://discourse.pangeo.io/u/chiaral)\
**Post date:** [October 10, 2019, 5:16pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/5 "2019-10-10T17:16:33Z")

</div>

🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉  
This is so much easier! I wasn’t sure if it was ready for accessing it, but it seems to have worked perfectly and sees my local environments as well. This is AWESOME!

Thanks so much to every one at NCAR! I am getting really excited to work on the hackathon!

---

<div class="post-metadata">

**Author:** ![jukent](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jukent/32/297_2.png) [@jukent](https://discourse.pangeo.io/u/jukent)\
**Post date:** [October 10, 2019, 11:24pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/6 "2019-10-10T23:24:25Z")

</div>

I’m glad it worked for you!

---

<div class="post-metadata">

**Author:** ![chiaral](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chiaral/32/430_2.png) [@chiaral](https://discourse.pangeo.io/u/chiaral)\
**Post date:** [October 25, 2019, 7:44pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/7 "2019-10-25T19:44:04Z")

</div>

Hello @jukent & @jhamman

This message is a bit OT compared to the previous topic, but still related to my quest of figuring things out on Cheyenne/Jupyter.

I created a package, I successfully (or so it looks) installed it in my conda environment in my home directory on Cheyenne. This package has some fortran code that gets compiled during the installation.  
I run  
`python setup.py build_ext`  
`pip install -e .`

everything works, the `py.test` in fact are successful.

When I call the fortran subroutine from a terminal

`python -c 'from package import fortran_call'`

I get no error whatsoever. It seems to work. (And again, all the tests written in the package that run through the many subroutines are successful).

If I open a jupyter notebook through the `https://jupyterhub.ucar.edu` (and I choose the same conda environment), the same call fails and I get this error:

`ImportError: libifcoremt.so.5: cannot open shared object file: No such file or directory`

It seems to be an issue with Jupyterhub because it seems like I can load it correctly within the terminal.

Any suggestions of what this could be? And how I can solve it?

I don’t know if it’s related, but during the build, I get this Warning:  
`UserWarning: LDFLAGS is used as is, not appended to flags already defined by numpy.distutils!`

I didn’t get that before on the other machines where I successfully used and installed this package.

UPDATE:  
I found this very old [issue](https://github.com/jupyterhub/jupyterhub/issues/1048) that sounds exactly what I am facing.  
Since I cannot do the changes suggested in the comments, I tried to set a `LD_LIBRARY_PATH` for the current environment.

`import os os.environ['LD_LIBRARY_PATH'] = '/ncar/opt/slurm/latest//lib:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/18.0.5:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2018u4/compilers_and_libraries/linux/lib/intel64:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib'`

which I got from my shell environment.

But I still get the same error. So I am not sure if this workaround cannot work, or this issue I linked to is not what would solve mine.

---

<div class="post-metadata">

**Author:** ![chiaral](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chiaral/32/430_2.png) [@chiaral](https://discourse.pangeo.io/u/chiaral)\
**Post date:** [October 27, 2019, 2:03pm UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/8 "2019-10-27T14:03:32Z")

</div>

@rabernat figured out the workaround!  
I leave it here for reference. These commands need to be run in a cell before the lauch of dask with `ncar_jobqueue`

1. I set up LD\_LIBRARY\_PATH (get details from your home directory with `env|grep LD`

`LD_LIBRARY_PATH = '/ncar/opt/slurm/latest//lib:/glade/u/apps/ch/opt/mpt_fmods/2.19/intel/18.0.5:/glade/u/apps/ch/opt/mpt/2.19/lib:/glade/u/apps/opt/intel/2018u4/compilers_and_libraries/linux/lib/intel64:/glade/u/apps/ch/os/usr/lib64:/glade/u/apps/ch/os/usr/lib:/glade/u/apps/ch/os/lib64:/glade/u/apps/ch/os/lib' `

1. set a command like:  
`env_command = f"export LD_LIBRARY_PATH='{LD_LIBRARY_PATH}'"`

2. pass it to your dask workers when you use `ncar_jobqueu`:  
`from ncar_jobqueue import NCARCluster cluster = NCARCluster(project='XXXXYYZZ', env_extra=[env_command])`

I still cannot run the package from the jupyter notebook, but the workers will see it.

---

<div class="post-metadata">

**Author:** ![jhamman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhamman/32/34_2.png) [@jhamman](https://discourse.pangeo.io/u/jhamman)\
**Post date:** [October 28, 2019, 1:43am UTC](https://discourse.pangeo.io/t/needing-help-and-clarification-about-jupyter-on-hpc-cheyenne/263/9 "2019-10-28T01:43:00Z")

</div>

Hi @chiaral - I’m glad you’ve found a workaround for now. I think I’ll just leave a few a notes that be helpful pointers in the future:

1. Make sure you compile your C/Fortran code on the system you will be running on. Cheyenne and the DAV systems are not 100% compatible ([https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/code-development-support/compiling-code](https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne/code-development-support/compiling-code))
2. Play close attention to environment variables like `LD_LIBRARY_PATH`, `LDFLAGS`, `PATH`, etc. When launching Jupyter or Dask jobs, its not a guarantee that you’ll end up with the same environment.
3. Think a bit about how dynamically linked libraries will play with the rest of your python setup. Presumably you’re going to want to share this with others so getting the linking working seamlessly will be important down the road.