Today I have been getting “spawn failed” errors on login like this:
Your server is starting up.
You will be redirected automatically when it's ready for you.
100% Complete
Spawn failed: Server at http://10.8.26.2:8888/user/tjcrone/ didn't respond in 30 seconds
Event log
Server requested
2022-09-08T10:09:07Z [Warning] 0/26 nodes are available: 1 Insufficient memory, 20 node(s) had taint {k8s.dask.org_dedicated: worker}, that the pod didn't tolerate, 5 node(s) didn't match Pod's node affinity/selector.
2022-09-08T10:09:15Z [Normal] pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/pangeo-integration-te-3eea/zones/us-central1-b/instanceGroups/gke-pangeo-hubs-cluster-nb-small-c97e04c1-grp 1->2 (max: 100)}]
2022-09-08T10:09:53Z [Warning] 0/27 nodes are available: 1 Insufficient memory, 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 20 node(s) had taint {k8s.dask.org_dedicated: worker}, that the pod didn't tolerate, 5 node(s) didn't match Pod's node affinity/selector.
2022-09-08T10:10:14Z [Normal] Successfully assigned prod/jupyter-tjcrone to gke-pangeo-hubs-cluster-nb-small-c97e04c1-z9hq
2022-09-08T10:10:18Z [Normal] Cancelling deletion of Pod prod/jupyter-tjcrone
2022-09-08T10:10:19Z [Normal] Pulling image "busybox"
2022-09-08T10:10:25Z [Normal] Successfully pulled image "busybox" in 5.448708427s
2022-09-08T10:10:25Z [Normal] Created container volume-mount-ownership-fix
2022-09-08T10:10:25Z [Normal] Started container volume-mount-ownership-fix
2022-09-08T10:10:30Z [Normal] Pulling image "pangeo/pangeo-notebook:2022.06.02"
2022-09-08T10:12:24Z [Normal] Successfully pulled image "pangeo/pangeo-notebook:2022.06.02" in 1m54.052795656s
2022-09-08T10:12:24Z [Normal] Created container notebook
2022-09-08T10:12:24Z [Normal] Started container notebook
Spawn failed: Server at http://10.8.26.2:8888/user/tjcrone/ didn't respond in 30 seconds
One possibility is that I may have caused this problem by putting a line in my .bash_profile file that changes the shell on login. If anyone has access to the home directories I think this can be fixed by deleting my .bash_profile file. I would greatly appreciate it if anyone can help. Thank you!
I believe that this one has been resolved now. My theory for what happened is that when you edited .bash_profile to use zsh, activating zsh causes the path to be clobbered, and the command that jupyterhub uses to launch the singleuser session is no longer available.
I opened a PR in our docs to describe the pitfalls of this, but not sure how to get ZSH working, if anybody know how please do chime in there!
Thank you very much for helping with this @choldgraf. Removing the .bash_profile file worked great. I greatly appreciate your assistance on this.
The correct way to install Zsh is probably to install it into the notebook image with apt and then change the default shell for the login user using chsh. Or maybe to let the user just start zsh when needed. I didn’t do it this way because I didn’t think there would be time to test a new image. Whether this method will still break the JupyterHub spawner or other functionality is not clear to me, but it might. Maybe there is no way to run Zsh, but I will say that it was working great for me when I started it from the command line. It only broke things when I needed to respawn.
I haven’t had time to find the JupyterHub code that calls the start script, the one with start "$@", but maybe these scripts could better define paths, or make sure to run in bash rather than zsh. Oh and the problem might actually be in the .zshrc file just to make things more complicated. Anyway in the meantime I will no longer test this sort of stuff in the production JuypterHub. I’m sorry for making this mistake. I may test this in my own JupyterHub and if I find a good fix I will offer a PR.