Accessing LLC2160 data from a cluster without internet access on its compute nodes

Science
1

I would like to access the high-res LLC data, as described in this post. The problem is that the HPC cluster at my university does not (for security reasons) allow internet access on any of its compute nodes. So, this snippet of code does not work:

from xmitgcm import llcreader
model = llcreader.ECCOPortalLLC2160Model()
ds = model.get_dataset(varnames=['Eta'], type='latlon')

What are my options here? [Apart from trying to run it on the login nodes, which are connected to the outside world but will be much slower than the compute nodes.]

I realize the data is also available elsewhere but they are in compressed format.

1 Like
2

Hi all,
I wanted to follow up on this. Does anybody have suggestions on what are the options for accessing the LLC2160 data from an HPC cluster without internet access on its compute nodes? Thanks and have a wonderful Thanksgiving break!

3

@saatvik - it is impossible to access data over the internet from a computer that does not have internet access. There is probably some way to download data to the cluster, perhaps from the head node, so that is probably your best bet.

These are not separate. The data from the ECCO data portal are the same exact data that llcreader downloads when you create a llcreader.ECCOPortalLLC2160Model().

You could create a local copy of the LLC2160 archive by manually downloading the files from the ECCO data portal and organizing them in the same way on the local filesystem. This is described in the doc here - llcreader — xmitgcm v0.5.2-9-g63ba751 documentation

Also an example here:

1 Like
4

Thanks! I will explore your suggestion of trying to recreate a local copy of the filesystem as per the link you provided.

1 Like
5

The challenge you will face is that downloading the whole files will be very slow from the ECCO data portal.

6

Right, I agree. But perhaps I can start by downloading small amounts of data to get a sense of how far I can go.