Best way to access CMIP6 data Cloud or HPC (in UK)?

[rabernat]

Hi @lhoupert! Welcome to Pangeo!

@lhoupert

As a UK researcher, I am wondering if someone from the PANGEO community knows if PANGEO is deployed on the UK “super-data-cluser” Jasmin.

Yes, we are aware about this and quite excited about it! We have helped them get some parts of the stack running, see e.g.

Storing CMIP6 data on JASMIN's object store

Hello, This is my first post to this community. I’m working on a pilot project with JASMIN, a data analysis facility for the NERC community in the UK. It’s a hybrid of traditional batch system and OpenStack cloud. We’ve recently set up a system so that groups make their own instance of Pangeo in OpenStack projects on our system. We are also experimenting with storing CMIP6 data on an object store which is also part of JASMIN. This is S3 compatible. I’ve looked at https://pangeo.io/data.ht…

@lhoupert

If yes, do you know if the whole PANGEO data catalogue is available? Correct me if I am wrong but my understanding of running PANGEO on HPC is that all the data have to be on the HPC?

The data you see at https://catalog.pangeo.io/ is available anywhere on the internet. It mostly points to data in google cloud. We need to do some work to clarify where the data are located and where they can be accessed.

@agstephens and collaborators have a very exciting project where they are building their own cloud-like object store for CMIP6 at CEDA, designed specifically for use with Jasmin. This is not released yet, but it sounds like it will be the ideal solution for you. Perhaps Ag can say more. I’m sure they’ll be excited to have some users to test out the system.