Best way to access CMIP6 data Cloud or HPC (in UK)?

[lhoupert]

Hi everybody,

Firsts, thanks for inviting me to join the pangeo community! As I mentioned in my sign up form, I am more and more interested in analysing the CMIP6 outputs and I think the PANGEO tools and community would be perfect for this.

I am particularly interested in evaluating how the (mis)-representation of the North Atlantic Current in climate models (e.g. pathways, mean, variability) are related to biases in the stratification of the subpolar gyre and/or artic sea-ice extent. I am planning a pilot study as part of a 3-month internship project mid-2021. Then, I am hoping using this preliminary results to apply for fellowship or grant application.

I have a few questions about the UK geoscience research computing landscape and PANGEO:

1. As a UK researcher, I am wondering if someone from the PANGEO community knows if PANGEO is deployed on the UK “super-data-cluser” Jasmin.
2. If yes, do you know if the whole PANGEO data catalogue is available? Correct me if I am wrong but my understanding of running PANGEO on HPC is that all the data have to be on the HPC?

[rabernat]

Hi @lhoupert! Welcome to Pangeo!

@lhoupert

As a UK researcher, I am wondering if someone from the PANGEO community knows if PANGEO is deployed on the UK “super-data-cluser” Jasmin.

Yes, we are aware about this and quite excited about it! We have helped them get some parts of the stack running, see e.g.

Storing CMIP6 data on JASMIN's object store

Hello, This is my first post to this community. I’m working on a pilot project with JASMIN, a data analysis facility for the NERC community in the UK. It’s a hybrid of traditional batch system and OpenStack cloud. We’ve recently set up a system so that groups make their own instance of Pangeo in OpenStack projects on our system. We are also experimenting with storing CMIP6 data on an object store which is also part of JASMIN. This is S3 compatible. I’ve looked at https://pangeo.io/data.ht…

@lhoupert

If yes, do you know if the whole PANGEO data catalogue is available? Correct me if I am wrong but my understanding of running PANGEO on HPC is that all the data have to be on the HPC?

The data you see at https://catalog.pangeo.io/ is available anywhere on the internet. It mostly points to data in google cloud. We need to do some work to clarify where the data are located and where they can be accessed.

@agstephens and collaborators have a very exciting project where they are building their own cloud-like object store for CMIP6 at CEDA, designed specifically for use with Jasmin. This is not released yet, but it sounds like it will be the ideal solution for you. Perhaps Ag can say more. I’m sure they’ll be excited to have some users to test out the system.

[agstephens]

@lhoupert We are indeed building a system on JASMIN (UK) for writing CMIP6 to Zarr in our own object store. The code being used to manage this is:

cedadev/cmip6-object-store

The object store endpoint is only currently visible from within the JASMIN firewall (but this is due to change soon). An instance of PANGEO and/or our JASMIN Notebook Service can access the CMIP6 Zarr store.

We have an example Notebook here:

cedadev/cmip6-object-store/blob/master/notebooks/cmip6-zarr-jasmin.ipynb

We are planning to build an intake-esm catalogue next week that can talk to the Zarr store.

[lhoupert]

That’s great news! I am going to investigate this in more details!