Best way to access CMIP6 data Cloud or HPC (in UK)?

Hi everybody,

Firsts, thanks for inviting me to join the pangeo community! As I mentioned in my sign up form, I am more and more interested in analysing the CMIP6 outputs and I think the PANGEO tools and community would be perfect for this.

I am particularly interested in evaluating how the (mis)-representation of the North Atlantic Current in climate models (e.g. pathways, mean, variability) are related to biases in the stratification of the subpolar gyre and/or artic sea-ice extent. I am planning a pilot study as part of a 3-month internship project mid-2021. Then, I am hoping using this preliminary results to apply for fellowship or grant application.

I have a few questions about the UK geoscience research computing landscape and PANGEO:

  1. As a UK researcher, I am wondering if someone from the PANGEO community knows if PANGEO is deployed on the UK “super-data-cluser” Jasmin.
  2. If yes, do you know if the whole PANGEO data catalogue is available? Correct me if I am wrong but my understanding of running PANGEO on HPC is that all the data have to be on the HPC?
1 Like

Hi @lhoupert! Welcome to Pangeo!

Yes, we are aware about this and quite excited about it! We have helped them get some parts of the stack running, see e.g.

The data you see at https://catalog.pangeo.io/ is available anywhere on the internet. It mostly points to data in google cloud. We need to do some work to clarify where the data are located and where they can be accessed.

@agstephens and collaborators have a very exciting project where they are building their own cloud-like object store for CMIP6 at CEDA, designed specifically for use with Jasmin. This is not released yet, but it sounds like it will be the ideal solution for you. Perhaps Ag can say more. I’m sure they’ll be excited to have some users to test out the system.

@lhoupert We are indeed building a system on JASMIN (UK) for writing CMIP6 to Zarr in our own object store. The code being used to manage this is:

The object store endpoint is only currently visible from within the JASMIN firewall (but this is due to change soon). An instance of PANGEO and/or our JASMIN Notebook Service can access the CMIP6 Zarr store.

We have an example Notebook here:

We are planning to build an intake-esm catalogue next week that can talk to the Zarr store.

2 Likes

That’s great news! I am going to investigate this in more details!

1 Like