# Latest

**URL:** https://discourse.pangeo.io/latest.md?page=1

[Latest](https://discourse.pangeo.io/latest.md) · [Categories](https://discourse.pangeo.io/categories.md) · [Tags](https://discourse.pangeo.io/tags.md)

**Page:** 2

---

## [Best Practice for Machine Learning with Huge Datasets](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635)

<div class="topic-metadata">

**Author:** [@yonghan](https://discourse.pangeo.io/u/yonghan)\
**Replies:** 2\
**Last updated:** [June 27, 2026, 2:19am UTC](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635 "2026-06-27T02:19:57Z")

</div>

Hey folks, I have been playing around with numerical weather forecasting using deep learning for a while. I mainly use the hourly reanalysis ERA5 datasets. The size of ERA5 datasets is huge, especially if you prefer a h…

---

## [Hiring a Senior Software Developer at OPeNDAP](https://discourse.pangeo.io/t/hiring-a-senior-software-developer-at-opendap/5765)

<div class="topic-metadata">

**Author:** [@arika\_v](https://discourse.pangeo.io/u/arika_v)\
**Replies:** 0\
**Last updated:** [June 26, 2026, 11:02pm UTC](https://discourse.pangeo.io/t/hiring-a-senior-software-developer-at-opendap/5765 "2026-06-26T23:02:18Z")

</div>

We’re seeking a Senior Software Developer who is passionate about working on open source solutions for scientific data access. OPeNDAP is hiring a Software Developer to help develop systems that work within NASA’s Earth…

---

## [How to see all the public repos in arraylake programmatically?](https://discourse.pangeo.io/t/how-to-see-all-the-public-repos-in-arraylake-programmatically/5764)

<div class="topic-metadata">

**Author:** [@ahuang11](https://discourse.pangeo.io/u/ahuang11)\
**Replies:** 2\
**Last updated:** [June 26, 2026, 6:43pm UTC](https://discourse.pangeo.io/t/how-to-see-all-the-public-repos-in-arraylake-programmatically/5764 "2026-06-26T18:43:37Z")

</div>

In Old format, no problem!: Cloud-optimizing the GOES-16 archive as Virtual Zarr | Earthmover it shows how to easily access: import arraylake as al import xarray as xr client = al.Client() client.login() repo = client…

---

## [Pangeo Showcase: "Cloud Native Data Loaders for Machine Learning Using Zarr and Xarray"](https://discourse.pangeo.io/t/pangeo-showcase-cloud-native-data-loaders-for-machine-learning-using-zarr-and-xarray/4110)

<div class="topic-metadata">

**Author:** [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Replies:** 7\
**Last updated:** [June 26, 2026, 6:08pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-cloud-native-data-loaders-for-machine-learning-using-zarr-and-xarray/4110 "2026-06-26T18:08:13Z")

</div>

Title: “Cloud Native Data Loaders for Machine Learning Using Zarr and Xarray” Invited Speaker: Joe Hamman (ORCID:0000-0001-7479-8439) When: Wednesday March 20, 12PM EDT Where: Launch Meeting - Zoom Abstr…

---

## [Favorite way to go from netCDF (&xarray) to torch/TF/Jax et al](https://discourse.pangeo.io/t/favorite-way-to-go-from-netcdf-xarray-to-torch-tf-jax-et-al/2663)

<div class="topic-metadata">

**Author:** [@ThomasMGeo](https://discourse.pangeo.io/u/ThomasMGeo)\
**Replies:** 8\
**Last updated:** [June 26, 2026, 6:04pm UTC](https://discourse.pangeo.io/t/favorite-way-to-go-from-netcdf-xarray-to-torch-tf-jax-et-al/2663 "2026-06-26T18:04:56Z")

</div>

Hello! I am researching best practices to load spatial, geoscience data (generally netCDF/Grib via xarray) to various ML packages. Two things kept coming up: xbatcher project: GitHub - pangeo-data/xbatcher: Batch gen…

---

## [Suitable platform for large climate risk modeling project](https://discourse.pangeo.io/t/suitable-platform-for-large-climate-risk-modeling-project/5754)

<div class="topic-metadata">

**Author:** [@gisborne](https://discourse.pangeo.io/u/gisborne)\
**Replies:** 4\
**Last updated:** [June 26, 2026, 4:16pm UTC](https://discourse.pangeo.io/t/suitable-platform-for-large-climate-risk-modeling-project/5754 "2026-06-26T16:16:06Z")

</div>

Hello, I am trying to determine the suitability of Postgres, ClickHouse or the likes of Zaar for a significant climate risk modelling project. We are batch processing a large (500 million) collection of geographical …

---

## [Pangeo at SciPy](https://discourse.pangeo.io/t/pangeo-at-scipy/5705)

<div class="topic-metadata">

**Author:** [@maxrjones](https://discourse.pangeo.io/u/maxrjones)\
**Replies:** 4\
**Last updated:** [June 26, 2026, 2:26pm UTC](https://discourse.pangeo.io/t/pangeo-at-scipy/5705 "2026-06-26T14:26:45Z")

</div>

Is anyone planning to attend SciPy this summer and interested in representing Pangeo during the SciPy 2026 Tools Plenary?

---

## [Using zstd frames and jump tables to extract partial grib files](https://discourse.pangeo.io/t/using-zstd-frames-and-jump-tables-to-extract-partial-grib-files/5756)

<div class="topic-metadata">

**Author:** [@colddata](https://discourse.pangeo.io/u/colddata)\
**Replies:** 0\
**Last updated:** [June 25, 2026, 2:07pm UTC](https://discourse.pangeo.io/t/using-zstd-frames-and-jump-tables-to-extract-partial-grib-files/5756 "2026-06-25T14:07:40Z")

</div>

I have been contributing to an open source AGPL project GitHub - HuskHoard/HuskHoard: An open-source (AGPLv3), Rust-based automated data-tiering and archiving engine for Linux. Transparently offload cold data to S3, HDD…

---

## [HTTP API for Statistics from Zarr stores?](https://discourse.pangeo.io/t/http-api-for-statistics-from-zarr-stores/5713)

<div class="topic-metadata">

**Author:** [@aimeeb](https://discourse.pangeo.io/u/aimeeb)\
**Replies:** 13\
**Last updated:** [June 24, 2026, 7:02pm UTC](https://discourse.pangeo.io/t/http-api-for-statistics-from-zarr-stores/5713 "2026-06-24T19:02:53Z")

</div>

:waving\_hand: My team on the NASA Office of Data Science and Informatics (ODSI) Data Systems Evolution (DSE) program is considering building an HTTP API for statistics generation from Zarr stores and we could use the co…

---

## [STAC catalogs for insitu data](https://discourse.pangeo.io/t/stac-catalogs-for-insitu-data/3328)

<div class="topic-metadata">

**Author:** [@keewis](https://discourse.pangeo.io/u/keewis)\
**Replies:** 8\
**Last updated:** [June 23, 2026, 9:02pm UTC](https://discourse.pangeo.io/t/stac-catalogs-for-insitu-data/3328 "2026-06-23T21:02:53Z")

</div>

Hi all, I’m currently trying to create a STAC catalog for insitu data, for which I couldn’t find any examples I could copy. The kind of data I’m working with (insitu data is pretty heterogenous, so I might be looking at…

---

## [AquaScope: one Python schema over 18 water-data collectors (12 agencies), plus hydrology analysis. Looking for feedback on the xarray/interop story](https://discourse.pangeo.io/t/aquascope-one-python-schema-over-18-water-data-collectors-12-agencies-plus-hydrology-analysis-looking-for-feedback-on-the-xarray-interop-story/5753)

<div class="topic-metadata">

**Author:** [@Rekin226](https://discourse.pangeo.io/u/Rekin226)\
**Replies:** 0\
**Last updated:** [June 16, 2026, 2:47am UTC](https://discourse.pangeo.io/t/aquascope-one-python-schema-over-18-water-data-collectors-12-agencies-plus-hydrology-analysis-looking-for-feedback-on-the-xarray-interop-story/5753 "2026-06-16T02:47:11Z")

</div>

Hi all, I’m a postdoc working on water resources, and I’ve been building AquaScope, an MIT-licensed Python toolkit that does two things: it pulls from 18 collectors across 12 water-data agencies (USGS, FAO AQUASTAT, FAO …

---

## [Join us at OceanHackWeek 2026, the Canadian edition](https://discourse.pangeo.io/t/join-us-at-oceanhackweek-2026-the-canadian-edition/5745)

<div class="topic-metadata">

**Author:** [@abkfenris](https://discourse.pangeo.io/u/abkfenris)\
**Replies:** 0\
**Last updated:** [June 10, 2026, 4:39pm UTC](https://discourse.pangeo.io/t/join-us-at-oceanhackweek-2026-the-canadian-edition/5745 "2026-06-10T16:39:03Z")

</div>

Applications are open for OceanHackWeek 2026! This year we’re headed North to Bamfield Marine Sciences Centre on Vancouver Island. OceanHackWeek is a unique collaborative learning experience that helps participants gain…

---

## [ERA5-Land climate pattern detection pipeline: requesting methodology critique](https://discourse.pangeo.io/t/era5-land-climate-pattern-detection-pipeline-requesting-methodology-critique/5741)

<div class="topic-metadata">

**Author:** [@Puravida](https://discourse.pangeo.io/u/Puravida)\
**Replies:** 0\
**Last updated:** [June 8, 2026, 2:35pm UTC](https://discourse.pangeo.io/t/era5-land-climate-pattern-detection-pipeline-requesting-methodology-critique/5741 "2026-06-08T14:35:07Z")

</div>

ERA5-Land climate pattern detection pipeline: requesting methodology critique I have built a pattern detection pipeline analyzing 75 years of ERA5-Land reanalysis data across four regions: Western Europe, Central US, Co…

---

## [Icechunk to be Adopted by the National Weather Service!](https://discourse.pangeo.io/t/icechunk-to-be-adopted-by-the-national-weather-service/5724)

<div class="topic-metadata">

**Author:** [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Replies:** 0\
**Last updated:** [June 4, 2026, 10:01pm UTC](https://discourse.pangeo.io/t/icechunk-to-be-adopted-by-the-national-weather-service/5724 "2026-06-04T22:01:06Z")

</div>

Icechunk is coming to weather.gov! Thank you everyone for ideating, beta testing, using, evangelising, and contributing - to have Zarr at the centre of the most important production system in the US is a huge milestone …

---

## [GOES-16 as a single virtual Zarr store](https://discourse.pangeo.io/t/goes-16-as-a-single-virtual-zarr-store/5721)

<div class="topic-metadata">

**Author:** [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Replies:** 2\
**Last updated:** [June 4, 2026, 2:17am UTC](https://discourse.pangeo.io/t/goes-16-as-a-single-virtual-zarr-store/5721 "2026-06-04T02:17:54Z")

</div>

I stress-tested VirtualiZarr and Icechunk by ingesting all of the GOES-16 Cloud and Moisture Imagery Product archive into a single store. As far as I know this is the biggest virtual Zarr store ever (by number of chunks…

---

## [What are the best practices to store data in zarr and how scalable is zarr?](https://discourse.pangeo.io/t/what-are-the-best-practices-to-store-data-in-zarr-and-how-scalable-is-zarr/5573)

<div class="topic-metadata">

**Author:** [@Kaboom\_Official](https://discourse.pangeo.io/u/Kaboom_Official)\
**Replies:** 4\
**Last updated:** [June 3, 2026, 2:49pm UTC](https://discourse.pangeo.io/t/what-are-the-best-practices-to-store-data-in-zarr-and-how-scalable-is-zarr/5573 "2026-06-03T14:49:30Z")

</div>

I know Zarr performs better, especially for cloud workflows, but I have a few practical concerns. I already have large archives of NetCDF/HDF files, and converting everything to Zarr will itself take a significant amoun…

---

## [Sharing pyGOTM: a Python/Numba implementation of GOTM for reproducible ocean/lake column modeling](https://discourse.pangeo.io/t/sharing-pygotm-a-python-numba-implementation-of-gotm-for-reproducible-ocean-lake-column-modeling/5712)

<div class="topic-metadata">

**Author:** [@Nick\_Bobolea](https://discourse.pangeo.io/u/Nick_Bobolea)\
**Replies:** 0\
**Last updated:** [May 29, 2026, 5:19pm UTC](https://discourse.pangeo.io/t/sharing-pygotm-a-python-numba-implementation-of-gotm-for-reproducible-ocean-lake-column-modeling/5712 "2026-05-29T17:19:24Z")

</div>

Hello Pangeo community, I would like to share an open-source project I have been working on called pyGOTM: https://github.com/alinbobolea/pygotm pyGOTM is a Python/Numba reimplementation of the General Ocean Turbulenc…

---

## [A Gridlook (a Vue/TypeScript 3D viewer for Zarr) fork for flexible Zarr & Icechunk viz](https://discourse.pangeo.io/t/a-gridlook-a-vue-typescript-3d-viewer-for-zarr-fork-for-flexible-zarr-icechunk-viz/5711)

<div class="topic-metadata">

**Author:** [@Elizabeth\_Holmes\_NOA](https://discourse.pangeo.io/u/Elizabeth_Holmes_NOA)\
**Replies:** 0\
**Last updated:** [May 29, 2026, 4:46pm UTC](https://discourse.pangeo.io/t/a-gridlook-a-vue-typescript-3d-viewer-for-zarr-fork-for-flexible-zarr-icechunk-viz/5711 "2026-05-29T16:46:50Z")

</div>

GridLook is a WebGL-based viewer for cloud-hosted Zarr datasets developed by the Max-Planck-Institute for Meteorology (MPI-M) and the German Climate Computing Center (DKRZ). What makes GridLook unique and interesting to …

---

## [Hiring Forest Science Specialist – ICP Forests / Climate-Smart Forestry](https://discourse.pangeo.io/t/hiring-forest-science-specialist-icp-forests-climate-smart-forestry/5710)

<div class="topic-metadata">

**Author:** [@Mondo](https://discourse.pangeo.io/u/Mondo)\
**Replies:** 0\
**Last updated:** [May 28, 2026, 4:09pm UTC](https://discourse.pangeo.io/t/hiring-forest-science-specialist-icp-forests-climate-smart-forestry/5710 "2026-05-28T16:09:15Z")

</div>

We are Mondosway OÜ delivering ForestSentinel — an AI-powered forest disturbance detection platform under EU grant G-01-2026 with the European Forest Institute (EFI). DURATION: 10 months (June 2026 – March 2027) COMM…

---

## [Spatial Data Science across Language 2026 in Jena, Germany Sept 16+ 17](https://discourse.pangeo.io/t/spatial-data-science-across-language-2026-in-jena-germany-sept-16-17/5545)

<div class="topic-metadata">

**Author:** [@felixcremer](https://discourse.pangeo.io/u/felixcremer)\
**Replies:** 1\
**Last updated:** [May 21, 2026, 2:33pm UTC](https://discourse.pangeo.io/t/spatial-data-science-across-language-2026-in-jena-germany-sept-16-17/5545 "2026-05-21T14:33:38Z")

</div>

We are organizing this years Spatial Data Science across languages workshop in Jena Germany from September 16 and 17 and a developing Day at 18th September. This is the fourth installment of these workshops with the aim…

---

## [Offre de thèse (3 ans) en océanographie côtière - Toulon, France](https://discourse.pangeo.io/t/offre-de-these-3-ans-en-oceanographie-cotiere-toulon-france/5693)

<div class="topic-metadata">

**Author:** [@Leaef26](https://discourse.pangeo.io/u/Leaef26)\
**Replies:** 0\
**Last updated:** [May 17, 2026, 10:24am UTC](https://discourse.pangeo.io/t/offre-de-these-3-ans-en-oceanographie-cotiere-toulon-france/5693 "2026-05-17T10:24:46Z")

</div>

:megaphone: Offre de thèse financée (3 ans) en océanographie côtière – Toulon, France :france: Nous recrutons une doctorante / un doctorant au MIO (Mediterranean Institute of Oceanography, Université de Toulon) sur le s…

---

## [Spring Showcase 2026 close-out: Lightning talks! (May 13, 2026 at 12 PM US Eastern)](https://discourse.pangeo.io/t/spring-showcase-2026-close-out-lightning-talks-may-13-2026-at-12-pm-us-eastern/5587)

<div class="topic-metadata">

**Author:** [@chuckwondo](https://discourse.pangeo.io/u/chuckwondo)\
**Replies:** 1\
**Last updated:** [May 15, 2026, 10:42pm UTC](https://discourse.pangeo.io/t/spring-showcase-2026-close-out-lightning-talks-may-13-2026-at-12-pm-us-eastern/5587 "2026-05-15T22:42:27Z")

</div>

Title: “Spring Showcase 2026 close-out: Lightning talks!” When: Wednesday, May 13, 2026 at 12 PM US Eastern Where: Launch Meeting - Zoom Details: :high\_voltage: Join us next Wednesday, May 13 at 12 PM US Eastern for…

---

## [Dawnloading ICON D2 data june 2021](https://discourse.pangeo.io/t/dawnloading-icon-d2-data-june-2021/5536)

<div class="topic-metadata">

**Author:** [@Nishant\_Pawar](https://discourse.pangeo.io/u/Nishant_Pawar)\
**Replies:** 1\
**Last updated:** [May 15, 2026, 7:07pm UTC](https://discourse.pangeo.io/t/dawnloading-icon-d2-data-june-2021/5536 "2026-05-15T19:07:07Z")

</div>

im looking for icon d2 data for the month june 2021, if anyone have leads share your thoughts.

---

## [Building Earth Data Workflows with Agents](https://discourse.pangeo.io/t/building-earth-data-workflows-with-agents/5586)

<div class="topic-metadata">

**Author:** [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Replies:** 5\
**Last updated:** [May 12, 2026, 2:53pm UTC](https://discourse.pangeo.io/t/building-earth-data-workflows-with-agents/5586 "2026-05-12T14:53:02Z")

</div>

How best to use Agents to help build Earth Data workflows? I’ve been just feeding the agents a bunch of existing workflows (e.g. Notebooks) and letting the agents use them as source material to construct new workflows. …

---

## [TLM Markers for Sentinel-2 (Baseline \< 05.12) and Their Equivalents for TIFFs (S-1) and NetCDFs (S-3, S-5)](https://discourse.pangeo.io/t/tlm-markers-for-sentinel-2-baseline-05-12-and-their-equivalents-for-tiffs-s-1-and-netcdfs-s-3-s-5/5579)

<div class="topic-metadata">

**Author:** [@MathewNWSH](https://discourse.pangeo.io/u/MathewNWSH)\
**Replies:** 3\
**Last updated:** [May 8, 2026, 12:49pm UTC](https://discourse.pangeo.io/t/tlm-markers-for-sentinel-2-baseline-05-12-and-their-equivalents-for-tiffs-s-1-and-netcdfs-s-3-s-5/5579 "2026-05-08T12:49:40Z")

</div>

Hi, I’m developing a solution similar to the Copernicus Browser, offering the flexibility to define on-the-fly data processing “recipes” and share the results via WMS (or OGC API - Maps). Everything is based on GDAL VRT…

---

## [Pangeo Showcase: Lightning talks (May 13, 2026 at 12 PM US Eastern) -- Sign up now!](https://discourse.pangeo.io/t/pangeo-showcase-lightning-talks-may-13-2026-at-12-pm-us-eastern-sign-up-now/5584)

<div class="topic-metadata">

**Author:** [@chuckwondo](https://discourse.pangeo.io/u/chuckwondo)\
**Replies:** 0\
**Last updated:** [May 4, 2026, 2:32pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-lightning-talks-may-13-2026-at-12-pm-us-eastern-sign-up-now/5584 "2026-05-04T14:32:30Z")

</div>

The speaker for our final showcase scheduled for May 6, 2026 at 4 PM US Eastern is unfortunately no longer able to present, but our meeting will still proceed! In the meantime, get your gears turning for ideas for light…

---

## [Zarr-layer: a typescript library for visualizing zarr & icechunk in the browser](https://discourse.pangeo.io/t/zarr-layer-a-typescript-library-for-visualizing-zarr-icechunk-in-the-browser/5574)

<div class="topic-metadata">

**Author:** [@norlandrhagen](https://discourse.pangeo.io/u/norlandrhagen)\
**Replies:** 0\
**Last updated:** [April 28, 2026, 6:43pm UTC](https://discourse.pangeo.io/t/zarr-layer-a-typescript-library-for-visualizing-zarr-icechunk-in-the-browser/5574 "2026-04-28T18:43:02Z")

</div>

Hey folks! Just wanted to share a library my co-worker Shane Loeffler has been working on. At CarbonPlan we’ve been using @carbonplan-maps for a while now for visualizing zarr directly in the browser. It works great, bu…

---

## [Insert a small zarr into a big zarr in time order](https://discourse.pangeo.io/t/insert-a-small-zarr-into-a-big-zarr-in-time-order/4526)

<div class="topic-metadata">

**Author:** [@Jiaming\_Tan](https://discourse.pangeo.io/u/Jiaming_Tan)\
**Replies:** 4\
**Last updated:** [April 28, 2026, 2:15pm UTC](https://discourse.pangeo.io/t/insert-a-small-zarr-into-a-big-zarr-in-time-order/4526 "2026-04-28T14:15:15Z")

</div>

I have a combined zarr named bigzarrno5.zarr，it has dimensions valid\_time, longitude, latitude, pressure\_level，and it contain day 1,2,3,4,6 and the valid\_time format is datatime64 like ‘2020-11-01T00:00:00.000000000’,eac…

---

## [Job: Executive Director at CarbonPlan](https://discourse.pangeo.io/t/job-executive-director-at-carbonplan/5572)

<div class="topic-metadata">

**Author:** [@norlandrhagen](https://discourse.pangeo.io/u/norlandrhagen)\
**Replies:** 0\
**Last updated:** [April 27, 2026, 8:25pm UTC](https://discourse.pangeo.io/t/job-executive-director-at-carbonplan/5572 "2026-04-27T20:25:55Z")

</div>

About CarbonPlan CarbonPlan is a nonprofit research organization using science and data to accelerate climate action. CarbonPlan’s highly interdisciplinary team analyzes the design and implementation of climate progra…

---

## [Optimizing data load/compute when using STAC API](https://discourse.pangeo.io/t/optimizing-data-load-compute-when-using-stac-api/5563)

<div class="topic-metadata">

**Author:** [@Valentina\_Premier](https://discourse.pangeo.io/u/Valentina_Premier)\
**Replies:** 1\
**Last updated:** [April 20, 2026, 4:00pm UTC](https://discourse.pangeo.io/t/optimizing-data-load-compute-when-using-stac-api/5563 "2026-04-20T16:00:47Z")

</div>

Hi all, I am wondering if I can use dask to speed up the computation/data load over a relatively large area (Euregio region). I have a single time step and I want to upload Sentinel-2 data though the CDSE STAC API. I a…

[Previous page](https://discourse.pangeo.io/latest.md)

[Next page](https://discourse.pangeo.io/latest.md?page=2)