# Data **URL:** https://discourse.pangeo.io/c/data/13.md [Latest](https://discourse.pangeo.io/latest.md) · [Categories](https://discourse.pangeo.io/categories.md) · [Tags](https://discourse.pangeo.io/tags.md) --- ## [About the Data category](https://discourse.pangeo.io/t/about-the-data-category/63)
**Author:** [@rabernat](https://discourse.pangeo.io/u/rabernat)\ **Replies:** 0
Data file formats (netCDF, HDF5, Zarr, Tiledb, COG, etc.), metadata, and data catalogs. --- ## [Feedback wanted: recipe-based tool for building aligned radar/satellite/model/obs datasets (CONUS)](https://discourse.pangeo.io/t/feedback-wanted-recipe-based-tool-for-building-aligned-radar-satellite-model-obs-datasets-conus/5832)
**Author:** [@yuchen\_wei3667](https://discourse.pangeo.io/u/yuchen_wei3667)\ **Replies:** 0\ **Last updated:** [October 6, 2026, 1:43am UTC](https://discourse.pangeo.io/t/feedback-wanted-recipe-based-tool-for-building-aligned-radar-satellite-model-obs-datasets-conus/5832 "2026-10-06T01:43:58Z")
Hi all, I’m designing an open-source Python tool for building analysis-ready datasets that combine US radar, satellite, NWP, surface observations, and severe-weather reports, and I’d really value this community’s input … --- ## [HTTP API for Statistics from Zarr stores?](https://discourse.pangeo.io/t/http-api-for-statistics-from-zarr-stores/5713)
**Author:** [@aimeeb](https://discourse.pangeo.io/u/aimeeb)\ **Replies:** 16\ **Last updated:** [September 30, 2026, 2:35am UTC](https://discourse.pangeo.io/t/http-api-for-statistics-from-zarr-stores/5713 "2026-09-30T02:35:29Z")
:waving\_hand: My team on the NASA Office of Data Science and Informatics (ODSI) Data Systems Evolution (DSE) program is considering building an HTTP API for statistics generation from Zarr stores and we could use the co… --- ## [Decode GeoTIFF to GPU memory](https://discourse.pangeo.io/t/decode-geotiff-to-gpu-memory/5214)
**Author:** [@weiji14](https://discourse.pangeo.io/u/weiji14)\ **Replies:** 17\ **Last updated:** [September 25, 2026, 4:18am UTC](https://discourse.pangeo.io/t/decode-geotiff-to-gpu-memory/5214 "2026-09-25T04:18:07Z")
Sharing this blog post on speeding up Cloud-optimized GeoTIFF (COG) reads using a CUDA GPU library called nvTIFF that I’ve been playing around with for the past month. Hopefully it’ll be useful for folks struggling to us… --- ## [Copernicus Marine Level-4 products](https://discourse.pangeo.io/t/copernicus-marine-level-4-products/5824)
**Author:** [@Maha\_Alj](https://discourse.pangeo.io/u/Maha_Alj)\ **Replies:** 1\ **Last updated:** [September 24, 2026, 8:03pm UTC](https://discourse.pangeo.io/t/copernicus-marine-level-4-products/5824 "2026-09-24T20:03:07Z")
Hi, I am studying mesoscale eddies. I found two Copernicus Marine Level-4 products: 1.Global Ocean Gridded L4 Sea Surface Heights and Derived Variables Reprocessed – Copernicus Climate Change Service, with a spatial r… --- ## [Public Virtual Zarr of the NASA (NEX-GDDP-CMIP6) Dataset](https://discourse.pangeo.io/t/public-virtual-zarr-of-the-nasa-nex-gddp-cmip6-dataset/5802)
**Author:** [@norlandrhagen](https://discourse.pangeo.io/u/norlandrhagen)\ **Replies:** 3\ **Last updated:** [August 24, 2026, 8:52am UTC](https://discourse.pangeo.io/t/public-virtual-zarr-of-the-nasa-nex-gddp-cmip6-dataset/5802 "2026-08-24T08:52:30Z")
In a recent project, I needed some downscaled CMIP6 data. I created a Virtual Zarr store of the NEX-GDDP-CMIP6 dataset. I wanted to share it if anyone has use of this dataset and doesn’t want to go through the work of re… --- ## [Problems downloading CMIP6 historical simulations from PANGEO gallery](https://discourse.pangeo.io/t/problems-downloading-cmip6-historical-simulations-from-pangeo-gallery/5575)
**Author:** [@nrchow](https://discourse.pangeo.io/u/nrchow)\ **Replies:** 5\ **Last updated:** [August 3, 2026, 12:38pm UTC](https://discourse.pangeo.io/t/problems-downloading-cmip6-historical-simulations-from-pangeo-gallery/5575 "2026-08-03T12:38:59Z")
Hi good folks. So I am using this library: GitHub - dr-ko/python\_pangeo\_cmip6: Python scripts to download CMIP6 data from Pangeo Gallery · GitHub, to download historical simulations, particularly large ensembles. For som… --- ## [Virtual GRIB ingestion via Gribberish](https://discourse.pangeo.io/t/virtual-grib-ingestion-via-gribberish/5785)
**Author:** [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\ **Replies:** 0\ **Last updated:** [July 15, 2026, 10:30pm UTC](https://discourse.pangeo.io/t/virtual-grib-ingestion-via-gribberish/5785 "2026-07-15T22:30:57Z")
Ever wonder why GRIB is the way it is? Read our new blog post on virtually ingesting GRIB data into Icechunk using Gribberish and VirtualiZarr! --- ## [Best Practice for Machine Learning with Huge Datasets](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635)
**Author:** [@yonghan](https://discourse.pangeo.io/u/yonghan)\ **Replies:** 2\ **Last updated:** [June 27, 2026, 2:19am UTC](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635 "2026-06-27T02:19:57Z")
Hey folks, I have been playing around with numerical weather forecasting using deep learning for a while. I mainly use the hourly reanalysis ERA5 datasets. The size of ERA5 datasets is huge, especially if you prefer a h… --- ## [How to see all the public repos in arraylake programmatically?](https://discourse.pangeo.io/t/how-to-see-all-the-public-repos-in-arraylake-programmatically/5764)
**Author:** [@ahuang11](https://discourse.pangeo.io/u/ahuang11)\ **Replies:** 2\ **Last updated:** [June 26, 2026, 6:43pm UTC](https://discourse.pangeo.io/t/how-to-see-all-the-public-repos-in-arraylake-programmatically/5764 "2026-06-26T18:43:37Z")
In Old format, no problem!: Cloud-optimizing the GOES-16 archive as Virtual Zarr | Earthmover it shows how to easily access: import arraylake as al import xarray as xr client = al.Client() client.login() repo = client… --- ## [Favorite way to go from netCDF (&xarray) to torch/TF/Jax et al](https://discourse.pangeo.io/t/favorite-way-to-go-from-netcdf-xarray-to-torch-tf-jax-et-al/2663)
**Author:** [@ThomasMGeo](https://discourse.pangeo.io/u/ThomasMGeo)\ **Replies:** 8\ **Last updated:** [June 26, 2026, 6:04pm UTC](https://discourse.pangeo.io/t/favorite-way-to-go-from-netcdf-xarray-to-torch-tf-jax-et-al/2663 "2026-06-26T18:04:56Z")
Hello! I am researching best practices to load spatial, geoscience data (generally netCDF/Grib via xarray) to various ML packages. Two things kept coming up: xbatcher project: GitHub - pangeo-data/xbatcher: Batch gen… --- ## [Suitable platform for large climate risk modeling project](https://discourse.pangeo.io/t/suitable-platform-for-large-climate-risk-modeling-project/5754)
**Author:** [@gisborne](https://discourse.pangeo.io/u/gisborne)\ **Replies:** 4\ **Last updated:** [June 26, 2026, 4:16pm UTC](https://discourse.pangeo.io/t/suitable-platform-for-large-climate-risk-modeling-project/5754 "2026-06-26T16:16:06Z")
Hello, I am trying to determine the suitability of Postgres, ClickHouse or the likes of Zaar for a significant climate risk modelling project. We are batch processing a large (500 million) collection of geographical … --- ## [STAC catalogs for insitu data](https://discourse.pangeo.io/t/stac-catalogs-for-insitu-data/3328)
**Author:** [@keewis](https://discourse.pangeo.io/u/keewis)\ **Replies:** 8\ **Last updated:** [June 23, 2026, 9:02pm UTC](https://discourse.pangeo.io/t/stac-catalogs-for-insitu-data/3328 "2026-06-23T21:02:53Z")
Hi all, I’m currently trying to create a STAC catalog for insitu data, for which I couldn’t find any examples I could copy. The kind of data I’m working with (insitu data is pretty heterogenous, so I might be looking at… --- ## [What are the best practices to store data in zarr and how scalable is zarr?](https://discourse.pangeo.io/t/what-are-the-best-practices-to-store-data-in-zarr-and-how-scalable-is-zarr/5573)
**Author:** [@Kaboom\_Official](https://discourse.pangeo.io/u/Kaboom_Official)\ **Replies:** 4\ **Last updated:** [June 3, 2026, 2:49pm UTC](https://discourse.pangeo.io/t/what-are-the-best-practices-to-store-data-in-zarr-and-how-scalable-is-zarr/5573 "2026-06-03T14:49:30Z")
I know Zarr performs better, especially for cloud workflows, but I have a few practical concerns. I already have large archives of NetCDF/HDF files, and converting everything to Zarr will itself take a significant amoun… --- ## [TLM Markers for Sentinel-2 (Baseline \< 05.12) and Their Equivalents for TIFFs (S-1) and NetCDFs (S-3, S-5)](https://discourse.pangeo.io/t/tlm-markers-for-sentinel-2-baseline-05-12-and-their-equivalents-for-tiffs-s-1-and-netcdfs-s-3-s-5/5579)
**Author:** [@MathewNWSH](https://discourse.pangeo.io/u/MathewNWSH)\ **Replies:** 3\ **Last updated:** [May 8, 2026, 12:49pm UTC](https://discourse.pangeo.io/t/tlm-markers-for-sentinel-2-baseline-05-12-and-their-equivalents-for-tiffs-s-1-and-netcdfs-s-3-s-5/5579 "2026-05-08T12:49:40Z")
Hi, I’m developing a solution similar to the Copernicus Browser, offering the flexibility to define on-the-fly data processing “recipes” and share the results via WMS (or OGC API - Maps). Everything is based on GDAL VRT… --- ## [Insert a small zarr into a big zarr in time order](https://discourse.pangeo.io/t/insert-a-small-zarr-into-a-big-zarr-in-time-order/4526)
**Author:** [@Jiaming\_Tan](https://discourse.pangeo.io/u/Jiaming_Tan)\ **Replies:** 4\ **Last updated:** [April 28, 2026, 2:15pm UTC](https://discourse.pangeo.io/t/insert-a-small-zarr-into-a-big-zarr-in-time-order/4526 "2026-04-28T14:15:15Z")
I have a combined zarr named bigzarrno5.zarr，it has dimensions valid\_time, longitude, latitude, pressure\_level，and it contain day 1,2,3,4,6 and the valid\_time format is datatime64 like ‘2020-11-01T00:00:00.000000000’,eac… --- ## [Data pipelining and cataloging best practices (? using intake-xarray to transform and combine data metadata?)](https://discourse.pangeo.io/t/data-pipelining-and-cataloging-best-practices-using-intake-xarray-to-transform-and-combine-data-metadata/5550)
**Author:** [@NickHodgskin](https://discourse.pangeo.io/u/NickHodgskin)\ **Replies:** 6\ **Last updated:** [April 8, 2026, 12:34am UTC](https://discourse.pangeo.io/t/data-pipelining-and-cataloging-best-practices-using-intake-xarray-to-transform-and-combine-data-metadata/5550 "2026-04-08T00:34:55Z")
Heyall - I would like to streamline the ingestion of various datasets. I have seen a lot of conversation around intake-xarray, so have started looking into that. One of the problems, however, with the NetCDF data is that… --- ## [Join us: Advanced Analysis with Sentinel GeoZarr (20 April) - EOPF Explorer + openEO](https://discourse.pangeo.io/t/join-us-advanced-analysis-with-sentinel-geozarr-20-april-eopf-explorer-openeo/5556)
**Author:** [@Sabrina](https://discourse.pangeo.io/u/Sabrina)\ **Replies:** 0\ **Last updated:** [April 7, 2026, 9:06am UTC](https://discourse.pangeo.io/t/join-us-advanced-analysis-with-sentinel-geozarr-20-april-eopf-explorer-openeo/5556 "2026-04-07T09:06:07Z")
Hello! Thank you again to everyone from the Pangeo community who joined the first EOPF Explorer webinar. Following the strong response from that session, we are organising a hands-on workshop that lets you try out analy… --- ## [Managing Parallel Dask Graphs](https://discourse.pangeo.io/t/managing-parallel-dask-graphs/5547)
**Author:** [@Tman](https://discourse.pangeo.io/u/Tman)\ **Replies:** 0\ **Last updated:** [March 27, 2026, 3:42pm UTC](https://discourse.pangeo.io/t/managing-parallel-dask-graphs/5547 "2026-03-27T15:42:56Z")
Hi all! I’ve been working with @darothen and the Brightband team on ExtremeWeatherBench (EWB) for a bit over a year now. We’ve done a lot of great work and I’ve been able to really dial in parts of the code seeing years… --- ## [Community Notebook Competition - Sentinel data in Zarr](https://discourse.pangeo.io/t/community-notebook-competition-sentinel-data-in-zarr/5510)
**Author:** [@Sabrina](https://discourse.pangeo.io/u/Sabrina)\ **Replies:** 4\ **Last updated:** [February 21, 2026, 10:45am UTC](https://discourse.pangeo.io/t/community-notebook-competition-sentinel-data-in-zarr/5510 "2026-02-21T10:45:42Z")
Dear PANGEO community, ESA is planning to distribute Sentinel data in Zarr format in the future and wants to give users a chance to try it out first. Here’s how you can participate: Community Notebook Competition Wri… --- ## [Comparing odc.stac.load and stackstac for raster composite workflow](https://discourse.pangeo.io/t/comparing-odc-stac-load-and-stackstac-for-raster-composite-workflow/4097)
**Author:** [@rabernat](https://discourse.pangeo.io/u/rabernat)\ **Replies:** 35\ **Last updated:** [January 29, 2026, 10:11am UTC](https://discourse.pangeo.io/t/comparing-odc-stac-load-and-stackstac-for-raster-composite-workflow/4097 "2026-01-29T10:11:34Z")
Overview I am a n00b when it comes to traditional GIS raster processing. I’m trying to better understand the available tools in this space. I’m doing this through what seems to be a very typical workflow: compute a mon… --- ## [Accessing nested HDF5 file from http via kerchunk](https://discourse.pangeo.io/t/accessing-nested-hdf5-file-from-http-via-kerchunk/3432)
**Author:** [@weiji14](https://discourse.pangeo.io/u/weiji14)\ **Replies:** 11\ **Last updated:** [January 13, 2026, 8:03pm UTC](https://discourse.pangeo.io/t/accessing-nested-hdf5-file-from-http-via-kerchunk/3432 "2026-01-13T20:03:25Z")
Hi there, I’m trying to figure out if there’s a way to access some HDF5 files stored on what seems like a HTTP server directly with xarray. The dataset is at chabud-team/chabud-extra at main, and looks to be on a Git LF… --- ## [Very slow zarr appends from non cf compliant hdf files](https://discourse.pangeo.io/t/very-slow-zarr-appends-from-non-cf-compliant-hdf-files/5484)
**Author:** [@Kaboom\_Official](https://discourse.pangeo.io/u/Kaboom_Official)\ **Replies:** 0\ **Last updated:** [January 12, 2026, 9:50pm UTC](https://discourse.pangeo.io/t/very-slow-zarr-appends-from-non-cf-compliant-hdf-files/5484 "2026-01-12T21:50:28Z")
Hey I am working with gpm dpr ku band data and data when read through xarray creates phony dims.The structure of hdf files is like this Dataset: AlgorithmRuntimeInfo, shape: (1,), dtype: |S1175 Group: FS Group: FS/CS… --- ## [Cloud-optimized access to Sentinel-2 JPEG2000](https://discourse.pangeo.io/t/cloud-optimized-access-to-sentinel-2-jpeg2000/5072)
**Author:** [@j.anger](https://discourse.pangeo.io/u/j.anger)\ **Replies:** 12\ **Last updated:** [December 22, 2025, 4:08pm UTC](https://discourse.pangeo.io/t/cloud-optimized-access-to-sentinel-2-jpeg2000/5072 "2025-12-22T16:08:39Z")
Dear Pangeo community, As you may know, Sentinel-2 rasters are distributed as JPEG2000. If you try to crop a S2 raster remotely, you’ll find that a lot of requests will be issued, and thus the format can be quickly cla… --- ## [How to convert hdf files to a single zarr store? What are the best practices?](https://discourse.pangeo.io/t/how-to-convert-hdf-files-to-a-single-zarr-store-what-are-the-best-practices/5464)
**Author:** [@Kaboom\_Official](https://discourse.pangeo.io/u/Kaboom_Official)\ **Replies:** 4\ **Last updated:** [December 12, 2025, 6:15am UTC](https://discourse.pangeo.io/t/how-to-convert-hdf-files-to-a-single-zarr-store-what-are-the-best-practices/5464 "2025-12-12T06:15:32Z")
I am converting my half hourly data in hdf files to zarr using xarray. I read the file and then using to\_zarr I keep appending in zarr store.Is there any way it can take any less time ,each file initialy was taking 10 s… --- ## [How to organize data from polar satellites and irregular grids in datacube format?](https://discourse.pangeo.io/t/how-to-organize-data-from-polar-satellites-and-irregular-grids-in-datacube-format/5457)
**Author:** [@Kaboom\_Official](https://discourse.pangeo.io/u/Kaboom_Official)\ **Replies:** 9\ **Last updated:** [November 30, 2025, 7:20am UTC](https://discourse.pangeo.io/t/how-to-organize-data-from-polar-satellites-and-irregular-grids-in-datacube-format/5457 "2025-11-30T07:20:33Z")
I have been exploring the use of VirtualiZarr and Icechunk to manage large satellite datasets efficiently using reference-based storage. I successfully created a virtual dataset for geostationary satellite imagery (INSAT… --- ## [Creating bindings for Python/Julia/R into a Rust Zarr implementation (zarrs)](https://discourse.pangeo.io/t/creating-bindings-for-python-julia-r-into-a-rust-zarr-implementation-zarrs/5446)
**Author:** [@NickHodgskin](https://discourse.pangeo.io/u/NickHodgskin)\ **Replies:** 2\ **Last updated:** [November 18, 2025, 12:07am UTC](https://discourse.pangeo.io/t/creating-bindings-for-python-julia-r-into-a-rust-zarr-implementation-zarrs/5446 "2025-11-18T00:07:01Z")
EDIT: I realised after creating this that it probably isn’t the best channel - I have created #Zarr \> Bindings from Zarrs to other languages (Julia, R etc.) in the Zulip where people are chiming in. Let’s continue discus… --- ## [Advice on creating a photometric RGB COG from an xarray source](https://discourse.pangeo.io/t/advice-on-creating-a-photometric-rgb-cog-from-an-xarray-source/5434)
**Author:** [@jalder](https://discourse.pangeo.io/u/jalder)\ **Replies:** 1\ **Last updated:** [October 31, 2025, 6:53am UTC](https://discourse.pangeo.io/t/advice-on-creating-a-photometric-rgb-cog-from-an-xarray-source/5434 "2025-10-31T06:53:17Z")
Hello all, I am considering using COGs as a data visualization tool in a web application and could use some advice. We’re currently using the ESRI JavaScript mapping API, which allows for COG layers, but only photometric… --- ## [Problems with COG images downloaded from the Planetary Computer API](https://discourse.pangeo.io/t/problems-with-cog-images-downloaded-from-the-planetary-computer-api/3094)
**Author:** [@seda](https://discourse.pangeo.io/u/seda)\ **Replies:** 7\ **Last updated:** [October 2, 2025, 3:57pm UTC](https://discourse.pangeo.io/t/problems-with-cog-images-downloaded-from-the-planetary-computer-api/3094 "2025-10-02T15:57:25Z")
Hi all, I have downloaded several Sentinel 2 images using the Microsoft Planetary Computer Python API and have come across many transparent/corrupted pixels in these images. These pixels appear mainly when downloading t… --- ## [Metadata rich hydrodynamic/atmospheric data following SGRID conventions - tooling and existing datasets](https://discourse.pangeo.io/t/metadata-rich-hydrodynamic-atmospheric-data-following-sgrid-conventions-tooling-and-existing-datasets/5403)
**Author:** [@NickHodgskin](https://discourse.pangeo.io/u/NickHodgskin)\ **Replies:** 2\ **Last updated:** [September 21, 2025, 3:00am UTC](https://discourse.pangeo.io/t/metadata-rich-hydrodynamic-atmospheric-data-following-sgrid-conventions-tooling-and-existing-datasets/5403 "2025-09-21T03:00:17Z")
We’re currently doing major version (v4) development of Parcels and we want to rework our ingestion of structured grid, hydrodynamic input data. Throughout the history of Parcels with struggled with users coming with dif… [Next page](https://discourse.pangeo.io/c/data/13.md?page=1)