Hi Pangeo team,
Is writing to netcdf from Zarr sometimes slow? I am working with CM2.6 1pct CO2 sea level data on my cluster, which I’ve rechunked into time chunks using the rechunker package. I would like to process it by detrending and removing the seasonal cycle. Since these operations each multiply the number of tasks, I’d like to save the data as netcdf between processing steps (for example, save the detrended timeseries and then save the detrended + seasonal cycle-removed timeseries).
For some reason on my HPC cluster this has been quite slow… even just resaving the rechunked Zarr dataset as NetCDF won’t complete within 2 days. This is despite the fact that I am able to run the analogous script for processing and saving sea level anomalies from the CM2.6 picontrol dataset as netcdf well within 6h time limit allowed. So I am rather confused as to what the issue could be.