ValueError: cannot reshape array of size 1 into shape (13968,)

Data
1

Hi, when I transform the NOAA netcdf dataset into zarr, I got ValueError: cannot reshape array of size 1 into shape (13968,) If I run the pipeline the first time, there is no error occurred. Everything is perfect. However, after I run the second time on the second day to update the zarr dataset (this NOAA dataset is updated daily), the reshape array error would happen.

How can I solve this error by using pangeo_forge_recipes? How to use pangeo_forge_recipes to append a new netcdf into zarr and update it daily?

My version is pangeo_forge_recipes==0.9.4. The pipeline runs in the argo workflow and uses Dask. Here is the resource of Dask.
- name: dask-num-workers value: 31 - name: dask-threads-per-worker value: 2 - name: dask-memory-per-worker value: "2GB"
And I use S3FileSystem to store zarr in s3 bucket.

Here is the details of the error:

    raise exception.with_traceback(traceback)
  File "/usr/local/lib/python3.10/dist-packages/dask/optimization.py", line 990, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/usr/local/lib/python3.10/dist-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "/usr/local/lib/python3.10/dist-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/usr/local/lib/python3.10/dist-packages/pangeo_forge_recipes/executors/dask.py", line 16, in wrapped
    return function(map_arg, config=config)
  File "/usr/local/lib/python3.10/dist-packages/pangeo_forge_recipes/recipes/xarray_zarr.py", line 652, in store_chunk
    zarr_array[zarr_region] = data
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 1391, in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 1486, in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 1790, in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 1862, in _set_selection
    self._chunk_setitems(lchunk_coords, lchunk_selection, chunk_values,
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 2084, in _chunk_setitems
    cdatas = {key: self._process_for_setitem(key, sel, val, fields=fields)
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 2084, in <dictcomp>
    cdatas = {key: self._process_for_setitem(key, sel, val, fields=fields)
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 2198, in _process_for_setitem
    chunk = self._decode_chunk(cdata)
  File "/usr/local/lib/python3.10/dist-packages/zarr/core.py", line 2258, in _decode_chunk
    chunk = chunk.reshape(expected_shape or self._chunks, order=self._order)
ValueError: cannot reshape array of size 1 into shape (13968,)
time="2023-08-22T00:34:56 UTC" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 1

Thank you very much for your help!

2

Hi Leo, apologies that you didn’t get a response here. I suggest you raise this as an issue on the pangeo-forge recipes repository. It would also help if you could distill the problem down to a Minimum Reproducible Example. Best of luck!