Another rechunking algorithm

[TomNicholas]

@mullenkamp also see this discussion for further context

Why learn rust? A pangeo perspective

I think we’re actually very close to this already using Cubed. You can already run Cubed today on a single machine utilizing all cores (through threads or processes), and it internally creates a query plan that it optimizes. By default, it would keep intermediate arrays in RAM; unless this would result in running out of RAM. In which case it’d borrow from Cubed’s approach, and write intermediate data to disk. Doing the rechunk in-memory is the only part that’s missing. A while ago I raised …