Best Practice for Machine Learning with Huge Datasets

[emfdavid]

Cross posting here

Favorite way to go from netCDF (&xarray) to torch/TF/Jax et al

The IO is fast now but there is still no standard tooling for chunked zarr compatible split/shuffle aware batching. I put some time (and tokens) into building GitHub - emfdavid/insitubatch: Train in place on n-dimensional cloud tensors. · GitHub this week. It is one Data Set abstraction compatible with Torch, Jax and TF. It is batteries included with tunable caching, pre-fechting and max-inflight. No heavy worker processes makes it much more efficient - lean memory and cpu (fetch once!) It …