# Best Practice for Machine Learning with Huge Datasets

**URL:** <https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635>\
**Category:** Data\
**Tags:** machine-learning\
**Created:** [October 26, 2024, 3:05am UTC](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635 "2024-10-26T03:05:35Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![emfdavid](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/emfdavid/32/2598_2.png) [@emfdavid](https://discourse.pangeo.io/u/emfdavid)\
**Post date:** [June 27, 2026, 2:19am UTC](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635/3 "2026-06-27T02:19:57Z")

</div>

Cross posting here

> [@Favorite way to go from netCDF (&xarray) to torch/TF/Jax et al](https://discourse.pangeo.io/t/favorite-way-to-go-from-netcdf-xarray-to-torch-tf-jax-et-al/2663/9):
>
> The IO is fast now but there is still no standard tooling for chunked zarr compatible split/shuffle aware batching. I put some time (and tokens) into building [GitHub - emfdavid/insitubatch: Train in place on n-dimensional cloud tensors. · GitHub](https://github.com/emfdavid/insitubatch) this week. It is one Data Set abstraction compatible with Torch, Jax and TF. It is batteries included with tunable caching, pre-fechting and max-inflight. No heavy worker processes makes it much more efficient - lean memory and cpu (fetch once!) It …

---

_[View the full topic](https://discourse.pangeo.io/t/best-practice-for-machine-learning-with-huge-datasets/4635)._