# Kerchunk planning

**URL:** <https://discourse.pangeo.io/t/kerchunk-planning/4002>\
**Category:** News & Announcements\
**Created:** [January 30, 2024, 3:40pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002 "2024-01-30T15:40:34Z")\
**Posts on this page:** 17\
**Page:** 2

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 1, 2024, 5:54pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/21 "2024-04-01T17:54:01Z")

</div>

Hello, I have been looking at virtualizarr, I think it makes sense and i really like the idea. One question that i have is, wouldn’t virtualizarr be a replacement for rechunker? If you generate indexes of some format, then you can rechunk it efficiently as needed.

I guess the metadata is part of zarr and is coming with v3, is that right?

For kerchunk I was wondering, is there any plan to add more formats?  
I don’t know about unidimensional data and how it would be represented using Zarr/xarray (for example fastq or fasta files from genomics, but i think tiledb does it anyway), but what about other formats such as imzML format for spectrometry data? I see that there’s a need for distributed and parallel processing in scientific pipelines (worked on a genomics pipeline and currently working on radio interferometry pipeline), and i feel like multidimensional arrays are a really powerful abstraction.

Thanks for your amazing work.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [April 1, 2024, 6:57pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/22 "2024-04-01T18:57:38Z")

</div>

> [@abourramouss](#):
>
> For kerchunk I was wondering, is there any plan to add more formats?  
> I don’t know about unidimensional data and how it would be represented using Zarr/xarray (for example fastq or fasta files from genomics), but what about other formats such as imzML format for spectrometry data?

Absolutely!  
The extent to which a format will be useful to kerchunk will depend on the specifics (like how well it meets zarr/xarray’s model) and how complex the encoding is, but I am happy to consider any array binary format.

I think imzML specifically is one I came across recently: base64-ascii blocks within an XML file, right? This can be kerchunked; whether it’s useful wil depend on how big the chunks are compared to the reference that represents them.

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 1, 2024, 7:09pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/23 "2024-04-01T19:09:36Z")

</div>

> [@martindurant](#):
>
> I think imzML specifically is one I came across recently: base64-ascii blocks within an XML file, right? This can be kerchunked; whether it’s useful wil depend on how big the chunks are compared to the reference that represents them.

Yes, you might find this useful:

> <https://github.com/aitorarjona/dataplug/blob/master/dataplug/formats/metabolomics/imzml.py>

It’s from a colleague of mine, but the principle is the same as kerchunk, parallel processing and data indexing.

There are more formats if you want to look at it.

What i miss is some abstraction underneath that has a “universal” api, I really like the virtualzarr idea since it might fit in there. Currently the framework needs for the developer to create a new “data abstraction” each time a new format is added.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [April 1, 2024, 7:27pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/24 "2024-04-01T19:27:41Z")

</div>

Yes, kerchunk aims for a universal API for all the formats, but the specifics of each format would still need to be handled _somewhere_ (once for each format, and scanning run once for each input file).

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 1, 2024, 8:12pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/25 "2024-04-01T20:12:30Z")

</div>

Sorry, i wasn’t clear, i was refering to the interaction of datasets after they’re ingested with kerchunk, basically I like how kerchunk integrates xarray, It doesn’t matter what the user wants to ingest, the resulting dataset will be on an xarray format.

The dataplug framework instead needs the developer to define the abstraction.

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [April 1, 2024, 8:26pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/26 "2024-04-01T20:26:42Z")

</div>

> [@abourramouss](#):
>
> Hello, I have been looking at virtualizarr, I think it makes sense and i really like the idea. One question that i have is, wouldn’t virtualizarr be a replacement for rechunker? If you generate indexes of some format, then you can rechunk it efficiently as needed.

I’m not sure your suggestion makes sense, at least for compressed chunks? Kerchunk / VirtualiZarr are manipulating references to compressed chunks on-disk, but they can’t change the content of those chunks. A rechunk would imply changing the contents of the chunks.

> [@abourramouss](#):
>
> I guess the metadata is part of zarr and is coming with v3, is that right?

In the VirtualiZarr model metadata either becomes part of the xarray model (i.e. the `dimension_names`) or it gets moved around as `.attrs`. It can then be written back out as part of a zarr store yes, but v3 won’t really change anything about how that works.

> [@abourramouss](#):
>
> For kerchunk I was wondering, is there any plan to add more formats?

> [@martindurant](#):
>
> Yes, kerchunk aims for a universal API for all the formats, but the specifics of each format would still need to be handled _somewhere_ (once for each format, and scanning run once for each input file).

> [@abourramouss](#):
>
> Sorry, i wasn’t clear, i was refering to the interaction of datasets after they’re ingested with kerchunk, basically I like how kerchunk integrates xarray, It doesn’t matter what the user wants to ingest, the resulting dataset will be on an xarray format.

If kerchunk keeps adding more backends to support creating reference dicts from other file formats, VirtualiZarr should still be able to consume those. That would be a nice separation of concerns.

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 1, 2024, 9:16pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/27 "2024-04-01T21:16:49Z")

</div>

> [@TomNicholas](#):
>
> I’m not sure your suggestion makes sense, at least for compressed chunks? Kerchunk / VirtualiZarr are manipulating references to compressed chunks on-disk, but they can’t change the content of those chunks. A rechunk would imply changing the contents of the chunks.

So if my understanding is correct, rechunker comes from the need of resizing chunks from zarr, since some chunk configurations can be suboptimal.

If you use kerchunk to read some format and have the reference-index as a virtualizarr, doesn’t it make sense that if the user wants to create arbitraty size chunks, to use the index? Since virtualizarr and kerchunk create a “virtual view” of the dataset, wouldn’t that mean a more efficient rechunking algorithm?

This would basically mean rearranging the chunk metadata/index to create bigger-smaller chunks.

Maybe I am missing something? I haven’t really used rechunker

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [April 1, 2024, 9:49pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/28 "2024-04-01T21:49:05Z")

</div>

@abourramouss this approach might work for “fusing” chunks together to make bigger chunks, but IIUC it won’t allow you to make smaller chunks than you started with, because you would then be trying to read only part of a compressed chunk.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [April 2, 2024, 1:32am UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/29 "2024-04-02T01:32:28Z")

</div>

Perhaps you mean that you can use rechunker across a set of files that don’t otherwise form a logical dataset, but that kerchunk/vitualizarr can “combine” for you? In that case, yes, you could use the tools before rechunker, but I’m not sure it provides any benefit over open\_mfdataset or other xarray API combiners. After all, you would only normally be doing this once.

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 2, 2024, 11:01pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/30 "2024-04-02T23:01:02Z")

</div>

On a different topic, do you think measurement sets could be potentially read by kerchunk?  
[https://casaguides.nrao.edu/index.php/Measurement\_Set\_Contents](https://casaguides.nrao.edu/index.php/Measurement_Set_Contents)

They are datasets used in radio interferometry, the complexity here is that the dataset consists of different tables, the dataset is like a relational database internally.

Another thing is that different formats are used phisically, so that would involve implementing different algorithms for each table format (If i remember correctly, there were 4)

Given the structure of the dataset, is this something worth pursuing?

Measurement set structure:  
 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/4/4adfce5bfa78210dada1a59fc507ddfcb1ffb199.jpeg)

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [April 3, 2024, 12:26am UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/31 "2024-04-03T00:26:10Z")

</div>

Kerchunk can probably find the binary blocks corresponding to the tables (are they chunked? compressed/encoded?), and assign a compound dtype to each. But in the zarr model, these would all just be free-standing arrays without any relations. Zarr works best for chunked multi-dimensional arrays, offering easy parallelism.

Kerchunk is not entirely tied to zarr, you can make a reference filesystem from anything and pass them to fsspec-expecting libraries, but there isn’t really much else you can do with these at the moment. Does CASA deal with zarr at all?

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 3, 2024, 11:54am UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/32 "2024-04-03T11:54:36Z")

</div>

> [@martindurant](#):
>
> Kerchunk can probably find the binary blocks corresponding to the tables (are they chunked? compressed/encoded?), and assign a compound dtype to each. But in the zarr model, these would all just be free-standing arrays without any relations. Zarr works best for chunked multi-dimensional arrays, offering easy parallelism.

I have been trying to learn more about the format as having a xarray interface to it as well as doing parallel processing would be amazing.

The measurement set tables are written and read on different formats, those formats are storage managers and each storage manager used to handle different tables. They are responsible for the physical storage of the data, including aspects like chunking and compression:

1. IncrementalStMan: Is used for columns where data changes incrementally, the idea is to use it with tables that repeat data, this way it can be compressed.
2. StandardStMan: serves as a general purpose manager without specific compression techniques
3. TiledColumnStMan: manages multidimensional arrays by dividing them into tiles (chunks). This is aligned with the concept of chunking in Zarr.

The intricate part is that storage managers are used per table and not per dataset.

My idea is to have a reference (byte-range reference, what virtualizarr is doing) to these type of files, this would ease parallel processing of the data as well as pipeline operations. Currently I am doing exactly that, but using casa-core tools, which aren’t cloud-native.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [April 3, 2024, 1:04pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/33 "2024-04-03T13:04:06Z")

</div>

From our point of view, each table would be a different variable/array, and having a different encoding for each is fine. We just have to write the three codecs in numcodecs-style classes (which is super easy if you already have an implementation). For the specific case of “tiled”, the chunking in the virtual zarr dataset would match the original chunking. For the other two, you probably can only have one chunk for each table in each input dataset. Whether multiple sets can be combined in any logical way, I leave for the domain experts 🙂 .

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 3, 2024, 1:15pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/34 "2024-04-03T13:15:23Z")

</div>

Yes, that’s another discussion, the data set is a collection of measurement sets, and at the end, a data set would be processed together in parallel.

I have worked a bit on the problem a couple months ago, altought i didn’t get any results, i started understanding the format.

Do you think it would be feasible to make a kerchunk reader and be able to create a reference using virtualizarr? it would immensely ease parallel data processing in the cloud.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [April 3, 2024, 1:28pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/35 "2024-04-03T13:28:59Z")

</div>

From what you have said here, I imagine a kerchunk scanner for the format is feasible, yes. Since it’s quite complicated, it will take some effort to get all the details right - perhaps you would start by just extracting a big tiled/chunked array at first to prove the concept.

The dataset not being regular, you would probably end up making your own combine routine rather than using kerchunk.combine or virtualizarr, but that part can wait.

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 3, 2024, 2:20pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/36 "2024-04-03T14:20:26Z")

</div>

Well you give me some confidence then, thank you!

---

<div class="post-metadata">

**Author:** ![abourramouss](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/abourramouss/32/2346_2.png) [@abourramouss](https://discourse.pangeo.io/u/abourramouss)\
**Post date:** [April 14, 2024, 9:06pm UTC](https://discourse.pangeo.io/t/kerchunk-planning/4002/37 "2024-04-14T21:06:58Z")

</div>

Sorry, I was wrong here, measurement sets storage managers (tiled, standard, incremental) are used per column on a single table.

When reading columns from a table, different storage managers are used, so tables are composed by columns that are managed by different storage managers.

Altought I still find some stuff confusing, I am still trying to do the kerchunk-measurement set reader.

[Previous page](https://discourse.pangeo.io/t/kerchunk-planning/4002.md?page=1)