Any tips on avoiding high AWS S3 cost when storing Zarr's with lots of objects

Data
1

Hello,

We are generating Zarr stores and push final data products to AWS S3 bucket. We were hit by unexpected high cost (~3K to store 40 data products) from AWS S3 to store our test data sets.
When using (250, 10, 10) chunking for (date, x, y) dimensions, we ended up with average ~3-4M objects per single Zarr store, thus the high S3 cost.

Our normal way of operation is to generate the data set locally and to push it to the S3 bucket. When new data that contributes to the data set becomes available, we copy existing Zarr store from S3 locally, update it with new data, remove original Zarr in S3 bucket, then push updated Zarr to the S3 bucket. Since we need to update our datasets often, there is a concern that high S3 costs will persist each time we push updated data to the S3 bucket.

In the past, we tried to write data directly to S3, but it was rather slow. We might need to revisit this approach as updating data set, which resides in S3 bucket, directly would minimize the number of objects pushed to the S3 bucket every time we need to update our dataset. I guess, we need to choose cost over runtime here.

Just for the record, we tested new (250, 50, 50) chunking when storing data to the Zarr. It does not affect data access time and definitely helps to cut the number of objects for the final Zarr store (from 3M to 120K objects). But since we will be updating Zarr’s with new data on a regular basis, the S3 cost still will be the issue as we will be generating thousands of data sets. It would also seem to limit us on how often we can update these data products if we push the whole updated Zarr to the bucket each time we update the data set.

Has anybody run into similar issues with storing Zarr’s in AWS S3?

I would appreciate if anybody could point us in the right direction on:

  1. Any known tricks and tips on how to avoid high S3 costs when pushing Zarr store with high number of objects or pushing Zarr with moderate number of objects frequently to the Cloud
  2. What is the best practice of updating Zarr store when it is stored in AWS S3 bucket (local update and push the whole store to the bucket, or update Zarr in S3 directly despite it being very slow).

Many thanks in advance,
Masha

1 Like
2

Hi @mliukis and welcome to the forum! :wave:

Thanks for raising this important topic.

Would you mind sharing a bit more detail about this cost breakdown? What is generating the costs?

As far as I understand from the S3 Pricing Scheme, there are three main categories of costs

  • Base storage cost (just proportional to the number of bytes stored)
  • Egress charges (proportional to outbound traffic)
  • Per-operation charges ($0.005 per 1000 PUT, COPY, POST, LIST requests; $0.0004 per 1000 GET, SELECT requests)

My first guess was that this was the requests charges. Doing the math, the cost to store (PUT) 3 million objects is $15. So you would need to store 200 such stores to get to $3000.

Overall, your chunks seem much too small. We tend to recommend 100 MB chunks, because these provide a good balance between latency and throughput.

I see a few different options to reduce your costs:

  • Depending on your update structure, you could append / overwrite parts of the arrays (just for the new data), rather than re-upload the whole thing from scratch. This only works if each update is for a small portion of the total dataset.
  • If you only need read-only access from the cloud, you could update a zipped version of the Zarr group. This would be a very large object, but you would only have a single PUT request. Fsspec should be able to handle the zip + s3fs combination and you would still likely get good performance. (This would need to be tested.)
1 Like
3

Hi @rabernat and thank you for your response!

The main contributor to the cost was PUT, COPY, POST, or LIST requests to the S3 bucket. For this particular set of data we ended up making 400,088,265 such requests which resulted in $2,000 charge. The ~3M objects per one Zarr store was just an example, other objects are much larger.

The chunking was selected based on the way we build our datasets and the way we access them. We append data in date dimension and access it by (x, y) coordinates. We had this discussion in my very first post to the forum :), which was very helpful.

Thank you for the suggestion to use zipped version of the Zarr - I will investigate and will post my findings if we are successful.

4

I’m curious why you don’t think appending is feasible.

5

Do you mean appending to existing Zarr as stored on the Cloud?

1 Like
6

Yes that is what I mean.

7

Tried it before and it was much slower than appending to the local store and then copy the whole set to the S3. As I mentioned in my original post, we might want to revisit this approach now that S3 cost is part of the problem.

8

I would try to understand why it was slow. It doesn’t really make sense what you are saying. Why would it be faster to upload a lot more data? :face_with_monocle: With appending, you are only uploading the new data, so that should be a lot faster (and also cheaper).

9

@rabernat Sorry to miss your response. Sorry for not being clear in my previous statement what was slower.
What we tested before was:

  1. Building the Zarr store locally by appending to it processed data in batches (as we can’t store all of our data in memory at the same time) and then uploading complete Zarr to the S3 bucket
    vs.
  2. Avoiding local version of the Zarr and building/updating the Zarr directly in the S3 bucket.

Building the Zarr locally and uploading complete set to the S3 using aws s3 cp by taking advantage of max_concurrent_requests configuration option was much faster than building the Zarr directly in the S3 bucket.

Another thing to consider is that we will need to remove some of the “t” layers from our our datasets when we update it with new data as new (reprocessed) data might need to take a precedence over existing data. Would that be a feasible thing to do if the Zarr is residing in the S3 bucket?

10

Not sure if it would help but here are a couple of practical considerations that I took when uploading to object storage with respect to cost:

  • I used Digital Ocean Space rather than AWS since it has simpler pricing and no separate charge for requests. It’s compatible with S3 interface.
  • The compute should also be done in the cloud within the same data centre - this saves on egress charges and also usually much, much faster.
1 Like
11

Thank you @josephyang! Anything else than AWS Cloud is not an option for us unfortunately. Increasing chunking a bit solves the problem and still provides compatible access time.

12

Just revisiting this topic a little bit as I’ve been calculating the costs of storing of storing multi-band data as GeoTIFFs or Zarr on an s3 compatible stores, where 1 GeoTIFF would convert into 10+ or 100+ files/objects. It does get very expensive on the scale of millions/billions of files in terms of PUT/GET request costs!

Is the current recommendation to go with Zarr v3’s newly accepted ZEP 2 Sharding codec? Whereby one shard is effectively a grouping of multiple chunks?

Sharding diagram in Zarr v3 ZEP2

Or are there other methods or file formats that are worth looking at to reduce the file/object count?

13

If you have existing Zarr data you want to expose for read-only use, and you’d like to reduce the file / object count, a great option (which does not require sharding) is to put it all inside a zip file. You can read from this zip file concurrently with no performance penalty.

If you want to continue writing to the store, the sharded Zarr V3 format would probably be a better fit. Support is not fully implemented yet in Zarr python, but it should be here soon.

14

Right, but this would rely on the Zarr reader implementation being able to understand zipfiles? E.g. with zarr-python’s ZipStore (requires Zarr-spec v3?), or by routing through fsspec’s ZipFileSystem? I’ll need to see if other libraries such as kvikIO (Zarr — kvikio 23.10.00 documentation) or those written in Rust/Javascript can handle this natively too. But will keep an eye on the v3 spec, especially PRs to xarray like added 'storage_transformers' to valid_encodings by JMorado · Pull Request #7540 · pydata/xarray · GitHub