# Cloud Optimized Geotiffs + Pangeo best practices

**URL:** <https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999>\
**Category:** Data\
**Created:** [November 5, 2020, 3:03am UTC](https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999 "2020-11-05T03:03:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![scottyhq](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/scottyhq/32/162_2.png) [@scottyhq](https://discourse.pangeo.io/u/scottyhq)\
**Post date:** [November 5, 2020, 3:03am UTC](https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999/1 "2020-11-05T03:03:25Z")

</div>

Hi everyone, I’m attempting to consolidate collective wisdom on working with Cloud Optimized Geotiffs on Pangeo infrastructure. Mainly focusing on using Xarray and Dask to efficiently run computations against collections of COGs from within the same Cloud datacenter.

I think there is quite a bit to explore here, so I’ve decided to put together a repository to add several notebooks to get started. More details in the repository readme:

> **[pangeo-data/cog-best-practices](https://github.com/pangeo-data/cog-best-practices)**
>
> Best practices with cloud-optimized-geotiffs (COGs) - pangeo-data/cog-best-practices

I think there are already some helpful resources and examples in the notebooks, for example, how setting GDAL environment variables when reading COGs can improve opening speed from seconds to milliseconds (GDAL\_DISABLE\_READDIR\_ON\_OPEN=EMPTY\_DIR)! Examples of how to combine many 2D COGS into a 3D DataArray… etc

My hope is that folks who are interested in this can leverage our current infrastructure and dig into a common dataset to address questions that often come up:

- Should we use processes or threads when working with a Dask LocalCluster?
- Do we have rules of thumb for CPU,RAM,nthreads for a Dask GatewayCluster options?
- How do file locks for concurrent reading and writing impact dask task graph execution?

I’d particularly love some feedback from folks versed in interpreting dask diagnostics for efficiency in the LocalCluster and GatewayCluster examples. Ultimately we can consolidate findings in a new pangeo gallery or blog post!

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 5, 2020, 3:19pm UTC](https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999/2 "2020-11-05T15:19:35Z")

</div>

This is a great post and amazing set of examples @scottyhq!

What do you think about sharing this via the Pangeo twitter account to solicit feedback. There are a lot of COG / geospatial people out there who probably don’t monitor this forum…

---

<div class="post-metadata">

**Author:** ![scottyhq](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/scottyhq/32/162_2.png) [@scottyhq](https://discourse.pangeo.io/u/scottyhq)\
**Post date:** [November 5, 2020, 4:42pm UTC](https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999/3 "2020-11-05T16:42:57Z")

</div>

Thanks @rabernat - Good idea. I just sent out a tweet, so please do retweet to get more visibility.

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [November 6, 2020, 5:20am UTC](https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999/4 "2020-11-06T05:20:28Z")

</div>

This is good, thanks!

---

<div class="post-metadata">

**Author:** ![RichardScottOZ](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/richardscottoz/32/752_2.png) [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)\
**Post date:** [January 21, 2021, 7:12am UTC](https://discourse.pangeo.io/t/cloud-optimized-geotiffs-pangeo-best-practices/999/5 "2021-01-21T07:12:01Z")

</div>

Considering a couple of the lambda examples - the important part is how many milliseconds? e.g. what would give us definitely the best performance if considering going that way.