Reader.load() issues with ATL06

Hi @tlohde,

While running region_a.order_granules(), I noticed a lot of lines like:

NSIDC returned these messages
['Granule 228833223 contained no data within the spatial and/or temporal '
 'subset constraints to be processed',
 'Granule 228825218 contained no data within the spatial and/or temporal '
 'subset constraints to be processed',
...

which seems to indicate some data is missing. So I tried reading the ICESat-2 HDF5 files using datatree.open_datatree and it does look like some of the laser beams are missing, e.g. this one only has 2 (gt3l and gt3r) out of 6 beams:

import datatree

is2dt = datatree.open_datatree(
    "icesat-2_data/processed_ATL06_20190108030055_01630205_005_01.h5",
    engine="h5netcdf",
    phony_dims="access",
)
print(is2dt)
DataTree('None', parent=None)
│   Dimensions:  ()
│   Data variables:
│       *empty*
│   Attributes: (12/49)
│       Conventions:                        CF-1.6
│       Processing Parameters:              This file was gernerated by the ICESa...
│       citation:                           Cite these data in publications as fo...
│       contributor_name:                   Thomas E Neumann (thomas.neumann@nasa...
│       contributor_role:                   Instrument Engineer, Investigator, Pr...
│       creator_name:                       GSFC I-SIPS > ICESat-2 Science Invest...
│       ...                                 ...
│       summary:                            ATL06 provides estimates of the ice-s...
│       time_coverage_duration:             325.0
│       time_coverage_end:                  2019-01-08T03:06:20.000000Z
│       time_coverage_start:                2019-01-08T03:00:55.000000Z
│       time_type:                          CCSDS UTC-A
│       title:                              ATLAS/ICESat-2 L3A Land Ice Height
├── DataTree('METADATA')
│   │   Dimensions:  ()
│   │   Data variables:
│   │       *empty*
│   │   Attributes:
│   │       Description:            ISO19115 Structured Metadata Represented within HDF5
│   │       iso_19139_dataset_xml:  <?xml version="1.0"?>\n<gmd:DS_Series xsi:schemaL...
│   │       iso_19139_series_xml:   <?xml version="1.0" encoding="UTF-8"?>\n<gmd:DS_S...
│   ├── DataTree('AcquisitionInformation')
│   │   ├── DataTree('lidar')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  ATLAS on ICESat-2 determines the range between the satellit...
│   │   │           identifier:   ATLAS
│   │   │           pulse_rate:   10000 pps
│   │   │           type:         Laser Altimeter
│   │   │           wavelength:   532 nm
│   │   ├── DataTree('lidarDocument')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           edition:          Pre-Release
│   │   │           publicationDate:  12/31/17
│   │   │           title:            A document describing the ATLAS instrument will be prov...
│   │   ├── DataTree('platform')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  Ice, Cloud, and land Elevation Satellite-2
│   │   │           identifier:   ICESat-2
│   │   │           type:         Spacecraft
│   │   └── DataTree('platformDocument')
│   │           Dimensions:  ()
│   │           Data variables:
│   │               *empty*
│   │           Attributes:
│   │               edition:          31-Dec-16
│   │               publicationDate:  31-Dec-16
│   │               title:            The Ice, Cloud, and land Elevation Satellite-2 (ICESat-...
│   ├── DataTree('DataQuality')
│   │   │   Dimensions:  ()
│   │   │   Data variables:
│   │   │       *empty*
│   │   │   Attributes:
│   │   │       scope:    NOT_SET
│   │   ├── DataTree('CompletenessOmission')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           evaluationMethodType:  directInternal
│   │   │           measureDescription:    TBD
│   │   │           nameOfMeasure:         TBD
│   │   │           unitofMeasure:         TBD
│   │   │           value:                 NOT_SET
│   │   └── DataTree('DomainConsistency')
│   │           Dimensions:  ()
│   │           Data variables:
│   │               *empty*
│   │           Attributes:
│   │               evaluationMethodType:  directInternal
│   │               measureDescription:    TBD
│   │               nameOfMeasure:         TBD
│   │               unitofMeasure:         TBD
│   │               value:                 NOT_SET
│   ├── DataTree('DatasetIdentification')
│   │       Dimensions:  ()
│   │       Data variables:
│   │           *empty*
│   │       Attributes: (12/14)
│   │           VersionID:                   005
│   │           abstract:                    ATL06 provides estimates of the ice-sheet me...
│   │           characterSet:                utf8
│   │           creationDate:                2021-09-05T11:16:31.000000Z
│   │           credit:                      The software that generates the ATL06 produc...
│   │           fileName:                    ATL06_20190108030055_01630205_005_01.h5
│   │           ...                          ...
│   │           purpose:                     ATL06 provides estimates of the ice-sheet me...
│   │           shortName:                   ATL06
│   │           spatialRepresentationType:   along-track
│   │           status:                      onGoing
│   │           topicCategory:               geoscientificInformation
│   │           uuid:                        6abf99e8-3be1-30e0-a623-28f0d12fc9b8
│   ├── DataTree('Extent')
│   │       Dimensions:  ()
│   │       Data variables:
│   │           *empty*
│   │       Attributes:
│   │           eastBoundLongitude:      -41.329960843275046
│   │           northBoundLatitude:      80.00112360641836
│   │           rangeBeginningDateTime:  2019-01-08T03:00:54.862971Z
│   │           rangeEndingDateTime:     2019-01-08T03:05:36.716995Z
│   │           southBoundLatitude:      62.24122894734801
│   │           westBoundLongitude:      -50.388809857370006
│   ├── DataTree('Lineage')
│   │   ├── DataTree('ANC06-01')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  MERIT 3 arcsec Digital Elevation Model reformatted into HDF...
│   │   │           fileName:     merit_3as_20200617_001_01.h5
│   │   │           shortName:    ANC06-01
│   │   │           uuid:         3db8ab65-d2ac-37c7-86db-17c762268673
│   │   │           version:      20200617
│   │   ├── DataTree('ANC06-02')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  ArcticDEM 32m Digital Elevation Model reformatted into HDF5.
│   │   │           fileName:     arcticdem32m_20190611_001_01.h5
│   │   │           shortName:    ANC06-02
│   │   │           uuid:         ce07ef72-0bf4-353b-8475-fb568b029905
│   │   │           version:      20190611
│   │   ├── DataTree('ANC06-03')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  REMA Antarctica 100m Digital Elevation Model filled and ref...
│   │   │           fileName:     atl06rema100m_20190628_001_01.h5
│   │   │           shortName:    ANC06-03
│   │   │           uuid:         3e0a81bd-bbd5-35fa-b868-3254f9355b7f
│   │   │           version:      20190628
│   │   ├── DataTree('ANC17')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  Land Ice Height SNR Significance Table
│   │   │           fileName:     anc17_snr_f_20170718_001_01.h5
│   │   │           shortName:    ANC17
│   │   │           uuid:         52fe4165-bac0-3dc9-a9b3-921a591a7dc5
│   │   │           version:      20170718
│   │   ├── DataTree('ANC19')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  TAI to UTC leapsecond file retrieved from ftp://maia.usno.n...
│   │   │           fileName:     tai_utc_2017.dat
│   │   │           shortName:    ANC19
│   │   │           uuid:         7c66d365-278a-31f7-8fe4-9c80e2f012e5
│   │   │           version:      001
│   │   ├── DataTree('ANC25-06')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  HDF5 template file that defines the organization and defaul...
│   │   │           fileName:     ANC25-06_20210831171721_045_01.h5
│   │   │           shortName:    ANC25-06
│   │   │           uuid:         d29cdb37-801e-3a00-a034-d20c51047ad6
│   │   │           version:      045
│   │   ├── DataTree('ANC26-06')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  HDF5 template file that defines the organization and defaul...
│   │   │           fileName:     ANC26-06_20210831171731_045_01.h5
│   │   │           shortName:    ANC26-06
│   │   │           uuid:         2093f4f5-2274-3520-8824-cae25668b08b
│   │   │           version:      045
│   │   ├── DataTree('ANC28')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  DTU Mean Sea Surface re-referenced to the WGS84 ellipsoid.
│   │   │           fileName:     dtu13_20180705_001_01.nc
│   │   │           shortName:    ANC28
│   │   │           uuid:         56f47040-a72e-3109-99c2-bc1658e6dda4
│   │   │           version:      20180705
│   │   ├── DataTree('ANC36-06')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  ISO 19139 XML file containing Series-level metadata informa...
│   │   │           fileName:     ANC36-06_20210831170309_005_01.xml
│   │   │           shortName:    ANC36-06
│   │   │           uuid:         7e1fe2fa-7a92-39a3-9f24-630dca51625a
│   │   │           version:      005
│   │   ├── DataTree('ANC38-06')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           description:  ISO 19139 XML file containing DataSet-level metadata inform...
│   │   │           fileName:     ANC38-06_20210831170314_005_01.xml
│   │   │           shortName:    ANC38-06
│   │   │           uuid:         f2325ab1-962d-3bf1-a1ef-7e66bdb9ffb3
│   │   │           version:      005
│   │   ├── DataTree('ATL03')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes: (12/15)
│   │   │           description:   ICESat-2 ATLAS L2A Global Geolocated Photon data products.
│   │   │           end_cycle:     2
│   │   │           end_geoseg:    671084
│   │   │           end_orbit:     1751
│   │   │           end_region:    5
│   │   │           end_rgt:       163
│   │   │           ...            ...
│   │   │           start_geoseg:  555757
│   │   │           start_orbit:   1751
│   │   │           start_region:  5
│   │   │           start_rgt:     163
│   │   │           uuid:          6050c1de-3d15-3ad2-9f5a-b7b6d56cef26
│   │   │           version:       005
│   │   ├── DataTree('ATL09')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes: (12/15)
│   │   │           description:   ICESat-2 ATLAS L3A atmosphere data products.
│   │   │           end_cycle:     2
│   │   │           end_geoseg:    2007126
│   │   │           end_orbit:     1751
│   │   │           end_region:    14
│   │   │           end_rgt:       163
│   │   │           ...            ...
│   │   │           start_geoseg:  12
│   │   │           start_orbit:   1751
│   │   │           start_region:  1
│   │   │           start_rgt:     163
│   │   │           uuid:          8672b328-3422-345d-b8b9-418830801f5d
│   │   │           version:       005
│   │   └── DataTree('Control')
│   │           Dimensions:  ()
│   │           Data variables:
│   │               *empty*
│   │           Attributes:
│   │               description:  Text-based keyword=value file generated automatically withi...
│   │               fileName:     CTL_atlas_l3a_is_007739532.ctl
│   │               shortName:    CNTL
│   │               version:      1
│   ├── DataTree('ProcessStep')
│   │   ├── DataTree('Browse')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           identifier:          atlas_brw
│   │   │           processDescription:  Browse processing is performed for each granule SIPS...
│   │   │           runTimeParameters:   CTL_atlas_l3a_is_007739532.ctl
│   │   │           softwareDate:        Jul 28 2021
│   │   │           softwareTitle:       Creates ATLAS HDF5 browse files
│   │   │           softwareVersion:     Version 2.5
│   │   │           stepDateTime:        2021-09-05T11:19:48.000000Z
│   │   ├── DataTree('Metadata')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           identifier:          atlas_meta
│   │   │           processDescription:  Metadata information is processed by the metadata ut...
│   │   │           runTimeParameters:   CTL_atlas_l3a_is_007739532.ctl
│   │   │           softwareDate:        Jul 28 2021
│   │   │           softwareTitle:       Creates ATLAS XML metadata files
│   │   │           softwareVersion:     Version 4.5
│   │   │           stepDateTime:        2021-09-05T11:19:59.000000Z
│   │   ├── DataTree('PGE')
│   │   │       Dimensions:  ()
│   │   │       Data variables:
│   │   │           *empty*
│   │   │       Attributes:
│   │   │           ATBDDate:            12/04/2019
│   │   │           ATBDTitle:           Algorithm Theoretical Basis Document (ATBD) For Land...
│   │   │           ATBDVersion:         N/A
│   │   │           documentDate:        Feb 2020
│   │   │           documentation:       ATLAS Science Algorithm Software Design Description ...
│   │   │           identifier:          atlas_l3a_is
│   │   │           processDescription:  Computes surface heights for each beam, along and ac...
│   │   │           runTimeParameters:   CTL_atlas_l3a_is_007739532.ctl
│   │   │           softwareDate:        Jul 28 2021
│   │   │           softwareTitle:       ASAS L3A Icesheet PGE
│   │   │           softwareVersion:     Version 4.5
│   │   │           stepDateTime:        2021-09-05T11:16:31.000000Z
│   │   └── DataTree('QA')
│   │           Dimensions:  ()
│   │           Data variables:
│   │               *empty*
│   │           Attributes:
│   │               identifier:          atl06_qa_util
│   │               processDescription:  QA processing is performed by an external utility on...
│   │               runTimeParameters:   CTL_atlas_l3a_is_007739532.ctl
│   │               softwareDate:        Jul 28 2021
│   │               softwareTitle:       ATL06 QA Utility
│   │               softwareVersion:     Version 4.5
│   │               stepDateTime:        2021-09-05T11:19:46.000000Z
│   ├── DataTree('ProductSpecificationDocument')
│   │       Dimensions:  ()
│   │       Data variables:
│   │           *empty*
│   │       Attributes:
│   │           ShortName:        ATL06_SDP
│   │           characterSet:     utf8
│   │           edition:          v4.3
│   │           language:         eng
│   │           publicationDate:  Feb 2020
│   │           title:            ICESat-2-SIPS-SPEC-4260 - ATLAS Science Algorithm Stand...
│   ├── DataTree('QADatasetIdentification')
│   │       Dimensions:  ()
│   │       Data variables:
│   │           *empty*
│   │       Attributes:
│   │           abstract:      An ASCII product that contains statistical information on ...
│   │           creationDate:  2021-09-05T11:19:46.000000Z
│   │           fileName:      ATL06_20190108030055_01630205_005_01.qa
│   └── DataTree('SeriesIdentification')
│           Dimensions:  ()
│           Data variables:
│               *empty*
│           Attributes: (12/19)
│               VersionID:                         005
│               abstract:                          ATL06 provides estimates of the ice-sh...
│               characterSet:                      utf8
│               credit:                            The software that generates the ATL06 ...
│               format:                            HDF
│               formatVersion:                     5
│               ...                                ...
│               purpose:                           ATL06 provides estimates of the ice-sh...
│               resourceProviderOrganizationName:  National Aeronautics and Space Adminis...
│               revisionDate:                      2016-06-09
│               shortName:                         ATL06
│               status:                            onGoing
│               topicCategory:                     geoscientificInformation
├── DataTree('ancillary_data')
│       Dimensions:              (phony_dim_0: 1)
│       Dimensions without coordinates: phony_dim_0
│       Data variables:
│           atlas_sdp_gps_epoch  (phony_dim_0) datetime64[ns] ...
│           data_end_utc         (phony_dim_0) |S27 ...
│           data_start_utc       (phony_dim_0) |S27 ...
│           end_delta_time       (phony_dim_0) datetime64[ns] ...
│           granule_end_utc      (phony_dim_0) |S27 ...
│           granule_start_utc    (phony_dim_0) |S27 ...
│           start_delta_time     (phony_dim_0) datetime64[ns] ...
│       Attributes:
│           Description:  Contains information ancillary to the data product. This ma...
│           data_rate:    Data within this group pertain to the granule in its entirety.
├── DataTree('gt3l')
│   │   Dimensions:  ()
│   │   Data variables:
│   │       *empty*
│   │   Attributes:
│   │       Description:         Contains subgroups organized by Ground Track (gt1l, ...
│   │       atlas_beam_type:     strong
│   │       atlas_pce:           pce3
│   │       atlas_spot_number:   5
│   │       atmosphere_profile:  profile_3
│   │       groundtrack_id:      gt3l
│   │       sc_orientation:      Backward
│   └── DataTree('land_ice_segments')
│           Dimensions:    (phony_dim_1: 1)
│           Dimensions without coordinates: phony_dim_1
│           Data variables:
│               h_li       (phony_dim_1) float32 ...
│               latitude   (phony_dim_1) float64 ...
│               longitude  (phony_dim_1) float64 ...
│           Attributes:
│               Description:  The land_ice_height group contains the primary set of deriv...
│               data_rate:    Data within this group are sparse.  Data values are provide...
├── DataTree('gt3r')
│   │   Dimensions:  ()
│   │   Data variables:
│   │       *empty*
│   │   Attributes:
│   │       Description:         Contains subgroups organized by Ground Track (gt1l, ...
│   │       atlas_beam_type:     weak
│   │       atlas_pce:           pce3
│   │       atlas_spot_number:   6
│   │       atmosphere_profile:  profile_3
│   │       groundtrack_id:      gt3r
│   │       sc_orientation:      Backward
│   └── DataTree('land_ice_segments')
│           Dimensions:    (phony_dim_2: 1)
│           Dimensions without coordinates: phony_dim_2
│           Data variables:
│               h_li       (phony_dim_2) float32 ...
│               latitude   (phony_dim_2) float64 ...
│               longitude  (phony_dim_2) float64 ...
│           Attributes:
│               Description:  The land_ice_height group contains the primary set of deriv...
│               data_rate:    Data within this group are sparse.  Data values are provide...
└── DataTree('orbit_info')
        Dimensions:         (sc_orient_time: 1)
        Coordinates:
          * sc_orient_time  (sc_orient_time) datetime64[ns] 2019-01-07T23:30:00
        Data variables:
            sc_orient       (sc_orient_time) int8 ...
        Attributes:
            Description:  Contains data that are common among all beams for the granu...
            data_rate:    These parameters are constant for a given granule.

So to answer your questions:

You’re not necessarily doing anything wrong since the subsetter is doing its job (removing laser tracks that aren’t needed)? The unfortunate thing is that sometimes there are 2 lasers returned, sometimes 6, and the current icepyx reader can’t handle the inconsistency. Maybe using region_a.order_granules(subset=False) could allow you to keep all 6 laser tracks, but that’s a question for @JessicaS11 perhaps

Depends on how you want to combine them :smiley: If you describe your scientific goal briefly (i.e. what you’re trying to do with the ICESat-2 elevation data), it’ll be easier for us to better advise on what’s the best way. At the moment, I can only guess that you you want to look at elevation change over time or something?