Add limited landsat-8 reflectance collection(s) for covid-19 water quality discovery

[anayeaye]

What

Add two landsat-8 products to support stories in the covid-19 dashboard water quality discovery. These products were previously populated through an external landsat-8 API; for VEDA we will just duplicate the two multiband products to veda-data-store-staging and generate one or two collections referring to the products.

• implementor's decision: Create one micro-collection for the two landsat products needed to tell the Iowa and Belize stories. OR Choose two separate collections because the scenes are spatially disparate. Below are notes for two collections that could instead be generalized for one collection.

EDIT: the original descriptions assumed level 2 data but the requested IDs were level 1; descriptions should reflect what ever processing level is backfilled for these collections
id: ls8-water-quality-covid-19-belize
scene id: LC08_L1TP_019048_20200303_20200314_01_T1
Description: “Selected Landsat-8 Level-1 Surface Reflectance data for demonstrating the usage of the agriculture band combination (6, 5, 2)”

id: ls8-water-quality-covid-19-iowa
scene id: LC08_L1TP_026031_20200304_20200314_01_T1
Description: “Selected Landsat-8 Level-1 Surface Reflectance data for demonstrating the usage of the bathymetric band combination (4,3,1).”

Other STAC collection metadata

• license: EDIT: "https://d9-wret.s3.us-west-2.amazonaws.com/assets/palladium/production/s3fs-public/atoms/files/Landsat_Data_Policy.pdf"
• temporal interval: 20200303/20200304
• dashboard is periodic: False
• dashboard time density is null, should not be set

Notes

• This collection appears in the Water Quality and Covid 19 story
• Covid-19 dashboard migration wrap up doc
• veda-config Covid 19 Water Quality discovery issue

AC

• Migrate both landsat-8 products to veda-data-store-staging and ingest
• Once approved, merge and close.
• Notify @danielfdsilva and @nerik that the landsat-8 data supporting the Monitoring water quality in Belize and Counting crops during a lockdown Iowa stories has been migrated

[anayeaye]

@moradology this is the landsat-8 full product micro-collection that might follow the landsat ingest routine used for the glaciers collections.

[moradology]

The ID and the description seem to be inconsistent with one another: LC08_L1TP_019048_20200303_20200314_01_T1 looks like Level 1 data (L1TP) but the description is for level 2 data. L2 data from the date and row/path in question does exist:

[moradology]

Actually, maybe I missed this warning: https://www.usgs.gov/landsat-missions/landsat-collection-1. Do we happen to know of an alternative way of sourcing this stuff? AWS seems not to have it and USGS isn't providing it so maybe the API we were using before?

[jsignell]

Probably not a good idea, but planetary computer has landsat-collection-1 data up.

[moradology]

I'm thinking maybe we just use L2 instead... Not sure what advantage L1 would offer us here 🤷

[anayeaye]

I synced the files obtained by @moradology to the veda-data-store-staging bucket. There are 34 files in all

aws s3 sync s3://covid-eo-data/l8-water-quality/ s3://veda-data-store-staging/l8-water-quality/ contains two granules, note that the processing date has been updated since these granules were originally used in the covid-19 dashboard.

New granule IDs:

LC08_L1TP_019048_20200303_20200822_02_T1
LC08_L1TP_026031_20200304_20200822_02_T1

[moradology]

Items are up on dev, here: https://dev-stac.delta-backend.com/collections/ls8-covid-19-example-data

[moradology]

As these are not following the standard ingest path (L8 doesn't fit nicely with the single asset strategy we've used so far) I'll include the item and collection json here as well
collection.ndjson:

{"id":"ls8-covid-19-example-data","type":"Collection","links":[],"title":"","extent":{"spatial":{"bbox":[[-96.72305087939858,14.643419735131204,-85.75967794165501,43.74584018603565]]},"temporal":{"interval":[["2020-03-03T00:00:00Z","2020-03-04T00:00:00Z"]]}},"license":"https://d9-wret.s3.us-west-2.amazonaws.com/assets/palladium/production/s3fs-public/atoms/files/Landsat_Data_Policy.pdf","description":"Selected Landsat-8 Level-1 Surface Reflectance data demonstrating usage of false composites used in analysis","stac_version":"1.0.0","dashboard:is_periodic":"false","dashboard:time_density":"null","item_assets":{"coastal_aerosol":{"type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Coastal Aerosol Band (B1)","description":"Collection 2 Level-2 Coastal Aerosol Band (B1) Surface Reflectance","roles":["data"]},"blue":{"type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Blue Band (B2)","description":"Collection 2 Level-2 Blue Band (B2) Surface Reflectance","roles":["data"]},"green":{"type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Green Band (B3)","description":"Collection 2 Level-2 Green Band (B3) Surface Reflectance","roles":["data"]},"red":{"type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Red Band (B4)","description":"Collection 2 Level-2 Red Band (B4) Surface Reflectance","roles":["data"]},"nir":{"type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Near Infrared Band (B5)","description":"Collection 2 Level-2 Near Infrared Band (B5) Surface Reflectance","roles":["data"]},"swir1":{"type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Shortwave Infrared Band 1 (B6)","description":"Collection 2 Level-2 Shortwave Infrared Band 1 (B6) Surface Reflectance","roles":["data"]},"cog_default":{"type":"image/tiff; application=geotiff; profile=cloud-optimized","roles":["data","layer"],"title":"Default COG Layer","description":"Cloud optimized default layer to display on map"}}}

items.ndjson:

{"type":"Feature","stac_version":"1.0.0","id":"ls8-water-quality-covid-19-iowa","properties":{"datetime":"2020-03-04T00:00:00.00Z","platform":"LANDSAT_8","instruments":["OLI","TIRS"],"proj:epsg":32615,"proj:shape":[7871,7761],"proj:transform":[30,0,212985,0,-30,2034915,0,0,1],"created":"2020-03-04T00:00:00Z","updated":"2020-08-22T00:00:00Z"},"geometry":{"coordinates":[[[-94.344496,40.683432],[-94.344496,42.808812],[-91.541914,42.808812],[-91.541914,40.683432],[-94.344496,40.683432]]],"type":"Polygon"},"links":[],"assets":{"coastal_aerosol":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_026031_20200304_20200822_02_T1_B1.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Coastal Aerosol Band (B1)","description":"Collection 2 Level-2 Coastal Aerosol Band (B1) Surface Reflectance","roles":["data"]},"blue":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_026031_20200304_20200822_02_T1_B2.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Blue Band (B2)","description":"Collection 2 Level-2 Blue Band (B2) Surface Reflectance","roles":["data"]},"green":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_026031_20200304_20200822_02_T1_B3.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Green Band (B3)","description":"Collection 2 Level-2 Green Band (B3) Surface Reflectance","roles":["data"]},"red":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_026031_20200304_20200822_02_T1_B4.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Red Band (B4)","description":"Collection 2 Level-2 Red Band (B4) Surface Reflectance","roles":["data"]},"nir":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_026031_20200304_20200822_02_T1_B5.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Near Infrared Band (B5)","description":"Collection 2 Level-2 Near Infrared Band (B5) Surface Reflectance","roles":["data"]},"swir1":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_026031_20200304_20200822_02_T1_B6.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Shortwave Infrared Band 1 (B6)","description":"Collection 2 Level-2 Shortwave Infrared Band 1 (B6) Surface Reflectance","roles":["data"]}},"bbox":[-94.3444955800593,40.683431600897165,-91.54191411030821,42.80881206985239],"stac_extensions":["https://stac-extensions.github.io/projection/v1.0.0/schema.json","https://stac-extensions.github.io/eo/v1.0.0/schema.json"],"collection":"ls8-covid-19-example-data","description":"Selected Landsat-8 Level-1 Surface Reflectance data for demonstrating the usage of the bathymetric band combination (4,3,1)."}
{"type":"Feature","stac_version":"1.0.0","id":"ls8-water-quality-covid-19-belize","properties":{"datetime":"2020-03-03T00:00:00.00Z","platform":"LANDSAT_8","instruments":["OLI","TIRS"],"proj:epsg":32616,"proj:shape":[7801,7641],"proj:transform":[30,0,212985,0,-30,2034915,0,0,1],"created":"2020-03-04T00:00:00Z","updated":"2020-08-22T00:00:00Z"},"geometry":{"coordinates":[[[-89.68542102058792,16.27178760146487],[-89.68542102058792,18.40349773681132],[-87.54710005104722,18.40349773681132],[-87.54710005104722,16.27178760146487],[-89.68542102058792,16.27178760146487]]],"type":"Polygon"},"links":[],"assets":{"coastal_aerosol":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_019048_20200303_20200822_02_T1_B1.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Coastal Aerosol Band (B1)","description":"Collection 2 Level-2 Coastal Aerosol Band (B1) Surface Reflectance","roles":["data"]},"blue":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_019048_20200303_20200822_02_T1_B2.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Blue Band (B2)","description":"Collection 2 Level-2 Blue Band (B2) Surface Reflectance","roles":["data"]},"green":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_019048_20200303_20200822_02_T1_B3.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Green Band (B3)","description":"Collection 2 Level-2 Green Band (B3) Surface Reflectance","roles":["data"]},"red":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_019048_20200303_20200822_02_T1_B4.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Red Band (B4)","description":"Collection 2 Level-2 Red Band (B4) Surface Reflectance","roles":["data"]},"nir":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_019048_20200303_20200822_02_T1_B5.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Near Infrared Band (B5)","description":"Collection 2 Level-2 Near Infrared Band (B5) Surface Reflectance","roles":["data"]},"swir1":{"href":"s3://veda-data-store-staging/l8-water-quality/LC08_L1TP_019048_20200303_20200822_02_T1_B6.TIF","type":"image/vnd.stac.geotiff; cloud-optimized=true","title":"Shortwave Infrared Band 1 (B6)","description":"Collection 2 Level-2 Shortwave Infrared Band 1 (B6) Surface Reflectance","roles":["data"]}},"bbox":[-89.68542102058792,16.27178760146487,-87.54710005104722,18.40349773681132],"stac_extensions":["https://stac-extensions.github.io/projection/v1.0.0/schema.json","https://stac-extensions.github.io/eo/v1.0.0/schema.json"],"collection":"ls8-covid-19-example-data","description":"Selected Landsat-8 Level-1 Surface Reflectance data for demonstrating the usage of the agriculture band combination (6, 5, 2)"}

[moradology]

@danielfdsilva @nerik Have a look at the state of these ingested records and let me know if there's anything further I can do to make them useful.
dev: https://dev-stac.delta-backend.com/collections/ls8-covid-19-example-data
staging: https://staging-stac.delta-backend.com/collections/ls8-covid-19-example-data

[danielfdsilva]

@moradology There are a couple of properties that we need on the API response:

• summaries with the datetime extent (no2-monthly as an example)
• the dashboard:time_density should be one of year|month|day
• the value of dashboard:is_periodic should be a boolean instead of a string

cc @anayeaye

[anayeaye]

@danielfdsilva I converted those strings to be json false and null in both dev and staging databases and reran the summaries so you now have a list of the two datetime instances in the collection summaries.

[danielfdsilva]

@anayeaye The time density should not be null. Looking at the summaries, this looks like it is a day dataset

[anayeaye]

@danielfdsilva updated, time density is now day.