Promote tornado collections to production

[anayeaye]

What

This issue is to collect the remaining data tasks required to publish the tornado vectors-as-cog datasets to the earthdata dashboard.

Blocked PR veda-config/Initial push of Spring 2024 tornados #420

Tasks

• Finish validating and updating tornado collections in staging backend
• Transfer files to production
• Publish collections to production
• Open veda-data PR with validated publication configuration

Raw config info (edit and publish in staging)

tornadoes-2024-polygons

{
  "collection": "tornadoes-2024-polygons",
  "title": "NWS Tornado Tracks from MAM 2024",
  "description": "Official NWS surveyed tornado tracks from March to May of 2024",
  "license": "CC0-1.0",
  "is_periodic": true,
  "time_density": "day",
  "spatial_extent": {
    "xmin": -130,
    "ymin": 17.6,
    "xmax": -57,
    "ymax": 52.5
  },
  "temporal_extent": {
    "startdate": "2024-03-01T00:00:00Z",
    "enddate": "2024-05-31T00:00:00Z"
  },
  "sample_files": [
    "s3://veda-data-store-staging/2024tornadoes/Tornado_Polygons_2024-05-31_cog.tif"
  ],
  "discovery_items": [
    {
      "discovery": "s3",
      "cogify": false,
      "upload": false,
      "dry_run": false,
      "prefix": "2024tornadoes/",
      "bucket": "veda-data-store-staging",
      "filename_regex": "(.*)Tornado_Polygons_(.*).tif$"
    }
  ],
  "data_type": "cog"
}

tornadoes-2024-paths

{
  "collection": "tornadoes-2024-paths",
  "title": "NWS Tornado Tracks (Paths) from MAM 2024",
  "description": "Official NWS surveyed tornado tracks from March to May of 2024. Only the centerline of the tornado paths, classified by maximum EF rating.",
  "license": "CC0-1.0",
  "is_periodic": true,
  "time_density": "day",
  "spatial_extent": {
    "xmin": -130,
    "ymin": 17.6,
    "xmax": -57,
    "ymax": 52.5
  },
  "temporal_extent": {
    "startdate": "2024-03-01T00:00:00Z",
    "enddate": "2024-05-31T00:00:00Z"
  },
  "sample_files": [
    "s3://veda-data-store-staging/2024tornadoes/Tornado_Tracks_2024-05-31_cog_retiled.tif"
  ],
  "discovery_items": [
    {
      "discovery": "s3",
      "cogify": false,
      "upload": false,
      "dry_run": false,
      "prefix": "2024tornadoes/",
      "bucket": "veda-data-store-staging",
      "filename_regex": "(.*)Tornado_Tracks_(.*).tif$"
    }
  ],
  "data_type": "cog"
}

Unblocked veda-config PR

Now veda-config/Initial push of Spring 2024 tornados #420 can be updated as needed and published to production

[acblackford]

@anayeaye The polygons dataset is ready for prod. The paths item still is not populating in the STAC despite no errors from the Workflows API upon ingestion.

[smohiudd]

Thumbnail for polygons and paths uploaded:

    "assets": {
        "thumbnail": {
            "title": "Thumbnail",
            "description": "Wedge tornado passing southeast of Wapakoneta, Ohio on March 14, 2024",
            "href": "https://thumbnails.openveda.cloud/tornado-2024-cover.png",
            "type": "image/jpeg",
            "roles": ["thumbnail"]
        }
    }

[smohiudd]

The tornado paths file Tornado_Tracks_2024-05-31_cog_retiled.tif is failing in build stac with this error:

{"error": "Read or write failed. /vsis3/veda-data-store-staging/2024tornadoes/Tornado_Tracks_2024-05-31_cog_retiled.tif, band 1: IReadBlock failed at X offset 364, Y offset 230: TIFFReadEncodedTile() failed."}

@acblackford do you have any details on how this COG was created?

[acblackford]

@smohiudd This COG was created the same way as the tornado polygons COG (which seems to be showing up correctly), using the GDAL translate method similar to the line below. I checked the COG in ArcGIS Pro after creation and it populates correctly there.

gdal_translate /path/to/input.tif /path/to/output.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW -co BLOCKXSIZE=512 -co BLOCKYSIZE=512

[smohiudd]

Hey @acblackford, do you have the original source file that you used to create the COG? I want to try recreating the COG and see if I'm getting the same error.

[anayeaye]

@smohiudd @acblackford if you do try recreating I'd recommend trying some different settings out too (you may have already had this in mind) gdal_translate -of COG -co COMPRESS=DEFLATE -a_nodata = <nodata value if known> <rawfilename>.tif <cogfilename>.tif

[smohiudd]

Thanks @anayeaye for the suggestion. I also did a validate of Tornado_Tracks_2024-05-31_cog_retiled.tif and got this warning:

The following warnings were found:
- The file is greater than 512xH or 512xW, it is recommended to include internal overviews

@acblackford I tried loading the file in QGIS and I'm getting the same error: band 1: IReadBlock failed at X offset 364, Y offset 230: TIFFReadEncodedTile()

[smohiudd]

I was able to rasterize the original vector files using this gdal command:

gdal_rasterize -a scale -dialect SQLITE -te -103.501720443 18.467974506 -66.701986698 46.592731627 -tr 0.005 0.005 -a_nodata -999 -at \
-sql "SELECT *, CASE efscale
    WHEN 'EFU' THEN 6 
    WHEN 'EF0' THEN 0 
    WHEN 'EF1' THEN 1 
    WHEN 'EF2' THEN 2 
    WHEN 'EF3' THEN 3 
    WHEN 'EF4' THEN 4 
    WHEN 'EF5' THEN 5 
    ELSE -999 END as scale FROM Tornado_Tracks" \
Tornado_Tracks.shp Tornado_Tracks.tif

I wasn't able to pass -of COG -co COMPRESS=DEFLATE so the file will need to be converted to a COG seperately.

gdal_translate -of COG -co COMPRESS=DEFLATE -a_nodata -999 Tornado_Tracks.tif Tornado_Tracks_cog.tif

[smohiudd]

I believe the colormap is causing WAF issues in production:

https://openveda.cloud/api/raster/collections/tornadoes-2024-polygons/items/Tornado_Polygons_2024-05-31_cog/map?bidx=1&assets=cog_default&nodata=-999&colormap=%7B%220%22%3A+%22%23add8e6%22%2C+%221%22%3A+%22%2390ee90%22%2C+%222%22%3A+%22%23ffe71f%22%2C+%223%22%3A+%22%23ffa500%22%2C+%224%22%3A+%22%23ff0000%22%2C+%225%22%3A+%22%23ff00ff%22%7D",