Test Transfer DAG endpoint in SM2A

[smohiudd]

Test Transfer DAG Endpoint to be used in github ingestion automation: NASA-IMPACT/veda-data#180

Example from event bridge: https://github.com/NASA-IMPACT/veda-data-airflow/blob/dev/sm2a/infrastructure/functions/s3_event_bridge_to_sfn_execute/lambda_function.py#L52

[smohiudd]

I tried the transfer DAG in SM2A staging using this config and it didn't work - no files in the veda-data-store bucket:

{
    "origin_bucket": "veda-data-store-staging",
    "origin_prefix": "2024tornadoes/",
    "target_bucket": "veda-data-store",
    "collection": "tornadoes-2024-polygons",
    "filename_regex": "(.*)Tornado_Polygons_(.*).tif$"
}

But the transfer DAG showed as succeeded:

I tried the same config in MWAA staging and it worked.

@amarouane-ABDELHAK @anayeaye

[anayeaye]

Resolved by adding role arns to the airflow configuration secret #260. Deleted the file in the example above and re-ran the transfer dag which successfully copied the file to veda-data-store