Long range ECMWF india data

[devsjc]

In order to do a 7 day forecast for RUVNL, as well as the Satellite (openclimatefix/Satip#223) and Meteomatics () data, we need India ECMWF data to go out to a larger horizon. Currently it is limited to only go as far as the hourly incrementing steps, i.e. 90. We want to enable pulling up to 192 hours. This covers step increments of 1, 3, and 6 hours. We want 2022-2024 initially.

[Part of RUVNL]

[devsjc]

Closed with #139

[devsjc]

Whilst MARS listing requests were smart enough to handle requesting steps that did not exist upstream, MARS retrival errors out with the expected number of fields differring. As such, a safety net needs to be put in place to ensure that only exactly what is being pulled from MARS gets requested to begin with, even if it differs from what we are asking for. See #140

[devsjc]

Backfill in progress.

[devsjc]

19-04-2024

2022 fully downloaded. (50% done)

[devsjc]

24-04-2024

Still at 50%. I realised that the UK backfill (openclimatefix/dagster-dags#92) was now taking priority due to 2023 india being its own backfill with an incorrect priority tag (priority instead of dagster/priority). I have recreated the backfill with the corrected priority so that it takes precedence from the UK backfill.

[devsjc]

Moving the backfill part of the process to an issue in dagster-dags: see openclimatefix/dagster-dags#103.