Migrate all cron jobs to dagster on Leonardo

[devsjc]

There remain a few cron jobs that want to be moved to dagster jobs. These probably want to be launched using the docker executor to run the consumer and pull the desired files. The required tasks are as follows:

• ICON Global to Dagster
• ICON EU to Dagster
• SILAM Global to Dagster
• MeteoFrance Global to Dagster
• MeteoFrance Europe to Dagster
• GPDS Global to Dagster

Many of these will also require getting working setups in the nwp consumer creating the archives and pushing them to huggingface.

For reference: Jacob's crontab:

``` #00 06 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "00" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global/" --delete #00 10 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "06" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global/" --delete #00 16 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "12" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global/" --delete #00 22 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "18" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global/" --delete #00 06 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "00" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU/" --delete #00 10 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "06" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU/" --delete #00 16 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "12" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU/" --delete #00 22 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "18" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU/" --delete #00 04 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "00" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global1/" --delete #00 11 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "06" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global1/" --delete #00 17 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "12" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global1/" --delete #00 23 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "18" --model "global" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_Global1/" --delete #00 04 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "00" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU1/" --delete #00 11 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "06" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU1/" --delete #00 17 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "12" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU1/" --delete #00 23 * * * source /home/jacob/miniconda3/bin/activate /home/jacob/miniconda3/envs/ocf/envs/modal && python /home/jacob/nwp/nwp/icon/app.py --run "18" --model "eu" --folder "/mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/NWP/DWD/ICON_EU1/" --delete 00 12 * * * aws s3 sync --no-sign-request s3://fmi-opendata-silam-surface-netcdf/ /mnt/storage_c/SILAM/ 00 12 * * * aws s3 sync --no-sign-request s3://fmi-opendata-silam-surface-zarr/ /mnt/storage_c/SILAM/ 00 23 * * * aws s3 sync --no-sign-request s3://mf-nwp-models/ /mnt/storage_c/MeteoFrance/ 00 04 * * * wget -r https://dd.weather.gc.ca/model_gem_global/15km/grib2/lat_lon/ --no-parent -m -P /mnt/storage_c/GPDS/ 00 17 * * * wget -r https://opendata.dwd.de/weather/nwp/icon/grib/ --no-parent -m -P /mnt/storage_b/DWD_Global_Backup/ 00 10 * * * wget -r https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/ --no-parent -m -P /mnt/storage_c/DWD_EU_Backup/ 00 12 * * * wget -r https://opendata.dwd.de/weather/nwp/icon-eu/grib/06/ --no-parent -m -P /mnt/storage_c/DWD_EU_Backup/ 00 18 * * * wget -r https://opendata.dwd.de/weather/nwp/icon-eu/grib/12/ --no-parent -m -P /mnt/storage_c/DWD_EU_Backup/ 00 22 * * * wget -r https://opendata.dwd.de/weather/nwp/icon-eu/grib/18/ --no-parent -m -P /mnt/storage_c/DWD_EU_Backup/ 00 01 * * * wget -r https://opendata.dwd.de/weather/nwp/icon/grib/ --no-parent -m -P /mnt/elements3/DWD_Global_Backup/ 00 17 * * * wget -r https://dd.weather.gc.ca/model_gem_global/10km/ --no-parent -m -P /mnt/storage_c/GPDS/10km/ 00 06 * * * rm -rf /mnt/storage_c/MeteoFrance/arome* 00 00 * * * wget -r https://opendata.dwd.de/weather/nwp/icon-art-eu/ --no-parent -m -P /mnt/storage_b/nwp/icon/art/eu/ 00 00 * * * wget -r https://opendata.dwd.de/weather/nwp/icon-art/ --no-parent -m -P /mnt/storage_b/nwp/icon/art/global/ ```