Regional Inverse Modelling

• Installation
  • Setting up the enviroment
• Overview
  • Plugins
  • Wrokflow structure
• Using the terminal
  • Checking codes and computers
  • Deleting groups
  • Monitoring processes
  • Generating provenance graphs
  • Enable caching
• Development guide
  • Repository contents
  • Setting up the enviroment
  • Changing widgets layout
  • Changing the query
• Contact

Installation

Install Docker in your local machine and start a new container with:

docker run -p 8888:8888 aiidalab/full-stack

Check Aiidalab user guide for more details on how to use and configure AiiDAlab. Once inside AiiDAlab, the app can be Downloaded from the app list in the App Store.

If installation was succesfull the app should look like this:

Make sure to check the first-time user's guide inside Submit FLEXPART-COSMO/IFS simulations

For an easy and quick CSCS connection, install the following app from the App Sotre:

MFA for CSCS

Additionally, to monitor the processes we recommend to install: AiiDAlab Base Widgets

Setting up the enviroment

To make changes in the app follow the next steps. Inside the container, navigate to /app and clone the repository.

cd app
git clone https://github.com/C2SM/aiidalab-flexpart.git

Necessary intallations to run the aiidalab flexpart app:

pip install aiidalab_widgets_base
pip install aiida-shell

The next step is to install aiida-flexpart plugin, which can be done in two ways: Using pip to install the plugin directly (recommended if changes to the plugin are not intended):

pip install aiida-flexpart

Alternatively, if one whises to make changes in the AiiDA flexpart plugin as well, navigate to /work and copy the following code.

git clone https://github.com/aiidaplugins/aiida-flexpart.git
cd aiida-flexpart
pip install -e .

Inversion workflow overview

Plugins

An AiiDAlab app to run FLEXPART COSMO/IFS simulations and inversions. The app uses the aiida-flexpart plugin:

• flexpart.cosmo: FLEXPART COSMO transport model plugin
• flexpart.ifs: FLEXPART IFS transport model plugin
• flexpart.post: Post-processing plugin of the transport model relsults
• collect.sensitivities
• inversion.calc

A data plugin:

• netcdf.data: NetCDF data plugin

and 3 workflows:

• flexpart.multi_workflow: A workflow that can be used to submit FLEXPART calculations for multiple days
• inspect.workflow: Workflow for importing external NetCDF files (obesrvations or sensitivities)
• inversion.workflow: Workflow to submit invesions

Workflow structure

The following is the workflow structure for the transport model. It will loop over all the given dates. If model offline is not none, integration_time_offline should be greater than zero. The available models for cosmo are: cosmo7, cosmo1 and kenda1. And the ECMWF models: IFS_GL_05, IFS_GL_1, IFS_EU_02 and IFS_EU_01. Both, model and model offline can be set as a list of the previous.

   graph TD;
      id1{MODEL}--cosmo models-->PREPARE_COSMO_METEO_FILES;
      id1{MODEL}--ecmwf models-->PREPARE_IFS_METEO_FILES;

      subgraph  -
      PREPARE_COSMO_METEO_FILES-->RUN_FLEXPART_COSMO;
      end
      subgraph IFS
      PREPARE_IFS_METEO_FILES-->RUN_FLEXPART_IFS;
      end

      RUN_FLEXPART_COSMO-->id2{MODEL_OFFLINE};
      id2{MODEL_OFFLINE}--ecmwf models-->PREPARE_IFS_METEO_FILES;
      id2{MODEL_OFFLINE}--none -->POST-PROCESSING;
      RUN_FLEXPART_IFS-->POST-PROCESSING;

Loading

The second step deals withthe various results of the post-processings produced previously.

   graph TD;

      POST-PROCESSING_1 -->COLLECT_SENSITIVITIES;
      POST-PROCESSING_2 -->COLLECT_SENSITIVITIES;
      POST-PROCESSING_3 -->COLLECT_SENSITIVITIES;
      POST-PROCESSING_4 -->COLLECT_SENSITIVITIES;

      COLLECT_SENSITIVITIES --> RESULTS(NetCFD);

Loading

Next:

   graph TD;

      NetCFD_1-- external --> INVERSION;
      NetCFD_2-- observations --> INVERSION;
      NetCFD_3-- FLEXPART --> INVERSION;

      INVERSION --> RESULTS;

Loading

Using the terminal

More info in AiiDA

Checking codes and computers

To see the list of computers, type in the terminal:

verdi computer list

To see the configuration of a specific computer:

verdi computer show <NAME>

Similarly, to check the installed codes:

verdi code list

Example of result:

Full label                           Pk  Entry point
---------------------------------  ----  -------------------
flexpart_cosmo@daint                  1  core.code.installed
flexpart_ifs@daint                    2  core.code.installed
post-processing@daint                 3  core.code.installed
check-cosmo-data@daint-direct-106     4  core.code.installed
check-ifs-data@daint-direct-106       5  core.code.installed

To see the code configuration:

verdi code show <PK>

Deleting groups

Check AiiDA: organizing data for more information about groups. This app defines two groups: one for locations and another one for the outgrids:

verdi group list

Should display:

  PK  Label      Type string    User
----  ---------  -------------  ---------------
  12  locations  core           aiida@localhost
  13  outgrid    core           aiida@localhost

To delete a group:

verdi group delete <PK>

Monitoring processes

The worflow can be monitored through the terminal using

verdi process list

or

verdi process report <PK>

To check the status of terminated jobs use:

verdi process list -a

Generating provenance graphs

verdi node graph generate <PK>

The graph in .pdf format will be stored in the working directory, go to File manager to view or donwload. Check Visualizing graphs for more information about provenance graphs. To create more artistic graphs, check aiida-graph-render.

Enable caching

Make sure caching is enabled by typing:

verdi config list caching

if False, enable it with:

verdi config set caching.default_enabled True

Development guide

Repositori content

• config/: Yaml files with default inputs
  • command.yaml: COMMAND file inputs
  • input_phy.yaml: INPUT_PHY file inputs
  • location_groups.yaml: Groups of locations
  • locations.yaml: List of default locations
  • meteo_inputs.yaml:
  • models.yaml: Model and offline model lists
  • outgrid.yaml: Default outgrids
  • releases.yaml: RELEASES file inputs
  • stash_files.yaml: List of files to stash for each calculations
• img/: Empa logo
• static/: CSS style file
• utils/:
  • default_codes.yaml: temporal sotrage of default codes and stsh address
  • make_query.py: FLEXPART querybuilder funtions
  • utils.py: utility fucntions
• widgets/: scripts with the widgets for different inputs
  • add_location.py/:
  • add_outgrid.py/:
  • basic.py/: basic inputs for FLEXPART simulations
  • code_setup.py/: code and stash address resources
  • command.py/:
  • filter.py/:
  • import_nc.py/: widgets and builder for importing external .nc flies
  • input_phy.py/:
  • inversion_params.py/: widgets for step 2 inversion inputs
  • locations.py/:
  • misc.py/: miscelaneous widgets for computational resources
  • outgrid.py/:
  • presetings.py/:
  • query.py/: FLEXPART results query
  • releases.py/:
  • sens_query.py/: observations and sensibilities query widgets
  • stack.py/:
• info.ipynb: Markdown guide to set up and configure the app for first time users
• inversion_query.ipynb: Inversion results search page
• main.ipynb: FLEXPART simulation submision page
• ncdump.ipynb: Displays NetCDF file as a ncdump
• plot.ipynb: Plot .png files from FLEXPART recovered results
• query.ipynb: Search page for FELXPART results and submitting sensitivity collection calc
• sens_query.ipynb: page for importing NetCDF files, and submitting inversions
• settings.py: Plugins used in the app
• setup.cfg: app setup file

Changing widgets layout

We refer to Box,Hbox and VBox for a guide on how to change the layout structure of the widgets.

Changing the query

For changes in the querybuilder check: AiiDA database. The query for the FLEXPART simulations can be found in uitls/make_query.py, and for the inversion in inversion_query.ipynb

Contact

[email protected]