Skip to content

Writing wrappers for software

Jump to bottom
Mike Hucka edited this page Nov 11, 2019 · 21 revisions

The SBML Test Runner is designed to run target software (which can be, for example, a simulation library) and pass it command-line arguments to indicate the test cases that the software should run. Since individual software tools are rarely designed to accept the same command-line arguments as the SBML Test Suite provides, it is usually necessary to write some kind of interface program or wrapper to act as an intermediary between the SBML Test Suite and the software being tested. The SBML Test Runner will invoke the application wrapper with specific arguments for each test case, and the wrapper's job is to invoke the to-be-tested application in whatever way is necessary to make things work for that application. The wrapper is responsible for making the application read an SBML file for a given test case, simulate the SBML model with specific simulation settings, and write an output file containing the numerical results of the simulation in a comma-separated value (CSV) format.

Configuring the SBML Test Suite to use a new application wrapper

When you first start the Runner, you will not have any wrapper configurations defined except for one pseudo-wrapper definition named -- no wrapper --. This is a "view-only" wrapper; it allows you to view the test cases provided in the SBML Test Suite, and nothing more. To let you test an actual SBML-compatible application, the SBML Test Runner will open a Preferences panel to let you define one or more wrapper configurations. The following image shows the wrapper configuration part of the preferences panel:

Each configuration has:

  1. A name
  2. The path to the wrapper program on your computer
  3. The path to a directory where the Test Runner will find the application's output after running a test case
  4. A list of test components or test tags that the application is known to be unable to understand
  5. Command-line arguments that should be passed to the wrapper

The SBML Test Runner provides you with the ability to specify a complete command line invoking the wrapper. In the command-line arguments given to the wrapper, the following substitution codes can be used. Their values will be substituted each time the wrapper is invoked:

  • %d = the path to the directory containing all test cases
  • %n = the current test case number (as a 5-digit number)
  • %o = the directory where the CSV output file should be written
  • %l = the SBML Level of the test case
  • %v = the Version of SBML within the SBML Level of the test case

The SBML Test Runner will set the specific values; they are not under user control. However, the order in which the arguments are handed to the wrapper is under your control, by writing the arguments in the desired order in the Arguments to wrapper field. You could also omit some of the arguments if they are unnecessary for your particular wrapper.

Each test case consists of an SBML file and a settings file. The files are located in the directory named %d/%n. The directory indicated by %d will contain a large number of subdirectories named after the test case number (i.e., 00001, 00002, 00003, etc. – these are the values that %n will take on). Inside each of these numbered directories, there will be multiple SBML files (for different SBML Level/Version combinations), a settings file (named %n-settings.txt), and other files.

You will need to create a wrapper such that it performs the following steps:

  1. Extracts the relevant simulation run settings from the file %d/%n/%n-settings.txt. These settings include the starting time of the simulation, the duration of the simulation, the variables whose values should appear in the output, the number of output steps to record in the output, and the tolerances to use.

  2. Tells the to-be-tested application to (i) read an SBML file named %d/%n/%n-sbml-lXvY.xml, where X is the SBML Level, and Y is the Version within the Level, (ii) execute a simulation with the settings determined in step (1), and (iii) write the output as a file named %o/%n.csv. The command-line arguments to be handed to the software depend on the software itself.

You can define multiple wrappers, but only one will be executed during any given test run. (You will be able to choose the wrapper from a pull-down menu in the main window of the SBML Test Runner.)

The definition of a wrapper also includes three options, which can be seen in the image above:

  • Pseudo-wrapper to view test cases only: When set, this option defines a wrapper as non-executable and disables some of the fields in the preferences panel. Use this option to define a wrapper for viewing existing results (e.g., results created in some other fashion than through the SBML Test Runner). The wrapper definition allows you to define the directory where the results will be found, as well as unsupported tags, but not program path or arguments.

  • Wrapper can handle any SBML Level/Version: Some applications may not be able to accept any SBML Level and Version.

  • Wrapper can be run in parallel: The Runner can execute tasks in parallel, but it must be informed whether multiple instances of a wrapper or application can be invoked simultaneously. Use this flag to indicate that multiple copies of the wrapper can be started simultaneously. Do not enable this option if the wrapper (or application) writes to the same file (e.g., a single log file), locks a unique resource, or does something else that would result in non-deterministic behavior if multiple copies of the wrapper or application are started simultaneously. If you use a shell script, also make sure that the script does not return before the application you're testing returns a result. (In other words, do not have the shell script start the application as a background process; make sure the script terminates only when the application itself terminates.)

Important tips for writing wrappers

In some operating system environments, when you start the SBML Test Runner application by (e.g.) double-clicking its icon in your GUI file viewer, the Test Runner process will not have the same environment as what you get in a terminal shell: it will instead have a basic environment with different values for environment variables such as PATH. Consequently, when the Test Runner invokes a wrapper as a subprocess, the wrapper will execute in a different process environment than what it would have if run manually in a terminal shell.

If your application is entirely self-contained, such as a statically-linked binary, then this may not matter. On the other hand, if your wrapper or your application is something that requires things from the user's unique environment (for example, a Python script that loads packages the user has installed in their own Python library path), then it will likely fail to work as expected.

On Unix-like systems (macOS, Linux), one solution is to write a Bourne shell script that explicitly loads the user's regular initialization files before proceeding. Here are the first few lines of such a script:

#!/bin/sh

# On MacOS and possibly other OSes, a non-login /bin/sh does not read the
# user's .profile, which means this script will not get the user's environment
# when it is run by SBML Test Runner and, consequently, will probably fail to
# work as expected.  The following few lines try to compensate by at least
# reading ~/.profile or ~/.bash_profile, as appropriate.
if [ ! -n "$BASH" ]; then
    if [ -f $HOME/.bash_profile ]; then
        . $HOME/.bash_profile
    fi
else
    if [ -f $HOME/.profile ]; then
        . $HOME/.profile
    fi
fi

# .... rest of the wrapper

If you wish to write the wrapper in Python instead, you will face another problem: the possibility that the default process environment of the Test Runner is unable to find the desired copy of Python. For example, on some systems, /usr/bin/python is Python 2, and Python 3 is elsewhere such as /usr/local/bin or /opt/local/bin, but a basic non-user non-login process may not have those directories in its $PATH. Consequently, writing a generic hash-bang line for your Python script may be difficult. A solution to this problem is to employ a trick of syntax that works in both Bournce/Bash shells and Python. The following polyglot script illustrates this for the case of running the script in Python 3 after loading the user's normal shell environment configuration files (as was done in the example above):

#!/bin/sh
# When this file is executed initially, Bourne shell will executed the lines
# below and exec Python on this whole file.  Python will then ignore the 
# following lines because, to Python, the following is just a quoted string.

''':'
# On MacOS and possibly other OSes, a non-login /bin/sh does not read the
# user's .profile, which means this script will not get the user's environment
# when it is run by SBML Test Runner and, consequently, will probably fail to
# work as expected.  The following few lines try to compensate by at least
# reading ~/.profile or ~/.bash_profile, as appropriate.
if [ ! -n "$BASH" ]; then
    if [ -f $HOME/.bash_profile ]; then
        . $HOME/.bash_profile
    fi
else
    if [ -f $HOME/.profile ]; then
        . $HOME/.profile
    fi
fi

# Now start Python on this file.
if type python3 >/dev/null 2>/dev/null; then
  exec python3 "$0" "$@"
else
  exec python "$0" "$@"
fi
'''

# The rest of this file should contain the desired Python code.

Tips for debugging wrappers

The SBML Test Runner provides a way to see the output from running a wrapper. It is the option View Process Output in the Test menu. Click on the number of a test case in the left-hand column of the main window, run the test, and then invoke the menu item:

To make use of this, the wrapper has to produce some diagnostic output in addition to doing the work of running the test case and saving the simulation output. Typically, one proceeds by writing the wrapper in such a way that it accepts an optional debugging flag, such that if the wrapper is executed with the debugging option turned on, it prints some pertinent information (whatever that may be for a given wrapper) while it is running. To pass the option to the wrapper to enable debugging, add the relevant command-line flag to the "Arguments to wrapper" field in the wrapper configuration panel (see above). For example, suppose you write your wrapper to accept an optional -d flag on the command-line. Then you might configure the wrapper's "Arguments to wrapper" field with the line -d %d %n %o %l %v instead of simply %d %n %o %l %v.

Clone this wiki locally