stochastic

The SBML Test Suite –– Stochastic Test Cases

The tests in this directory are of two types: tests that check that a stochastic simulation of the reactions in the SBML models are correct (with the testType StochasticTimeCourse), and tests that check that assignments drawn from distributions (from the 'Distributions' package) are correct (with the testType StatisticalDistribution).

The first 39 tests are all of type StochasticTimeCourse and were drawn from the SBML Discrete Stochastic Model Test Suite (DSMTS), developed and contributed by Thomas Evans, Colin Gillespie and Darren Wilkinson. Each test case consists of an SBML model intended for simulation in a discrete stochastic regime; the models have been solved either analytically or using numerical methods, and the expected time course data, together with expected means and standard deviations of model species quantities, are provided for each test case. The combination of models and known results may be used to test the behavior of SBML-compatible stochastic simulation systems.

The StatisticalDistribution tests were developed by Lucian Smith to test implementations of the Distributions package. They are included here because the same basic testing structure (repeated tests where the means and standard deviations are compared to expectations) is required.

---

Main Authors: Thomas W. Evans^a, Colin S. Gillespie^b, Darren J. Wilkinson^b, Lucian P. Smith^c,d

Institutions:
      ^a University of Liverpool, Liverpool, UK
      ^b Newcastle University, Newcastle, UK
      ^c California Institute of Technology, Pasadena, CA, US
      ^d University of Washington, Seattle, WA, US

Repository: https://github.com/sbmlteam/sbml-test-suite

Bug tracker: https://github.com/sbmlteam/sbml-test-suite/issues

Developers' discussion group: https://groups.google.com/forum/#!forum/moccasin-dev

Pivotal tracker: https://www.pivotaltracker.com/n/projects/68714

Please cite the DSMTS paper!

If you use this test suite, please cite the following paper:

Evans, T. W., Gillespie, C. S., Wilkinson, D. J. The SBML discrete stochastic models test suite, Bioinformatics, 24:285-286, 2008.

Please also indicate the specific version of the SBML Test Suite you use, to improve other people's ability to reproduce your results.

Organization of folders and files

The directories and files have been renamed from their original versions according to the conventions used elsewhere in the SBML Test Suite, but many of the files in each subdirectory are still named as they were originally so that the correspondences between this and the DSMTS should be easy to identify.

Here are the files in each directory (where N, Y and Z are digits):

• NNNNN-model.m – model description
• NNNNN-plot.html – interactive plot of simulation results
• NNNNN-plot.jpg – image of plot of simulation results
• NNNNN-results.csv – SBML Test Suite-style results file
• NNNNN-sbml-l2v1.xml – SBML Level 2 Version 1 model file
• NNNNN-sbml-l2v2.xml – SBML Level 2 Version 2 model file
• NNNNN-sbml-l2v3.xml – SBML Level 2 Version 3 model file
• NNNNN-sbml-l2v4.xml – SBML Level 2 Version 4 model file
• NNNNN-sbml-l2v5.xml – SBML Level 2 Version 5 model file
• NNNNN-sbml-l3v1.xml – SBML Level 3 Version 1 model file
• NNNNN-sbml-l3v2.xml – SBML Level 3 Version 2 model file
• NNNNN-settings.txt – SBML Test Suite settings file
• dsmts-YYY-ZZ.mod – original model definition
• dsmts-YYY-ZZ-mean.csv – means of the simulation results
• dsmts-YYY-ZZ-sd.csv – standard deviations of the results

The files NNNNN-sbml-l3v1.xml are renamed versions of the original model files, which had names of the form dsmts-YYY-ZZ.xml. One slight change was made for each: all compartments were given a spatialDimensions attribute so that they could be translated to SBML Level 2.

The files NNNNN-sbml-l2vN.xml are translated versions of the original SBML Level 3 version of the model.

The file NNNNN-results.csv is a new file added for the purposes of the SBML Test Suite. NNNNN-results.csv is simply a combination of the mean and standard deviation results files that are separated in the DSMTS; the other parts of the SBML Test Suite use a single results file, so we simply combined the DSMTS results into one file.

The file NNNNN-model.m is a new file added for the purposes of the SBML Test Suite. It mirrors the model file in the semantic test suite, with the same overall format, same suite of component tags and test tags, and same overall description. The only difference is that the testType listed for all stochastic test suite models is StochasticTimeCourse.

The file NNNNN-settings.txt is a new file added for the purposes of the SBML Test Suite. It mirrors the settings files in the semantic test suite, with a few differences:

• The absolute and relative settings are unused, as the method used to calculate a stochastic test success or failure is very different from that of a normal semantic test suite simulation.

• A new output setting is used, which lists all the X-mean and X-sd entries present in the NNNNN-results.csv file. The relevant output model variables are still listed in the variables and amount settings – output appends both -mean and -sd to those variables.

• New meanRange and outputRange settings are included, to be used in assessing a stochastic run's success or failure (see below).

The files NNNNN-plot.* are plots of the simulation results. The HTML version of the file is interactive; mousing over the plot lines brings up a dynamic display of the value at that point in the plot. The JPG image is a static version of the plot.

Finally, the files whose names begin with dsmts- are the original files from the DSMTS created by Evans, Gillespie and Wilkinson.

Getting started

For all tests, the stochastic simulations of these models should be run n times, where n is at a minimum 1,000, but more reasonably set to 10,000 for repeated tests, and which will need to be 100,000 or 1,000,000 to detect more subtle implementation errors. Once the tests have been run, the NNNNN-settings.txt file will contain one or more of the following entries, where X is the name of the variable in question:

• X-mean: The average value of X for each time point t (X_t)
• X-sd: The standard deviation of the set of X values for each time point t (S_t)
• ln(X)-mean: The average value of the natural log of X for each time point t (ln(X)_t)
• X-sd: The standard deviation of the set of the natural log of X values for each time point t (S_{ln, t})

Any requested calculated values are then compared with expected values found in the NNNNN-results.csv files, with the same column titles as above. If we designate the expected means as mu_t, and the expected standard deviations as sigma_t, we then calculate Z (if X-mean or ln(X)-mean are in the expected output):

Z_t = sqrt(n) * (X_t - mu_t)/sigma_t

and Y (if X-sd or ln(X)-sd are in the expected output):

Y_t = sqrt(n/2) * (S_t²/sigma_t² - 1)

Z_t should fall in the range meanRange from the settings file (often the range (-3,3)).

Y_t should fall in the range sdRange from the settings file (often the range (-5,5)).

Due to the nature of stochastic simulation, a correct simulator will still occasionally fail a test or two here or there, especially when multiple tests are being performed. Z or Y failing 0-1 times per test is great, and failing 2-3 times should probably be checked to see if this is consistent, and/or at the same time points.

Note that for some tests, X-sd is provided in XXXX-results.txt but not requested in XXXXX-settings.txt. This is because the expected standard deviations (sigmas) are requried to calculate Z, but the distribution in question has no expected range for the standard deviations themselves.

More details about the first 39 tests are included in the document DSMTS-userguide-31v2.pdf included in this directory.

(Lack of) Integration with the rest of the SBML Test Suite

As of this writing, the DSMTS is not integrated into the SBML Test Suite Test Runner or the database of test results at http://sbml.org/Facilities/Database. The DSMTS is distributed with the SBML Test Suite in order to encourage developers to begin investigating how they might be able to use it, but users and developers need to develop their own approaches to running the tests in software.

Please see the user guide in the file DSMTS-userguide-31v2.pdf in this directory for more information about the DSMTS.

History

Originally developed circa 2007-2010. The SBML Level 3 Version 1 version of DSMTS (dated October 21, 2010) was added to the SBML Test Suite distribution in 2015.

License and distribution terms

The DSMTS is licensed under the GNU LGPL by the original authors (Evans, Gillespie, Wilkinson).

More information

More information about the SBML Test Suite is available online at http://sbml.org/Software/SBML_Test_Suite.