Skip to content
oweidner edited this page Jul 19, 2012 · 22 revisions

This document describes the roadmap and goals for the SAGA-BlisS testing framework, including concrete work-items and the current progress.

People Involved

The people working on BlisS testing are:

Components

There are three components that define the BlisS testing framework

  • The unit tests (code) are supposed to ensure that the API behaves as expected. The unit tests try cover all classes and methods of the Bliss API with both, valid input (positive testing) as well as invalid input (negative testing). While the unit-tests don't focus on plug-in testing, most unit tests will inevitably trigger a plug-in.

  • The compliance tests (code) the compliance tests are supposed to ensure that all plug-ins implement the same semantics of the API. For example, the PBS plug-in and the SGE plug-in should display the exact same behavior for a given job submission code. The compliance tests do not necessarily aim to cover the whole spectrum of classes and methods, but rather a relevant spectrum of real-world use-cases.

  • The buildbot service (link) executes both, the unit tests and the compliance tests on different local and remote platforms with a focus on remote capabilities like job submission and file transfer. Buildbot services are supposed to be triggered by git commits and send results to the mailing list. However, that feature is currently disabled.

Roadmap

Overview

Rigorous testing is absolutely crucial for a protocol-converter like SAGA/BlisS. Besides having a functioning API, it is absolutely mandatory that all plug-ins behave sematnically equivalent in any given situation, otherwise the concept of a unified API is pretty worthless. Unfortunately, the SAGA standard (GFD.90) spans many different topics i.e., jobs, files, replica, ...), is rather complex and extensive, and leaves quite some room for interpretation. The tests try make the SAGA BlisS implementation as homogeneous as possible.

The testing frameworks described above have been growing with BlisS since its initial release. While they are working quite nicely, there is lots of room for improvement on all three fronts: unit tests, compliance tests and buildbot. Here we describe the roadmap and goals for the thee components.

1. Unit Tests

1.1 Audit: Assessment of Test and API Coverage

Tests have been developed alongside the API, but currently we don't know what is covered by the tests and what not. A document needs to be created that maps API classes / methods to test cases. Since this involves going through the API documentation class-by-class and method-by-method, two other tasks can and should be carried out in this context: (1) Documentation assessment (are all method and all parameters documented properly?) and (2) GFD-90-copliance assessment (are all methods and class signatures identical with the signatures in GFD.90?).

Input documents/files for this work-item:

Output documents/files for this work-item:

The audit results will be collected in a spreadsheet that contains at least the following columns:

  • Classname.methodname (Based on gfd.90)
  • Existis in BlisS? (If yes, filename & path)
  • Compliant with GFD.90?
  • Documentation complete?
  • Covered by unit tests? (If yes, test case name(s), e.g., 'test_job_wait()')
  • Comments (e.g., what is different from GFD.90, etc)

Procedure

A spreadsheet is created in Excel format and made available through the BlisS GitHub repository. As a first step, all GFD.90 class and method signatures for the Job and the File API packages as well as for the Error, Exception, Session, and Url classes should be written down in the first column, i.e.,

  • job.service.create_job(params)
  • job.service.list(params)

Once that is done, the remaining columns are populated by looking at the API documentation (online) and the source files of the unit tests. This document is crucial for all subsequent test-imporvement work-items!

Expected Duration

3-4 weeks

2. Compliance Tests

3. Buildbot Service