SAGA Tutorial Part 2: Local Job Submission

This is part 2 of the SAGA Tutorial.

Overview / Background

One of the most important feature of SAGA Python is the capability to submit jobs to local and remote queueing systems and resource managers. The job submission and management capabilities of SAGA Python are packaged in the saga.job module (API Doc). Three classes are defined in this module:

• The job.Service class (API Doc) provides a handle to the resource manager, like for example a remote PBS cluster.
• The job.Description class (API Doc) is used to describe the executable, arguments, environment and requirements (e.g., number of cores, etc) of a new job.
• The job.Job class (API Doc) is a handle to a job associated with a job.Service. It is used to control (start, stop) the job and query its status (e.g., Running, Finished, etc).

Hands-On: Local Job Submission

Before we discuss the individual API call in more detail, let's get down and dirty and run the first example. Create a new file saga_example_1.py and paste the following code:

import sys
import bliss.saga as saga

def main():
    try: 
        # create a job service for lonestar
        js = saga.job.Service("fork://localhost")

        # describe our job
        jd = saga.job.Description()

        jd.environment     = {'MYOUTPUT':'"Hello from Bliss"'}       
        jd.executable      = '/bin/echo'
        jd.arguments       = ['$MYOUTPUT']
        jd.output          = "my1stjob.stdout"
        jd.error           = "my1stjob.stderr"

        # create the job (state: New)
        myjob = js.create_job(jd)

        print "Job ID    : %s" % (myjob.jobid)
        print "Job State : %s" % (myjob.get_state())

        print "\n...starting job...\n"
        # run the job 
        myjob.run()

        print "Job ID    : %s" % (myjob.jobid)
        print "Job State : %s" % (myjob.get_state())

        print "\n...waiting for job...\n"
        # wait for the job to either finish or fail
        myjob.wait()

        print "Job State : %s" % (myjob.get_state())
        print "Exitcode  : %s" % (myjob.exitcode)

    except saga.Exception, ex:
        print "An error occured during job execution: %s" % (str(ex))
        sys.exit(-1)

if __name__ == "__main__":
    main()

Save the file and execute it via the python interpreter (make sure your virtualenv is activated):

python saga_example_1.py

The output should look something like this:

Job ID    : [fork://localhost]-[None]
Job State : saga.job.Job.New

...starting job...

Job ID    : [fork://localhost]-[644240]
Job State : saga.job.Job.Pending

...waiting for job...

Job State : saga.job.Job.Done
Exitcode  : None

Once the job has completed, you can have a look at the output file my1stjob.stdout.

Note: Because you're working on a local system instead of submitting to a job cluster, the job state will immediately go to "Running" instead of "Pending." This is because your machine does not have to wait to start executing the job. In a similar way, the exitcode will most likely be "0" instead of "None." This is because your machine is actually returning "0" as the exitcode, whereas some SGE clusters won't return any exitcode at all.

More Details & Discussion

The following paragraph explains the building blocks and how to use the SAGA job package.

In order to use the Bliss Job API in your scripts, we first need to import the Bliss module:

import bliss.saga as saga

Next, we create a job service object that represents your local machine. The job service takes a single URL as parameter. The URL parameter is passed to Bliss' plug-in mechanism and based on the URL scheme, a specific plug-in is selected to connect to the specified location. The URL is a way to tell Bliss what type of queueing system or middleware you want to use and where it is. For example, to use the local machine:

js = saga.job.Service("fork://localhost")

Once the job.Service object has been created, it can be used to create and start new jobs. To define a new job, a job.Description object needs to be created that contains information about the executable we want to run, the arguments that we need to passed to it, the environment that needs to be set and what requirements we have for our job. Here's an example:

jd = saga.job.Description()
    
# environment, executable & arguments
jd.environment = {'MYOUTPUT':'"Hello from Bliss"'}       
jd.executable  = '/bin/echo'
jd.arguments   = ['$MYOUTPUT']

# output options
jd.output = "my1stjob.stdout"
jd.error  = "my1stjob.stderr"

---

Proceed to SAGA Tutorial Part 3: Remote Job Submission