SAGA Tutorial Part 2: Local Job Submission

One of the most important feature of SAGA Python is the capability to submit jobs to local and remote queueing systems and resource managers. This first example explains how to define a SAGA job using the Job API and run it on your local machine.

The job submission and management capabilities of SAGA Python are packaged in the saga.job module (API Doc). Three classes are defined in this module:

• The job.Service class (API Doc) provides a handle to the resource manager, like for example a remote PBS cluster.
• The job.Description class (API Doc) is used to describe the executable, arguments, environment and requirements (e.g., number of cores, etc) of a new job.
• The job.Job class (API Doc) is a handle to a job associated with a job.Service. It is used to control (start, stop) the job and query its status (e.g., Running, Finished, etc).

Hands-On: Local Job Submission

Before we discuss the individual API call in more detail, let's get down and dirty and run our first example: creating and running a SAGA job on your local machine.

Create a new file saga_example_1.py and paste the following code:

import sys
import bliss.saga as saga

def main():
    try: 
        # create a job service for lonestar
        js = saga.job.Service("fork://localhost")

        # describe our job
        jd = saga.job.Description()

        jd.environment     = {'MYOUTPUT':'"Hello from Bliss"'}       
        jd.executable      = '/bin/echo'
        jd.arguments       = ['$MYOUTPUT']
        jd.output          = "my1stjob.stdout"
        jd.error           = "my1stjob.stderr"

        # create the job (state: New)
        myjob = js.create_job(jd)

        print "Job ID    : %s" % (myjob.jobid)
        print "Job State : %s" % (myjob.get_state())

        print "\n...starting job...\n"
        # run the job 
        myjob.run()

        print "Job ID    : %s" % (myjob.jobid)
        print "Job State : %s" % (myjob.get_state())

        print "\n...waiting for job...\n"
        # wait for the job to either finish or fail
        myjob.wait()

        print "Job State : %s" % (myjob.get_state())
        print "Exitcode  : %s" % (myjob.exitcode)

    except saga.Exception, ex:
        print "An error occured during job execution: %s" % (str(ex))
        sys.exit(-1)

if __name__ == "__main__":
    main()

Save the file and execute it via the python interpreter (make sure your virtualenv is activated):

    python saga_example_1.py

The output should look something like this:

Job ID    : [fork://localhost]-[None]
Job State : saga.job.Job.New

...starting job...

Job ID    : [fork://localhost]-[644240]
Job State : saga.job.Job.Pending

...waiting for job...

Job State : saga.job.Job.Done
Exitcode  : None

Once the job has completed, you can have a look at the output file my1stjob.stdout.

A Quick Note on Logging

Since working with distributed systems is inherently complex and much of the complexity is hidden within SAGA Python, it is necessary to do a lot of internal logging. By default, logging output is disabled, but if something goes wrong or if you're just curious, you can enable the logging output by setting the environment variable SAGA_VERBOSE to a value between 1 (print only critical messages) and 6 (print all messages). Give it a try with the above example:

    SAGA_VERBOSE=6 python saga_example_1.py

Details & Discussion

Now that we have successfully run our first job with SAGA Python, we will discuss some of the the building blocks and details of the code.

In order to use the Bliss Job API, we first need to import the SAGA Python (bliss) module:

import bliss.saga as saga

Next, we create a job service object that represents the compute resource you want to use (see figure above). The job service takes a single URL as parameter. The URL is a way to tell SAGA Python what type of resource or middleware you want to use and where it is. The URL parameter is passed to SAGA Python's plug-in selector and based on the URL scheme, a plug-in is selected. In this case the Local job plug-in is selected for "fork://". URL scheme - Plug-in mapping is described on the SAGA Plug-Ins page.

js = saga.job.Service("fork://localhost")

Once the job.Service object has been created, it can be used to create and start new jobs. To define a new job, a job.Description object needs to be created that contains information about the executable we want to run, the arguments that we need to passed to it, the environment that needs to be set and what requirements we have for our job. Here's an example:

jd = saga.job.Description()
    
# environment, executable & arguments
jd.environment = {'MYOUTPUT':'"Hello from Bliss"'}       
jd.executable  = '/bin/echo'
jd.arguments   = ['$MYOUTPUT']

# output options
jd.output = "my1stjob.stdout"
jd.error  = "my1stjob.stderr"

---

Back: [Tutorial Home](SAGA Tutorial)    Next: SAGA Tutorial Part 3: Remote Job Submission