Performance of saga python

Job Execution via shell adaptor:

Compare shell adaptor to raw local shell performance:

#!/bin/bash

N=1000
TIMEFORMAT="%3Rs   %3Us   %3Ss   %P%%"

echo "====================================================================="
echo "run $N jobs"
echo "====================================================================="

echo
echo "====================================================================="
echo "/bin/sleep 1 &"
echo "---------------------------------------------------------------------"
time (for i in $(seq 1 $N); do
  /bin/sleep 1 &
done) > /dev/null
echo "====================================================================="

echo
echo "====================================================================="
echo "RUN sleep 1"
echo "---------------------------------------------------------------------"
time (for i in $(seq 1 $N); do
  echo "RUN sleep 1"
done | sh ~/.saga/adaptors/ssh_job/wrapper.sh) > /dev/null
echo "====================================================================="

echo
echo "====================================================================="
echo "js = saga.job.Service ('fork://localhost/')"
echo "j  = js.create_job ({executable:'/bin/data', arguments:['1']})"
echo "---------------------------------------------------------------------"
time python -c "
import saga
jd = saga.job.Description ()
jd.executable = '/bin/sleep'
jd.arguments  = ['1']
js = saga.job.Service ('fork://localhost/')
for i in range (0, $N) :
  j=js.create_job (jd)
  j.run ()
"
echo "====================================================================="

echo
echo "====================================================================="
echo "js = saga.job.Service ('fork://localhost/')"
echo "j  = js.run_job (\"/bin/sleep 1\")"
echo "---------------------------------------------------------------------"
time python -c "
import saga
js = saga.job.Service ('fork://localhost/')
for i in range (0, $N) :
  j=js.run_job ('/bin/sleep 1')
"
echo "====================================================================="

echo
echo "====================================================================="
echo "js = saga.job.Service ('ssh://localhost/')"
echo "j  = js.create_job ({executable:'/bin/data', arguments:['1']})"
echo "---------------------------------------------------------------------"
time python -c "
import saga
jd = saga.job.Description ()
jd.executable = '/bin/sleep'
jd.arguments  = ['1']
js = saga.job.Service ('ssh://localhost/')
for i in range (0, $N) :
  j=js.create_job (jd)
  j.run ()
"
echo "====================================================================="

This results in the following numbers:

id	type	Real Time (s)	System Time (s)	User Time (s)	Utilization (%)
1	''/bin/sleep 1 & ``	0.211	0.000	0.124	58.73
2	''RUN sleep 1 ``	11.368	0.196	0.848	9.18
3	''js = saga.job.Service ('fork://localhost/') `` ''j = js.create_job ({executable:'/bin/data',`` '' arguments :['1']}) ``	17.303	8.105	0.764	51.25
4	''js = saga.job.Service ('fork://localhost/') `` ''j = js.run_job ("/bin/sleep 1") ``	17.643	8.241	0.816	51.33
5	''js = saga.job.Service ('ssh://localhost/') `` ''j = js.create_job ({executable:'/bin/data',`` '' arguments:['1']}) ``	17.626	8.169	0.768	50.70

1. : Plain shell is (as expected) very quick, and needs virtually no system resources. Time is spent mostly in the shell internals.

(2) : The shell wrapper we use adds an order of magnitude, mostly on user time. That wrapper makes sure that job information (state, pid) are written to disk, spawns a monitoring daemon, and reports job ID etc. But most of the time is actually to wait for those state information to be consistent and to be writtent (waitpid, sync) -- thus the low system utilization.

3. : The python layer on top of the wrapper adds a factor of ~1.5, which is quite acceptable, as that includes the complete SAGA

   and Python stack, and I/O capturing/parsing etc.

4. : The shortcut methods run_job() does not add significantly, despite the fact that it creates a new job description per job.

5. : Exchanging fork://localhost/ with ssh://localhost/ again adds almost nothing -- the overhead is solely due to the

   increased startup time.