Skip to content

Install Your Own BigJob

Jump to bottom
Melissa edited this page Jan 22, 2014 · 25 revisions

THIS PAGE IS DEPRECATED. PLEASE SEE http://saga-project.github.io/BigJob/sphinxdoc/install/index.html

1. Overview and Concepts

BigJob is a pilot-job framework written in Python. It uses the Simple API for Grid Applications (SAGA) with its different middleware plug-ins (adaptors) to access distributed systems, like e.g. via Globus, PBS or Torque.

Furthermore, BigJob relies on Redis as the underlying communication infrastructure. Consequently, before you can install BigJob itself, you will have to install both, SAGA and Redis. This article covers the basic installation procedures - for more detailed informations, please refer to the Redis and SAGA websites.

The pre-requisites can be summarized as follows:

  1. Python 2.6 or higher. Python 2.7.x is recommended
  2. A running Redis server
  3. SAGA-Python

2. Install SAGA-Python

SAGA-Python will be automatically installed if you install the BigJob package from PyPi.

3. Install Redis

Download and install Redis (>2.2) as described here

4. Install BigJob

There are 4 ways to install BigJob.

a. Python Packaging:

Use this method if you want to install BigJob in system space. Else use 'Virtualenv' or the 'Manual Bootstrapping' process which is documented below.

BigJob can now be installed using the standard Python packaging tools:

easy_install bigjob

You can change the installation directory by calling:

easy_install --prefix=<target-dir> bigjob

To update a bigjob package execute:

easy_install -U bigjob

b. Python Packaging & Virtualenv:

If you would like to install BigJob in your home directory the utilization of the virtualenv environment is recommended.

$ virtualenv ~/.bigjob/python/
New python executable in /Users/luckow/.bigjob/python/bin/python
Installing setuptools............done.
Installing pip...............done.

$ . ~/.bigjob/python/bin/activate

$ easy_install bigjob

c. Manual installation:

BigJob contains a bootstrap script, which can be used to setup a virtualenv environment including BigJob. The code below installs BigJob in $HOME directory.

curl -k https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py -o bigjob-bootstrap.py 
python bigjob-bootstrap.py $HOME/.bigjob/python/
. $HOME/.bigjob/python/bin/activate

The agent utilizes this script for automatic bootstrapping on machines that don't have BJ installed. In order to load your BigJob environment at login add the following line to your .bashrc (.profile, etc.):

. $HOME/.bigjob/python/bin/activate

d. Install development version via easy_install and Github:

easy_install https://github.com/saga-project/BigJob/tarball/master

e. BigJob source code:

To check out BigJob source code from Github use the following commands.

git clone http://github.com/saga-project/BigJob.git

or

svn co https://github.com/saga-project/BigJob.git

Then install the package to a location of your choice:

python setup.py install  --prefix=<target-dir>

To use BigJob export PYTHONPATH environment variable to the BigJob installation location.

5. Test the Installation

Login to your python shell and execute the following commands:

$ python
>> import saga # if no error then it means saga python bindings installed successfully. 
>> import bigjob #if no error then it means bigjob installed successfully. logout from python shell

Run the test-bigjob (script is installed with bigjob Python package). The test-bigjob command assumes that the default job adaptor and advert adaptor (w/ SQLITE3) is correctly installed on your machine. if SQLITE3 is not present, then an advert error is thrown. Please ignore this try the examples under tutorial section.

Clone this wiki locally