-
Notifications
You must be signed in to change notification settings - Fork 8
BigJob Tutorial Part 2: Installation
This page is part of the BigJob Tutorial.
This tutorial will explain how to set up your environment and install BigJob.
- Python 2.6 or higher. Python 2.7.x is recommended.
- Redis Server (see below: Install Redis Server)
- SAGA-Python Installation (automatically installed when installing BigJob following this tutorial)
For more information on SAGA-Python, please read the SAGA Tutorial.
Assuming you don't want to mess with your system Python installation, you need a place were you can install BigJob locally. A small tool called virtualenv allows you to create a local Python software repository that behaves exactly like the global Python repository, with the only difference that you have write access to it. This is referred to as a 'virtual environment.'
To create your local Python environment run the following command (you can install virtualenv on most systems via apt-get or yum, etc.):
virtualenv $HOME/.bigjob/python
If you don't have virtualenv installed and you don't have root access to your machine, you can use the following script instead:
curl --insecure -s https://raw.github.com/pypa/virtualenv/master/virtualenv.py | python - $HOME/.bigjob/python
You need to activate your Python environment in order to make it work. Run the command below. It will temporarily modify your PYTHONPATH so that it points to $HOME/.bigjob/python/lib/python2.7/site-packages/ instead of the the system site-package directory:
source $HOME/.bigjob/python/bin/activate
Activating the virtualenv is very important. If you don't activate your virtual Python environment, the rest of this tutorial will not work. You can usually tell that your environment is activated properly if your bash command-line prompt starts with (python).
BigJob uses a Redis server for coordination and task management. Redis is the most stable and fastest backend (requires Python >2.5) and the recommended way of using BigJob. Redis can easily be run in user space. It can be downloaded at: http://redis.io/download (just ~500 KB). Once you downloaded it, start a Redis server on the machine of your choice:
$ redis-server
[489] 13 Sep 10:11:28 # Warning: no config file specified, using the default config. In order to specify a config file use 'redis-server /path/to/redis.conf'
[489] 13 Sep 10:11:28 * Server started, Redis version 2.2.12
[489] 13 Sep 10:11:28 * The server is now ready to accept connections on port 6379
[489] 13 Sep 10:11:28 - 0 clients connected (0 slaves), 922160 bytes in use
Then set the COORDINATION_URL parameter in the example to the Redis endpoint of your Redis installation, e.g.
redis://localhost:6379
You can install redis on a persistent server and use this server as your coordination server.
After your virtual environment is active, you are ready to install BigJob. BigJob is available via PyPi and can be installed using pip by typing:
easy_install BigJob
You can change the installation directory by calling:
easy_install --prefix=<target-dir> bigjob
To update a bigjob package execute:
easy_install -U bigjob
To install github version of bigjob package execute:
pip install git+git://github.com/saga-project/BigJob.git
To make sure that your installation works, run the following command to check if the BigJob module can be imported by the interpreter:
python -c "import pilot; print pilot.version"
Prior to running these examples, you will need to create a directory called 'agent' in the same location that you are running your scripts from. BigJob uses this as its working directory. For this tutorial, we'll create the agent directory in the $HOME directory by typing:
mkdir $HOME/agent
If you are planning to submit from one resource to another, you must have SSH password-less login enabled to the submitting resource. This is achieved by placing your public key on one resource in the authorized_keys file on the target machine. Please see our guide to configuring SSH Password-Less Login.
Examples of when you would need password-less login: (1) You want to submit from your local machine to an XSEDE resource, (2) You want to submit from one XSEDE resource to another, (3) You want to submit from your local cluster to external clusters, etc. etc.
Back: [Tutorial Home](BigJob Tutorial) Next: BigJob Tutorial Part 3: Simple Ensemble Example