Home
BigJob, a SAGA-based Pilot-Job, is a general purpose Pilot-Job framework. The [Simple API for Grid Applications (SAGA)] (http://saga-project.github.com/) is a high-level, easy-to-use API for accessing distributed resources. SAGA BigJob supports a wide range of application types, and is usable over a broad range of infrastructures, i.e., it is general-purpose, extensible and interoperable. It is 100% written in python.
Pilot-Jobs support the decoupling of workload submission from resource assignment. This results in a flexible execution model, which in turn enables the distributed scale-out of applications on multiple and possibly heterogeneous resources. Pilot-Jobs support the use of container jobs with sophisticated workflow management to coordinate the launch and interaction of actual computational tasks within the container. It allows the execution of jobs without the necessity to queue each individual job.
The pilot job provides a container for many sub-jobs, i.e applications submit these sub-jobs through the Pilot-Job and not the resource manager. A major advantage of this approach is that the waiting time at the local resource manager, which usually significantly contributes the overall time-to-completion is avoided.
Unlike other common Pilot-Job systems, SAGA BigJob: (i) natively supports MPI job and (ii) works on a variety of back-end systems, generally reflecting the advantage of using a SAGA-based approach.
An overview of the BigJob architecture can be found [here] (https://github.com/saga-project/BigJob/wiki/BigJob-Architecture).
- BigJob Tutorial - Beginner's Guide to BigJob (useful for developing workflows)
- Installation Guide - Explains how to install BigJob.
Two types of interfaces are provided by BigJob. Users can choose one of them based on their needs.
Pilot API provides a programmable interface to develop workflows which can manage compute and data concurrently on distributed cyberinfrastructures. The API documentation can be found here
The Pilot command line interface provides an interface for a simple workflow involving job submission on a single machine. The pilot-cli is shipped along with the BigJob installation. Instructions on how to use pilot-cli can be found below.
Check the [Frequently Asked Questions] (https://github.com/saga-project/BigJob/wiki/Frequently-Asked-Questions).
For questions and comments, please join the bigjob-users group:
|
| Subscribe to bigjob-users |
| Email: |
| Visit this group |