Working with Agave

The core of the CyVerse UK infrastructure revolves around the Agave API (http://agaveapi.co). There are two levels to using this API: As an end user submitting jobs and as a developer creating apps. Most of the initial setup e.g. creating tokens etc are the same for both of these use cases. In this document, I will focus on using Agave as a developer (e.g. for app creation).

App creation FAQ

• General questions
  • What do I need to create an Agave app?
  • What should the app description file contain?
  • What should the wrapper script contain?
  • What does the HTCondor submit template do?
  • Where do my app assets live (deployment system)?
  • Where does my app execute (execution system)?
  • Where should I put my app files?
  • Where is the app working directory?
  • How do I register Agave credentials?
  • How do I start an Agave job?
  • I get an error saying "Invalid credentials".
  • I get an error saying "No permissions to publish app on system X" when trying to register an app.
• Running your app, HTCondor & Docker
  • How is my app submitted to the job system?
  • How do I create a Docker image for running a job?
  • Where do my input files go?
• Job management
  • How are input files transfered?
  • How are output files transfered?
  • My job is stalled, what to do?
  • How can I track my jobs progress?
  • How do I use user parameters and files with my job?

General questions

What do I need to create an Agave app?

You will need an app description file (JSON), a wrapper script, a test script and a template submission file for HTCondor (optional). See the renseq app repository for examples.

What should the app description file contain?

The app description file contains JSON describing the app. It is documented in detail here. This is an example app description file, containing the minimum fields that are needed:

{
  "name"          : "example",
  "label"         : "Example app",
  "version"       : "0.1.1",
  "executionType" : "CLI",
 
  "executionSystem"  : "cyverseUK-Batch2",
  "deploymentSystem" : "cyverseUK-Storage2",
  "deploymentPath"  : "yourdirectory/app",
  "templatePath"    : "wrapper.sh",
  "testPath"        : "test.sh",
  "parallelism"     : "SERIAL",
  "shortDescription" : "A simple example app",
  "longDescription"  : "A simple example app",
  "helpURI" : "http://example.com",
  "datePublished": "23rd Aug 2016",
  "author" : "John Doe",
  "tags": [ "testing", "example"],
  "ontology": [ ],
 
  "inputs" : [ {
      "id": "input1",
      "details" : {
        "label": "An example input file",
        "description": "As simple example input
      },
      "value": {
        "required" : "true",
        "default" : "agave://yourstoragesystem/yourdirectory/file",
        "visible":true
      },
      "semantics": {
        "minCardinality": 1,
    "maxCardinality": -1
      }
    }
  ],
  "parameters" : [
    {
      "id": "param1",
      "details" : {
        "label": "An example parameter" ,
        "description": "This parameter is a number value"
      },
      "value": {
        "required" : "true",
        "type": "number",
        "default": 7000000,
        "visible":"true"
      }
    },
    {
      "id": "param2",
      "details" : {
        "label": "Another example parameter",
        "description": "This parameter is a string"
      },
      "value": {
        "required" : "false",
        "type":"string",
        "visible":"true",
        "default":"somevalue"
      }
    }
  ]
}