Skip to content

App creation FAQ

Jump to bottom Edit New page
Alice Minotto edited this page Mar 29, 2017 · 23 revisions

General questions

What do I need to create an Agave app?

You will need an app description file (JSON), a wrapper script, a test script and a template submission file for HTCondor (optional). See the renseq app repository for examples.

What should the app description file contain?

The app description file contains JSON describing the app. It is documented in detail here. This is an example app description file, containing the minimum fields that are needed:

Expand source 


{
"name"          : "example",
"label"         : "Example app",
"version"       : "0.1.1",
"executionType" : "CLI",


"executionSystem"  : "cyverseUK-Batch2",
"deploymentSystem" : "cyverseUK-Storage2",
"deploymentPath"  : "yourdirectory/app",
"templatePath"    : "wrapper.sh",
"testPath"        : "test.sh",
"parallelism"     : "SERIAL",
"shortDescription" : "A simple example app",
"longDescription"  : "A simple example app",
"helpURI" : "http://example.com",
"datePublished": "23rd Aug 2016",
"author" : "John Doe",
"tags": [ "testing", "example"],
"ontology": [ ],


"inputs" : [ {
"id": "input1",
"details" : {
"label": "An example input file",
"description": "As simple example input
},
"value": {
"required" : "true",
"default" : "agave://yourstoragesystem/yourdirectory/file",
"visible":true
},
"semantics": {
"minCardinality": 1,
"maxCardinality": -1
}
}
],
"parameters" : [
{
"id": "param1",
"details" : {
"label": "An example parameter" ,
"description": "This parameter is a number value"
},
"value": {
"required" : "true",
"type": "number",
"default": 7000000,
"visible":"true"
}
},
{
"id": "param2",
"details" : {
"label": "Another example parameter",
"description": "This parameter is a string"
},
"value": {
"required" : "false",
"type":"string",
"visible":"true",
"default":"somevalue"
}
}
]
}

What should the wrapper script contain?

The wrapper script is executed on your execution system by Agave after expansion of the variables in it. See here for details. This is an example wrapper script for the app described above:

Expand source 

#!/bin/bash

#create the submission file from the template
cp lib/templatesubmit.htc lib/condorsubmit.htc


#create the arguments entry using our parameters (the ids are the same as given in the app description JSON)
echo arguments = ${param1} ${param2} >> lib/condorsubmit.htc


#create the input_files entry using out inputs
echo transfer_input_files = ${input1} >> lib/condorsubmit.htc


#add the 'queue' line to submit 1 job
echo queue >> lib/condorsubmit.htc


#submit our job to HTCondor and store the job id
jobid=condor_submit lib/condorsubmit.htc
jobid=echo $jobid | sed -e 's/Sub.*uster //'
jobid=echo $jobid | sed -e 's/\.//'


#follow the progress of our job with condor_tail, which exits when the job is done
condor_tail -f $jobid


exit 0

What does the HTCondor submit template do?

The submit template contains the 'front matter' for a condor submission file, as documented here. After the arguments and inputs lines are added, it can be submitted to the condor cluster using the condor_submit command.There are by now many CyVerseUK submit template examples which you can get inspired from. However, if you are not sure about any syntax, this is a good website to go to.

Where do my app assets live (deployment system)?

The files related to your app (wrapper, submit template, any other files you need to execute) are stored on the deployment system. For CyVerse UK at EI, this is named 'cyverseUK-storage2'.

Where does my app execute (execution system)?

The execution of your app consists of the copying of all input files from wherever they may be to a working directory on the execution system. After that the wrapper script will be processed by Agave and executed. This is all handled by Agave.

Where should I put my app files?

Your app assets need to be stored on the deployment system. They will be copied to the execution system whenever a job is run, together with the user inputs.

Where is the app working directory?

The app working directory is created on the execution system. The absolute location may vary, so you should not use absolute paths anywhere.

How do I register Agave credentials?

You will need to register for a (free) CyVerse account first here. Then, setup Agave credentials by following the following steps. It might help to download and install the agave command line tool on your PATH to allow you to run agave commands easier. We will assume you have the tools installed in the following guide:

Expand source 

user@server:~$ tenants-init
Please select a tenant from the following list:
[0] agave.prod
[1] araport.org
[2] designsafe
[3] iplantc.org
[4] irec
[5] irmacs
[6] sgci
[7] tacc.prod
[8] vdjserver.org
Your choice [3]: 3
user@server:~$ clients-create -N cli_client -u username -S
API password: *enter your CyVerse password*
Successfully created client cli_client
key: *somekeytoken*
secret: *somesecrettoken*
user@server:~$ auth-tokens-create -S
API password:
Token for iplantc.org:username successfully refreshed and cached for 14400 seconds
*sometoken*
user@server:~$

How do I start an Agave job?

After you've created credentials and install the agave command line tools, run jobs-submit -F runexample.json to submit a new job. An example job submission file is shown here:

Expand source 

{
  "name" : "example",
  "appId": "example-0.1.1",
  "archive": "true",
  "batchQueue": "normal",
  "inputs": {
    "input1" : "http://github.com/youruser/yourrepo/file.txt",
  },
  "parameters": {
    "param1" : 2
    "param2" : "something interesting"
  }
}

I get an error saying "Invalid credentials".

After you've registered your credentials, your token will expire after 4 hours. You can run auth-tokens-refresh -S to refresh it.

I get an error saying "No permissions to publish app on system X" when trying to register an app.

You have to be an admin to publish applications: you can ask the CyVerse team to publish the app for you.

Is there a Cyverse-sdk command manual?

The closest to a Cyverse-sdk command line manual is this tutorial here: manual/tutorial

Running your app, HTCondor & Docker

How is my app submitted to the job system?

The app is submitted to the HTCondor scheduler through the wrapper script. This means your wrapper script must call condor_submit with a valid submission file (you can see the template repository for some examples of how to handle this).

How do I create a Docker image for running a job?

Current practice is to publish a Dockerfile (described here and in Docker) in a GitHub repository, preferably a specific repository for your single app. This will allow you to create an Automated Build on the Docker hub, which builds a Docker image from your Dockerfile. If you then specify the name of your docker image in your HTCondor submission template, HTCondor will pull the image automatically.

Remember to version your images! Pulling the :latest tag will not always get the most up to date image!

How do I connect my app on Github with Docker (automated builds from github)?

When you push changes of your App onto Github, Docker can automatically adapt those changes. This is called an automatic build. How to configure automated builds is described here.

How do I version my Docker file using git?

You can version your Docker-build using git. But you need to manually change the name of your Docker container in the specific HTCcondor submission file. In your local git repository do:

git tag -a v0.0.X -m "version X" 
git push origin v0.0.X

(You can push all the tags with --tags). Note that you need to specify origin, as otherwise the tagging does not work. Then, manually add the tag to the HTCondor submission file. Your version name will appear in the Tag column in the build details on the specific docker automated build table.

How do I publish my Docker with the CyVerseUK docker hub?

Please ask someone from the CyVerseUK team to invite you to the CyVerseUK Organisation and provide them with the email address you have used for your docker account. Please move your automated builds there when it's ready for documentation purposes.

Where do my input files go?

Input files are submitted through Agave. If they are stored in an Agave system, you can refer to them using an agave link, e.g. agave://some-storage-system/home/yourdir/file.txt. They can also be in a web accessible location, e.g. http://github.com/you/repo/file.txt. These files will then be copied by agave into the execution working directory.

Job Management

How are input files transferred?

Once the inputs are copied into place by Agave, they are distributed to the relevant compute node by HTCondor. When running a job using Docker, it will copy your files into a volume in the docker container and make them accessible. It is important to explicitly list all the input files you expect to be transferred in the transfer_inputs line in the submission file.

How are output files transferred?

HTCondor monitors your jobs working directory for changes, and will transfer any files that have been newly created since the job start. If you want to explicitly list the outputs that you want transferred back, use the transfer_output_files directive, documented in more detail here.

How do I use user parameters and files with my job?

User parameters and inputs are transferred into your jobs working directory. Their name, locations and values will be substituted in your wrapper.sh and are thus present in your execution script. They can be fed to the HTCondor submission file using the 'arguments' and transfer_input_files directives after which HTCondor will manage their transfer.

Add a custom footer
Clone this wiki locally