A portable, scalable eukaryotic genome annotation pipeline implemented in Nextflow.
This software is a comprehensive computational pipeline for the annotation of eukaryotic genomes (like protozoan parasites). It performs the following tasks:
- Fast generation of pseudomolecules from scaffolds by ordering and orientating against a reference
- Accurate transfer of highly conserved gene models from the reference
- De novo gene finding as a complement to the gene transfer
- Non-coding RNA detection (tRNA, rRNA, sn(o)RNA, ...)
- Pseudogene detection
- Functional annotation (GO, products, ...)
- ...by transferring reference annotations to the target genome
- ...by inferring GO terms and products from Pfam pHMM matches
- Consistent gene ID assignment
- Preparation of validated GFF3, GAF and EMBL output files for jump-starting manual curation and quick turnaround time to submission
It supports parallelized execution on a single machine as well as on large cluster platforms (LSF, SGE, ...).
This should get you up & running on an Ubuntu system, but please read the full documentation before before doing any work "for real".
Execute these commands as root, e.g. using sudo
apt-get install default-jre
curl -fsSL get.nextflow.io | bash && \
mv nextflow /usr/local/bin
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
apt-get update && \
apt-cache policy docker-ce && \
apt-get install --yes docker-ce && \
systemctl enable docker
To enable you to use docker with your normal user account (i.e. without being root or needing to use sudo),
run the following command, with your username in place of <username>.
usermod -aG docker <username>
Log out and log back in again for this to take effect.
java -versionshould say you have Java 1.8 or greaternextflow infowill print system information if nextflow has been installed successfullysystemctl status dockerwill tell you if docker is active (running)docker infowill print information if docker has been installed successfully, and you have permission to use it
Execute these commands in the directory you want to keep your Companion work in. Do this as a normal user, i.e. not as root or using sudo.
Use a name that is meaningful to you in place of <my-companion-project>
curl -L -o companion-master.zip https://github.com/sanger-pathogens/companion/archive/master.zip && \
unzip companion-master.zip && \
mv companion <my-companion-project>
docker pull sangerpathogens/companion
Companion is distributed with configuration and data (including a few pregenerated reference annotations) for a small test run. Run
the following command (using the name you chose for your project directory in place of my-companion-project).
nextflow run my-companion-project -profile docker
This will create a directory my-companion-project/example-output with the results of the run.
The file params_default.config configures the pipeline, and will need to be edited for your annotation run. You will probably need to change at least the following parameters:
inseq Your input FASTA file (${baseDir}/example-data/L_donovani.1.fasta in the example parameter file included wirth the distribution)
ref_dir The directory containing your reference genomes (${baseDir}/example-data/references in the example file)
ref_species The "short name" for your reference species (LmjF.1 in the example file)
dist_dir The directory that will contain the newly created output files (${baseDir}/example-data-output in the example file)
GENOME_PREFIX Text pattern matching your genome prefix (LDON in the example file)
CHR_PATTERN Pattern matching your chromosome names (LDON_(%w+) in the example file, where %w+ matches one or more letters or numbers)
ABACAS_BIN_CHR Abacas bin chromosome (LDON_0 in the example file)
EMBL_AUTHORS etc.; please provide suitable EMBL metadata (dummy values in the example file)
TAXON_ID Please provide suitable value for the GAF output (4711 in the example file)
The reference annotations used in the pipeline need to be pre-processed before they can be used. To add a reference organism, you will need:
- a descriptive name of the organism
- a short abbreviation for the organism
- the genome sequence in a single FASTA file
- a structural gene annotation in GFF3 format (see below for details)
- functional GO annotation in GAF 1.0 format, on the gene level
- a pattern matching chromosome headers, describing how to extract chromosome numbers from them
- an AUGUSTUS model, trained on reference genes
Insert these file names, etc., where <placeholders> appear in the steps below:
- Create a new data directory (i.e. the equivalent of the
example-datadirectory included in the distribution) - Edit
nextflow.config(and any config files that are referenced) and change parameters such asinseqandref_dirto your new data directory. - Copy the new reference genome (FASTA) into
<new_data_dir>/genomes - Copy GFF3 and GAF files into
<new_data_dir>/genomes - Copy Augustus model files into
data/augustus/species/<species_name>/ - Create new directory
<new_data_dir>/references/<short_name>/ - Add new section to
<new_data_dir>/references/references-in.json, using the short name (same as the directory name in the previous step); in this section add the names/paths of the files copied (above), a descriptive name, and a pattern for matching chromosomes in the FASTA files (in this example, <short_name>_, where n in any integer).
"<short_name>" : { "gff" : "../genomes/<gff3_filename>.gff3",
"genome" : "../genomes/<ref_genome_name>.fasta",
"gaf" : "../genomes/<ref_annot_filename>.gaf",
"name" : "<Descriptive Name of Reference Genome>",
"augustus_model" : "../../data/augustus/species/<species_name>/",
"chromosome_pattern" : "<short_name>_(%d+)"
}
- Finally, change directory to
<new_data_dir>/references(you must execute the following command in this directory) and run../../bin/update_references.lua. This writes the file<new_data_dir>/references/references.json.
The following command (using the name you chose for your project directory in place of my-companion-project) will
start your annotation run:
nextflow run my-companion-project -profile docker
Companion has the following dependencies:
To check if you have Java installed, and the version, use the command java -version. Note that this will give you a version number
of 1.8 for Java 8, 1.9 for Java 9, etc.
To install Java 8 on an Ubuntu or Debian system, run:
apt-get install openjdk-8-jre
On Fedora, Centos or Red Hat (etc.) systems:
yum install java-1.8.0-openjdk
To install Nextflow, run:
curl -fsSL get.nextflow.io | bash
This will create an executable called 'nextflow', which should be moved to a suitable directory, for example:
mv nextflow /usr/local/bin/
Use the command which nextflow to check that it is found in your path.
Docker is required if you intended to use the Docker image, as recommended below, to satisfy the dependencies.
To install Docker, see the installation guide for Ubuntu, Centos, Debian or Fedora.
Users running Companion with Docker will need to be added to the docker group (unix users can belong to one or more groups, which determine
whether they can peform certain actions; adding a user to the docker group allows them to execute docker commands). To add the user with
username <username>, to the docker group, run:
usermod -aG docker <username>
Some Linux systems may not have
usermodinstalled, as there are different programs that can be used to change user settings; please consult your Linux distribution documentation if necessary.
There are a number of ways to install Companion; details for an installation using Docker are described below. If you encounter an issue when installing Companion please contact your local system administrator. If you encounter a bug please log it here or email us at [email protected].
The easiest way to use the pipeline is to use the prepared Docker image which contains all external dependencies.
docker pull sangerpathogens/companion
To create a local copy of companion, you can download this repo from github (if you are familiar with github, you may of course prefer to clone or fork it).
curl -L -o companion-master.zip https://github.com/sanger-pathogens/companion/archive/master.zip # or click the green button on the guthub web page
unzip companion-master.zip
mv companion-master my-companion-project # renaming it to something meaningful to you is a good idea
Now you can run Companion. There is an example dataset and parameterization included in the distribution, so to get started just run:
nextflow run my-companion-project -profile docker
The argument -profile docker instructs nextflow to run the sangerpathogens/companion docker image for the dependencies;
the nextflow.config file (and files referenced within it) define the docker profile and the docker image to be used.
If you run nextflow with the name of a github repository, it will pull (download) the contents of the repository and run with those. For example, the following command will do the same as the "local copy" example above:
nextflow run sanger-pathogens/companion -profile docker
It is best to use this with some caution. After the command above is
run, nextflow will have stored a local copy of the repository in .nextflow/assets/sanger-pathogens
(note that .nextflow is a hidden directory, and will not usually be visible; use the command ls -la .nextflow to see it).
If you run the same command again it will this time use the local copy instead of pulling a copy from the repository. You can edit the files in your local copy, and nextflow will work from your (now different) version of sanger-pathogens/companion.
If you are familiar with repositories, and the workflow appropriate to using them, this can be a very convenient way of working. You can create your own github repository to store and share your work, and track versions.
If you are not familiar with git repositories, it can become quite confusing, and you should probably work with a simple local copy.
Further documentation on preparing reference data can be found in the GitHub wiki.
Companion is free software, licensed under ISC.
Please report any issues to the issues page or email [email protected]
If you use this software please cite:
Companion: a web server for annotation and analysis of parasite genomes.
Steinbiss S, Silva-Franco F, Brunk B, Foth B, Hertz-Fowler C et al.
Nucleic Acids Research, 44:W29-W34, 2016.
DOI: 10.1093/nar/gkw292