Merge branch 'release/v2.1.1'

.travis.yml

@@ -14,9 +14,16 @@ before_install:
   - mvn clean install -DskipTests -B && cd ..
   - mv ./repseqio/.cache .cache
 
+script:
+  - mvn test -B && ./itests.sh test
+
 jdk:
   - openjdk7
 
+before_cache:
+  - rm -f $HOME/.m2/repository/io/repseqio/
+  - rm -f $HOME/.m2/repository/com/milaboratory/
+
 cache:
   directories:
   - $HOME/.m2

CHANGELOG

@@ -1,4 +1,33 @@
 
+MiXCR 2.1.1 ( 3 Mar 2017)
+========================
+
+-- Additional check for the same V hits in merging sequences in `assemblePartial`
+-- Fix: list of C hits limited by `relativeMinScore` in `assemblePartial`
+-- Not aligned mates from fully aligned opposite PE reads are now taken into `assemblePartial` 
+   procedure (case when V and J parts of the target read are shorter than 12 nt)
+-- Additional consistency cehcks in `assemblePartial`, overlaps of non-CDR3-covering mates of PE 
+   reads are also taken into account
+-- More sophisticated algorithm for alignment-guided merging of paired-end reads in `align`
+-- Several anchor point positions in export points changed their meaning.
+-- Fixed absent space in `AA. Seq. ...` column name and excessive space in the end of several 
+   column names (affects `exportClones` and `exportAlignments` with `-v` option; fixes compatibility 
+   with VDJTools v1.1.3 and below).
+-- Fixes NPE in `exportClones` in some cases with mmu reference
+-- `-p` option in `assemblePartial` enabled by default; deprecated; will be removed in 2.2
+-- Added `-d` option in `assemblePartial` to drop non-assembled partial reads to reduce output file 
+   size
+-- Added `sortAlignments` action. Sort `*.vdjca` files by read id, allows diffing alignments after 
+   `assembleParial` and other actions that shuffle records inside file.
+-- Fixed incorrect behaviour of `exportClones` and `exportAlignments` in cases like `mixcr 
+   exportAlignments -p full -descrR1 ...`
+-- minor: Multiple reads in `exportAlignmentsPretty` and `filterAlignments` should be now specified 
+   as `-i 123 -i 456` or `-i 123,456` etc.
+-- minor: `--reads-ids` in `exportAlignmentsPretty` and `--readsIds` in `filterAlignments` both 
+   renamed to `--read-ids`
+-- minor: infrastructure for post-build tests in Travis CI
+
+
 MiXCR 2.1 ( 6 Feb 2017)
 ========================
 

README.md

@@ -101,6 +101,12 @@ Dependancy:
 
 To build MiXCR from source:
 
+- Clone repository
+
+  ```
+  git clone https://github.com/milaboratory/mixcr.git
+  ```
+
 - Refresh git submodules
 
   ```

doc/assemble.rst

@@ -271,72 +271,41 @@ In order to turn off clustering one should use the following parameters:
 
     mixcr assemble -OcloneClusteringParameters=null alignments.vdjca output.clns
 
-Clone factory parameters
-------------------------
+.. Clone factory parameters
+.. ------------------------
 
-Parameters which control final alignment of clonal sequences are placed
-in ``cloneFactoryParameters`` group. These parameters includes separate
-groups for V, D, J and C aligners: ``vParameters``, ``dParameters``,
-``jParameters`` and ``cParameters``. The D aligner is the same as used
-in ``align`` and thus all its parameters and their default values are
-the same as described for :ref:`D aligner in align <ref-dAlignerParameters>`. One
-can override these parameters in the following way:
+.. Parameters which control final alignment of clonal sequences are placed in ``cloneFactoryParameters`` group. These parameters includes separate groups for V, D, J and C aligners: ``vParameters``, ``dParameters``, ``jParameters`` and ``cParameters``. The D aligner is the same as used in ``align`` and thus all its parameters and their default values are the same as described for :ref:`D aligner in align <ref-dAlignerParameters>`. One can override these parameters in the following way:
 
-::
-
-    mixcr assemble -OcloneFactoryParameters.dParameters.absoluteMinScore=10 alignments.vdjca output.clns
-
-::
-
-    mixcr assemble -OcloneFactoryParameters.dParameters.scoring.gapOpenPenalty=-10 alignments.vdjca output.clns
-
-The aligners used to build alignments with V, J and C genes are
-different from those used by ``align``.
+.. 
 
-V, J and C aligner parameters
-'''''''''''''''''''''''''''''
+..     mixcr assemble -OcloneFactoryParameters.dParameters.absoluteMinScore=10 alignments.vdjca output.clns
 
-The following table lists parameters of V, J and C aligners:
+.. 
 
+..     mixcr assemble -OcloneFactoryParameters.dParameters.scoring.gapOpenPenalty=-10 alignments.vdjca output.clns
 
+.. The aligners used to build alignments with V, J and C genes are different from those used by ``align``.
 
-+----------------------+-----------------+-----------------+------------------+-----------------------------------------------------------+
-| Parameter            | Default V value | Default J value | Default C value  |Description                                                |
-+======================+=================+=================+==================+===========================================================+
-| ``featureToAlign``   | ``VTranscript`` | ``JRegion``     | ``CExon1``       | Gene region used to build alignments.                     |
-+----------------------+-----------------+-----------------+------------------+-----------------------------------------------------------+
-| ``relativeMinScore`` | ``0.8``         | ``0.8``         | ``0.8``          | Relative minimal score of hit: hits with score less than  |
-|                      |                 |                 |                  | ``relativeMinScore * maxScore`` (``maxScore`` is score of |
-|                      |                 |                 |                  | best hit) will be dropped.                                |
-+----------------------+-----------------+-----------------+------------------+-----------------------------------------------------------+
+.. The scoring parameters are placed in group ``alignmentParameters.scoring``:
 
-One can override these parameters in the following way
+.. +-------------------------+----------------------------------------+--------------------------------------------------------------------+
+.. | Parameter               | Default value (same for V, J, C)       | Description                                                        |
+.. +=========================+========================================+====================================================================+
+.. | ``subsMatrix``          | ``simple(match = 5,``                  | Substitution matrix. Available types:                              |
+.. |                         |  ``mismatch = -9)``                    |                                                                    |
+.. |                         |                                        |  - ``simple`` --- a matrix with diagonal elements equal to         |
+.. |                         |                                        |    ``match`` and other elements equal to ``mismatch``              |
+.. |                         |                                        |  - ``raw`` --- a complete set of 16 matrix elements should be      |
+.. |                         |                                        |    specified; for  example:                                        |
+.. |                         |                                        |    ``raw(5,-9,-9,-9,-9,5,-9,-9,-9,-9,5,-9,-9,-9,-9,5)``            |
+.. |                         |                                        |     (*equivalent to the default value*)                            |
+.. +-------------------------+----------------------------------------+--------------------------------------------------------------------+
+.. | ``gapPenalty``          | ``-12``                                | Penalty for gap.                                                   |
+.. +-------------------------+----------------------------------------+--------------------------------------------------------------------+
 
-::
-
-    mixcr assemble -OcloneFactoryParameters.jParameters.featureToAlign=JRegion(-6,0) alignments.vdjca output.clns
-
-The scoring parameters are placed in group
-``alignmentParameters.scoring``:
+.. One can override these parameters in the following way
 
-+-------------------------+----------------------------------------+--------------------------------------------------------------------+
-| Parameter               | Default value (same for V, J, C)       | Description                                                        |
-+=========================+========================================+====================================================================+
-| ``subsMatrix``          | ``simple(match = 5,``                  | Substitution matrix. Available types:                              |
-|                         |  ``mismatch = -9)``                    |                                                                    |
-|                         |                                        |  - ``simple`` --- a matrix with diagonal elements equal to         |
-|                         |                                        |    ``match`` and other elements equal to ``mismatch``              |
-|                         |                                        |  - ``raw`` --- a complete set of 16 matrix elements should be      |
-|                         |                                        |    specified; for  example:                                        |
-|                         |                                        |    ``raw(5,-9,-9,-9,-9,5,-9,-9,-9,-9,5,-9,-9,-9,-9,5)``            |
-|                         |                                        |     (*equivalent to the default value*)                            |
-+-------------------------+----------------------------------------+--------------------------------------------------------------------+
-| ``gapPenalty``          | ``-12``                                | Penalty for gap.                                                   |
-+-------------------------+----------------------------------------+--------------------------------------------------------------------+
-
-One can override these parameters in the following way
-
-::
+.. 
 
-    mixcr assemble -OcloneFactoryParameters.vParameters.alignmentParameters.scoring.gapPenalty=-5 \
-                   alignments.vdjca output.clns
+..     mixcr assemble -OcloneFactoryParameters.vParameters.alignmentParameters.scoring.gapPenalty=-5 \
+..                    alignments.vdjca output.clns

doc/export.rst

@@ -175,34 +175,44 @@ The following table shows the correspondance between anchor point and positions
 +--------------------------+---------------------+--------------------+
 | FR3End / CDR3Begin       | 9                   | 10                 |
 +--------------------------+---------------------+--------------------+
-| VEnd / *PSegmentBegin*   | 10                  | 11                 |
-+--------------------------+---------------------+--------------------+
-| Number of 3' V deletions | 11                  | 12                 |
+| Number of 3' V deletions | 10                  | 11                 |
 | (negative value), or     |                     |                    |
 | length of 3' V P-segment |                     |                    |
 | (positive value)         |                     |                    |
 +--------------------------+---------------------+--------------------+
-| Number of 5' D deletions | 12                  | 13                 |
+| VEndTrimmed, next        | 11                  | 12                 |
+| position after last      |                     |                    |
+| aligned nucleotide of V  |                     |                    |
+| gene                     |                     |                    |
++--------------------------+---------------------+--------------------+
+| DBeginTrimmed, position  | 12                  | 13                 |
+| of first aligned         |                     |                    |
+| nucleotide of D gene     |                     |                    |
++--------------------------+---------------------+--------------------+
+| Number of 5' D deletions | 13                  | 14                 |
 | (negative value), or     |                     |                    |
 | length of 5' D P-segment |                     |                    |
 | (positive value)         |                     |                    |
 +--------------------------+---------------------+--------------------+
-| DBegin / *PSegmentEnd*   | 13                  | 14                 |
-+--------------------------+---------------------+--------------------+
-| DEnd / *PSegmentBegin*   | 14                  | 15                 |
-+--------------------------+---------------------+--------------------+
-| Number of 3' D deletions | 15                  | 16                 |
+| Number of 3' D deletions | 14                  | 15                 |
 | (negative value), or     |                     |                    |
 | length of 3' D P-segment |                     |                    |
 | (positive value)         |                     |                    |
 +--------------------------+---------------------+--------------------+
-| Number of 3' J deletions | 16                  | 17                 |
+| DEndTrimmed, next        | 15                  | 16                 |
+| position after last      |                     |                    |
+| aligned nucleotide of D  |                     |                    |
+| gene                     |                     |                    |
++--------------------------+---------------------+--------------------+
+| JBeginTrimmed, position  | 16                  | 17                 |
+| of first aligned         |                     |                    |
+| nucleotide of J gene     |                     |                    |
++--------------------------+---------------------+--------------------+
+| Number of 3' J deletions | 17                  | 18                 |
 | (negative value), or     |                     |                    |
 | length of 3' J P-segment |                     |                    |
 | (positive value)         |                     |                    |
 +--------------------------+---------------------+--------------------+
-| JBegin / *PSegmentEnd*   | 17                  | 18                 |
-+--------------------------+---------------------+--------------------+
 | CDR3End / FR4Begin       | 18                  | 19                 |
 +--------------------------+---------------------+--------------------+
 | FR4End                   | 19                  | 20                 |
@@ -212,8 +222,6 @@ The following table shows the correspondance between anchor point and positions
 | CExon1End                | 21                  | 22                 |
 +--------------------------+---------------------+--------------------+
 
-Positions of anchor points like ``VEnd`` are printed only if corresponding P-segment was detected in the sequence, in this case e.g. P-segment of V gene can be found between positions of ``VEnd`` and ``VEndTrimmed``.
-
 Examples
 --------
 

doc/quickstart.rst

@@ -239,9 +239,9 @@ MiXCR allows to extract TCR and BCR CDR3 repertoires from RNA-Seq data. Extracti
 
   .. code-block:: console
 
-    > mixcr assemblePartial -p alignments.vdjca alignmentsRescued_1.vdjca
+    > mixcr assemblePartial alignments.vdjca alignmentsRescued_1.vdjca
 
-    > mixcr assemblePartial -p alignmentsRescued_1.vdjca alignmentsRescued_2.vdjca
+    > mixcr assemblePartial alignmentsRescued_1.vdjca alignmentsRescued_2.vdjca
 
 3. Extend TCR alignments with uniquely determined V and J genes and having incomplete coverage of CDR3s using germline sequences:
 

doc/rnaseq.rst

@@ -75,10 +75,8 @@ Typical analysis workflow
 
   ::
 
-    mixcr assemblePartial -p alignments.vdjca alignments_rescued_1.vdjca
-    mixcr assemblePartial -p alignments_rescued_1.vdjca alignments_rescued_2.vdjca
-
-  ``-p`` option tells MiXCRsquences to pass unassembled alignments to the output file.
+    mixcr assemblePartial alignments.vdjca alignments_rescued_1.vdjca
+    mixcr assemblePartial alignments_rescued_1.vdjca alignments_rescued_2.vdjca
 
 3. (optional) Perform extension of incomplete TCR CDR3s with uniquely determined V and J genes using germline sequences. As described in the :ref:`last paragraph of introduction <ref-rna-seq-extend-description>`
 

itests.sh

@@ -1,5 +1,8 @@
 #!/bin/bash
 
+set -e
+set -o pipefail
+
 # "Integration" tests for MiXCR
 # Test standard analysis pipeline results
 
@@ -46,6 +49,26 @@ case $os in
     ;;
 esac
 
+create_standard_results=false
+run_tests=false
+while [[ $# > 0 ]]
+do
+    key="$1"
+    shift
+    case $key in
+        std)
+            create_standard_results=true
+        ;;
+        test)
+            run_tests=true
+        ;;
+        *)
+            echo "Unknown option $key";
+            exit 1
+        ;;
+esac
+done
+
 rm -rf ${dir}/test_target
 mkdir ${dir}/test_target
 
@@ -60,17 +83,27 @@ which mixcr
 mixcr -v
 
 function go_assemble {
-  mixcr assemble -r $1.clns.report $1.vdjca $1.clns || exit 1
+  mixcr assemble -r $1.clns.report $1.vdjca $1.clns
   for c in TCR IG TRB TRA TRG TRD IGH IGL IGK ALL
   do
-    mixcr exportClones -c ${c} -s $1.clns $1.clns.${c}.txt || exit 1
+    mixcr exportClones -c ${c} -s $1.clns $1.clns.${c}.txt
   done
 }
 
-for s in sample_IGH test;
-do
-  mixcr align -r ${s}_paired.vdjca.report ${s}_R1.fastq ${s}_R2.fastq ${s}_paired.vdjca || exit 1
-  go_assemble ${s}_paired
-  mixcr align -r ${s}_single.vdjca.report ${s}_R1.fastq ${s}_single.vdjca || exit 1
-  go_assemble ${s}_single
-done
+if [[ $create_standard_results == true ]]; then
+  for s in sample_IGH test;
+  do
+    mixcr align -r ${s}_paired.vdjca.report ${s}_R1.fastq ${s}_R2.fastq ${s}_paired.vdjca
+    go_assemble ${s}_paired
+    mixcr align -r ${s}_single.vdjca.report ${s}_R1.fastq ${s}_single.vdjca
+    go_assemble ${s}_single
+  done
+fi
+
+# UseCase 1
+
+if [[ $run_tests == true ]]; then
+  echo "Running test case 1"
+  mixcr align -OvParameters.geneFeatureToAlign=VGeneWithP test_R1.fastq test_R2.fastq case1.vdjca
+  mixcr assemble case1.vdjca case1.clns
+fi

pom.xml

@@ -32,7 +32,7 @@
 
     <groupId>com.milaboratory</groupId>
     <artifactId>mixcr</artifactId>
-    <version>2.1</version>
+    <version>2.1.1</version>
     <packaging>jar</packaging>
     <name>MiXCR</name>
 
@@ -44,14 +44,14 @@
 
     <properties>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <milib.version>1.7.1</milib.version>
+        <milib.version>1.7.2</milib.version>
     </properties>
 
     <dependencies>
         <dependency>
             <groupId>io.repseq</groupId>
             <artifactId>repseqio</artifactId>
-            <version>1.2.6</version>
+            <version>1.2.7</version>
         </dependency>
 
         <dependency>