Test run completes, throws warnings and produces no bins

[btemperton]

Installed binny into a fresh conda environment on a ubuntu 19 box

conda create -n binny -c conda-forge -c bioconda python=3.9
conda activate binny
git clone https://github.com/a-h-b/binny.git
cd binny
mkdir database
cd database
wget https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz
cd ..
./binny -i config/config.init.yaml

Which gave a message that the installation checks had passed:

Cython correctly compiled!
Checking extra files
Passed installation check on: /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/8fcdf17dd1864a5bcea3966e66a16a88/lib/python3.9/site-packages/Resources/essential_genes
Passed installation check on: /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/8fcdf17dd1864a5bcea3966e66a16a88/lib/python3.9/site-packages/Resources/NCBI/gc.prt.dmp
Checking references installation
Passed installation check on: /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/database/hmms/checkm_tf/
Passed installation check on: /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/database/hmms/checkm_pf/
------------------------------------------
--------INSTALLATION CHECK PASSED!--------
------------------------------------------
Done.

The conda environment for binny looks like this following install:

name: binny
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2022.9.24=ha878542_0
  - ld_impl_linux-64=2.39=hc81fddc_0
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=12.2.0=h65d4601_18
  - libgomp=12.2.0=h65d4601_18
  - libnsl=2.0.0=h7f98852_0
  - libsqlite=3.39.4=h753d276_0
  - libuuid=2.32.1=h7f98852_1000
  - libzlib=1.2.13=h166bdaf_4
  - ncurses=6.3=h27087fc_1
  - openssl=3.0.5=h166bdaf_2
  - pip=22.3=pyhd8ed1ab_0
  - python=3.9.13=h2660328_0_cpython
  - readline=8.1.2=h0f457ee_0
  - setuptools=65.5.0=pyhd8ed1ab_0
  - sqlite=3.39.4=h4ff8645_0
  - tk=8.6.12=h27826a3_0
  - tzdata=2022e=h191b570_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0

Then ran the test command:

./binny -l -n "TESTRUN" -r config/config.test.yaml

and got the following output:

./binny -l -n "TESTRUN" -r config/config.test.yaml
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Will use conda source path: /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda
Running workflow in current session - don't use this setting except with small datasets and databases.
/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Traceback (most recent call last):
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/snakemake/__init__.py", line 699, in snakemake
    success = workflow.execute(
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/snakemake/workflow.py", line 1052, in execute
    logger.run_info("\n".join(dag.stats()))
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/snakemake/dag.py", line 2187, in stats
    yield tabulate(rows, headers="keys")
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/tabulate/__init__.py", line 2048, in tabulate
    list_of_lists, headers = _normalize_tabular_data(
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/tabulate/__init__.py", line 1471, in _normalize_tabular_data
    rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/tabulate/__init__.py", line 1471, in <lambda>
    rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/tabulate/__init__.py", line 107, in _is_separating_line
    (len(row) >= 1 and row[0] == SEPARATING_LINE)
  File "/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda/snakemake_env/lib/python3.8/site-packages/snakemake/rules.py", line 1127, in __eq__
    return self.name == other.name and self.output == other.output
AttributeError: 'str' object has no attribute 'name'
Building DAG of jobs...
Creating report...
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/binny.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/assembly.contig_depth.txt. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/assembly_contig_depth_reads_4bins.txt. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/assembly.formatted.fa. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/assembly.fa. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/reads_sample_000001_sorted.bam. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/reads_sample_000002_sorted.bam. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/assembly_contig_depth_reads_4bins2.txt. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/prokka.gff. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/annotation.filt.gff. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/prokka.faa. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/mantis_out/output_annotation.tsv. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/mantis_out/integrated_annotation.tsv. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/mantis_out/consensus_annotation.tsv. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Missing metadata for file /mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/test_output/intermediary/mantis_out. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning.
Downloading resources and rendering HTML.
Report created: report.html.

The directory test_output has binny.config.yaml and report.html and an empty tmp folder.

report.html.gz

[ohickl]

Hi Ben,

I could reproduce you error, it is because of this.
A quick fix, while waiting for a new release, would be removing binny and then reinstalling with an updated Snakemake version:

git clone https://github.com/a-h-b/binny.git
cd binny
sed -i -e '202s/snakemake=6.9.1/snakemake=7.16.0/g' binny
# Optional stuff if needed
./binny -i config/config.init.yaml
./binny -l -n "TESTRUN" -r config/config.test.yaml

If you're getting an error related to HDBSCAN you can try this fix.

Let me know if this works for you.
Thanks for your interest!

Best

Oskar

[btemperton]

Thanks Oskar,
That fix worked. I also hit the HDBSCAN issue and the suggested fix had issues resolving the conda environment due to a conflict when trying to run conda install joblib==1.1.0 --yes, but I was able to get round this by using pip to install the required library, so the final fix looks like:

conda create -n binny -y -c conda-forge -c bioconda python=3.9
conda activate binny
git clone https://github.com/a-h-b/binny.git
cd binny
sed -i -e '202s/snakemake=6.9.1/snakemake=7.16.0/g' binny
mkdir database
cd database
wget https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz
cd ..

./binny -i config/config.init.yaml

#This bit is needed to fix an issue with hdbscan
conda_envs_path=/mnt/b6e3667a-f310-4295-a05c-690f8b516eb1/rparsons/binny/conda
env_manager="conda"
for i in ${conda_envs_path}/*.yaml; do
  env_name=$(head -n 1 ${i} | cut -d' ' -f2)
  if [[ ${env_name} == 'binny_linux' ]]; then
    binny_env=${i%.yaml}
    echo "Loading binny env from: ${mantis_env}"
    ${env_manager} activate ${binny_env}
  fi
done
pip install joblib==1.1.0
${env_manager} deactivate

Test run complete successfully and yielded four bins.