Error to finish testrun

[0819shumpeitakeuchi]

Hi, I'm trying to use binny and am facing a test run error. This is my first post to github and I know it would be poor, but I would appreciate your response.

I'm using conda and have completed the installation of binny as follows.

conda create -n binny_env -y #python==3.11.4
git clone https://github.com/a-h-b/binny.git
cd binny

./binny -i config/config.init.yaml

Passed installation check on: /data/stakeuchi/230406tega/230621metaspades/binny/database/hmms/checkm_tf/
Passed installation check on: /data/stakeuchi/230406tega/230621metaspades/binny/database/hmms/checkm_pf/

--------INSTALLATION CHECK PASSED!--------

Done.

Then, tried ./binny -l -n "TESTRUN" -r config/config.test.yaml, but it doesn't work.
The error message is as follows.

Error in rule binny:
jobid: 1
output: $PWD/binny/test_output/binny.done
log: $PWD/binny/test_output/logs/binning_binny.log (check log file(s) for error message)
conda-env: $PWD/binny/conda/2330ca7cc0c5a80dff3499f6c4521d92_

RuleException:
CalledProcessErrorin line 453 of $PWD/binny/Snakefile:
Command 'source $PWD/binny/conda/snakemake_env/bin/activate '$PWD/binny/conda/2330ca7cc0c5a80dff3499f6c4521d92_'; set -euo pipefail; python $PWD/binny/test_output/.snakemake/scripts/tmp_oxil_ue.binny_main.py' returned non-zero exit status 1.
File "$PWD/binny/Snakefile", line 453, in __rule_binny
File "$PWD/binny/conda/snakemake_env/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Exiting because a job execution failed. Look above for error message

I downgraded joblib from 1.3.0 to 1.1.0 based on an issue #38.
I also tried sed -i -e '202s/snakemake=6.9.1/snakemake=7.16.0/g' binny based on an issue #40, but both didn't work.
I would appreciate it if you could let me know.

Thanks.

[ohickl]

Hi,
thanks for your interest.
Can you also post $PWD/binny/test_output/logs/binning_binny.log?

I also tried sed -i -e '202s/snakemake=6.9.1/snakemake=7.16.0/g' binny based on an issue #40, but both didn't work.

Did the command fail or did it not change the snakemake version?

Best

Oskar

[0819shumpeitakeuchi]

Thank you for your reply!

• $PWD/binny/test_output/logs/binning_binny.log was empty.
• The command seems succeeded echo "Creating snakemake environment" ${env_manager} create --prefix ${CONDA_SOURCE}/snakemake_env snakemake=7.16.0 mamba>=0.22.1 unzip python=3.8 -c conda-forge -c bioconda --yes fi.

Best

Takeuchi

[ohickl]

Ok. If you try to activate the Snakemake env your/path/to/snakemake_env it works and also if you do e.g. conda list | grep -i snakemake it shows the correct version? Same with $PWD/binny/conda/2330ca7cc0c5a80dff3499f6c4521d92_

[0819shumpeitakeuchi]

Sorry for the late reply.

• I could activate snakemake_env (succeeded conda activate snakemake_env)
• When I did conda list | grep -i snakemake, it showed

packages in environment at /home/stakeuchi/anaconda3/envs/snakemake_env:
snakemake 7.16.0 hdfd78af_0 bioconda
snakemake-minimal 7.16.0 pyhdfd78af_0 bioconda

• But I couldn't find snakemake from $PWD/binny/conda/2330ca7cc0c5a80dff3499f6c4521d92_ (there wasn't snakemake package in the environment)
  conda activate $PWD/binny/conda/2330ca7cc0c5a80dff3499f6c4521d92_
conda list | grep -i snakemake
  (no results)

Is this the cause?

[ohickl]

No, thats ok. The binny environment doesnt have snakemake. If you were able to activate it, it should be fine.
Could you post the config file, so we can have a look? Otherwise it might be a resource problem. Where/How are you running it? HPC or local computer? You could try e.g. -t 8 to give it more cores.

[0819shumpeitakeuchi]

• The attached file is the config file I'm using for test run. Since github doesn't support yaml file type, I changed the extension to .txt.
• I think it wouldn't be a resource problem. I use remote server which has an enough resource.
• Maybe I should learn more about snakemake.

config.test_2307Shump.yaml.txt

[ohickl]

Looks fine. You are running it from the binny dir? Might be that absolute paths for the inputs and outputs help.
Could yo urun ./binny -l -d -n "TESTRUN" -r config/config.test.yaml so we can look at a dry-run?

[0819shumpeitakeuchi]

• Absolute path doesn't seem to solve the problem. I mean $PWD/binny -l -n "TESTRUN" -r $PWD/config/config.test.yaml showed the same result above.

• The output of ./binny -l -d -n "TESTRUN" -r config/config.test.yaml was as follows.

Will use conda source path: $PWD/conda
Dryrun.
Config file $PWD/config/config.default.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
ALL          1              1              1
binny        1              1              1
total        2              1              1


[Wed Jul  5 18:40:43 2023]
Job 1: binny: Running Python Binny.
Reason: Missing output files: $PWD/test_output/binny.done

[Wed Jul  5 18:40:43 2023]
localrule ALL:
    input: $PWD/test_output/binny.done
    jobid: 0
    reason: Input files updated by another job: $PWD/test_output/binny.done
    resources: tmpdir=/tmp

Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
ALL          1              1              1
binny        1              1              1
total        2              1              1

Reasons:
    (check individual jobs above for details)
    input files updated by another job:
        ALL
    missing output files:
        binny

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

[ohickl]

testrun looks normal. maybe you can try ./binny -l -t 8 -n "TESTRUN" -r config/config.test.yaml, since its trying with one core and your mem per corre might not be sufficient.

[0819shumpeitakeuchi]

Thanks.
I tried ./binny -l -t 8 -n "TESTRUN" -r config/config.test.yaml, but the same error appeared. It seems that the capacity of memory doesn't induce this.
Maybe I should learn more about this.

[ohickl]

Hey, sorry for the late answer. could you maybe try the solution from here #48 ? There were apparently some breaking changes in HDBSCAN recently that require a new version.

[0819shumpeitakeuchi]

It seems to be working. Thanks!