playback - FAST5 file cannot be open

[diego-ds-souza]

Hello everyone,

I am trying to run a playback analysis on Ubuntu 22.04, but my run constantly fails after ~five minutes with the following error message:

MinION Mk1B · MN23485
Stopping protocol due to internal error. All data will be saved.

Checking log files, I found two messages that are potentially associated with this error:

In the bream-*.txt log file, the error message says:

INFO - Failed to retrieve calibration data from /home/nanoporeseq/Desktop/readfish/fast5_readfish/fast5_readfish/FAL43730_b1347b8a1cd19a950e6bf1c7d70fb7a32d50ecc7_10.fast5

And in the control_server_log-*.txt log file, it says:

WARNING: hdf5_error (fast5)
    cause: unable to open file: name = '/home/nanoporeseq/Desktop/readfish/readfish.fast5', errno = 13, error >
    function: H5Fopen
    message: unable to open file
    stack_trace:
                   #000: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: File accessibility
                     minor: Unable to open file
                   #001: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: Virtual Object Layer
                     minor: Can't open object
                   #002: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: Virtual Object Layer
                     minor: Can't open object
                   #003: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: File accessibility
                     minor: Unable to open file
                   #004: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: File accessibility
                     minor: Unable to open file
                   #005: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: Virtual File Layer
                     minor: Unable to initialize object
                   #006: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/>
                     major: File accessibility
                     minor: Unable to open file
2023-05-25 17:46:21.729293   ERROR: exception_during_observer_callback (state_control)
    callback_mode: transitioning
    cause: Dynamic exception type: boost::filesystem::filesystem_error
           std::exception::what: boost::filesystem::status: Permission denied [system:13]: "/home/nanoporeseq/>

I assume the first error message is just a consequence of this latter, but I cannot figure out why the my FAST5 file cannot be open. I am using the bulk FAST5 file downloaded from the link on the readfish instructions.

I have checked the permissions of the FAST5 file, and everything seems okay (no restrictions to open the file, as -rwxr-xr-x 1). I have also checked the integrity the FAST5 using the hdf5 commands h5dump and h5debug and confirmed that it is not corrupted.

I am wondering now if there is a problem with my system's hdf5, which maybe is not being able to access the H5Fopen function. I have tried reinstalling hdf5-tools and the HDF5 library and development files libhdf5-dev, but the problem persists...

Has anyone faced a problem like this? Do you have any idea what the problem could be and how I could resolve it?

I very much appreciate any help!
Diego

[alexomics]

It looks like the FAST5 file that you are trying to use is a read FAST5 file from a run. Simulation requires a bulk FAST5 file. We provide one at http://s3.amazonaws.com/nanopore-human-wgs/bulkfile/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5 that can be used for playback, please note that this file is 21Gb

[mattloose]

I don't think you are using the correct file for playback.

The file you should be using is:
http://s3.amazonaws.com/nanopore-human-wgs/bulkfile/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5

The file you appear to be using is
/home/nanoporeseq/Desktop/readfish/fast5_readfish/fast5_readfish/FAL43730_b1347b8a1cd19a950e6bf1c7d70fb7a32d50ecc7_10.fast5

This file is not a bulkfile.

I can see that @alexomics has just said the same thing :-)

[diego-ds-souza]

Thanks for the quick response, @alexomics and @mattloose !

Sorry, I mixed things up and copied the error messages from two different runs. The copied part from the control_server_log-*.txt is from a run using the correct file, PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5, which I had renamed as readfish.fast5.

But I guess there was something wrong with my bulk file, because I re-download the bulk FAST5 file and reinstalled everything from scratch, and now my playback run works!

Thank you very much!

[diego-ds-souza]

Hi everyone,

I am now having trouble passing the step 5 of the Testing basecalling and mapping, following https://github.com/LooseLab/readfish/tree/refactor

I am using these commands with the conda readfish environment activated:

readfish targets --device MN23485 --experiment-name "RU Test basecall and map" --toml /home/nanoporeseq/Desktop/human_chr_selection.toml --log-file ru_test.log

And I get the following Connection error:

(readfish) nanoporeseq@nanoporeseq-Dell-G16-7620:~$ readfish targets --device MN23485 --experiment-name "RU Test basecall and map" --toml /home/nanoporeseq/Desktop/human_chr_selection.toml --log-file ru_test.log
2023-06-09 12:23:40,324 ru.ru_gen /home/nanoporeseq/miniconda3/envs/readfish/bin/readfish targets --device MN23485 --experiment-name RU Test basecall and map --toml /home/nanoporeseq/Desktop/human_chr_selection.toml --log-file ru_test.log
2023-06-09 12:23:40,324 ru.ru_gen batch_size=512
2023-06-09 12:23:40,324 ru.ru_gen cache_size=512
2023-06-09 12:23:40,324 ru.ru_gen channels=[1, 512]
2023-06-09 12:23:40,324 ru.ru_gen chunk_log=None
2023-06-09 12:23:40,324 ru.ru_gen command=targets
2023-06-09 12:23:40,324 ru.ru_gen device=MN23485
2023-06-09 12:23:40,324 ru.ru_gen dry_run=False
2023-06-09 12:23:40,324 ru.ru_gen experiment_name=RU Test basecall and map
2023-06-09 12:23:40,324 ru.ru_gen func=<function run at 0x7f5ad1f5c820>
2023-06-09 12:23:40,324 ru.ru_gen host=127.0.0.1
2023-06-09 12:23:40,324 ru.ru_gen log_file=ru_test.log
2023-06-09 12:23:40,324 ru.ru_gen log_format=%(asctime)s %(name)s %(message)s
2023-06-09 12:23:40,324 ru.ru_gen log_level=info
2023-06-09 12:23:40,324 ru.ru_gen paf_log=None
2023-06-09 12:23:40,324 ru.ru_gen port=None
2023-06-09 12:23:40,324 ru.ru_gen run_time=172800
2023-06-09 12:23:40,324 ru.ru_gen throttle=0.4
2023-06-09 12:23:40,324 ru.ru_gen toml=/home/nanoporeseq/Desktop/human_chr_selection.toml
2023-06-09 12:23:40,324 ru.ru_gen unblock_duration=0.1
2023-06-09 12:23:40,324 ru.ru_gen workers=1
2023-06-09 12:23:40,327 ru.ru_gen Initialising minimap2 mapper
2023-06-09 12:23:45,628 ru.ru_gen Mapper initialised
2023-06-09 12:23:45,693 ru.ru_gen This experiment has 1 region on the flowcell
2023-06-09 12:23:45,693 ru.ru_gen Using reference: /home/nanoporeseq/Desktop/minimap2/Homo_sapiens_assembly38.fasta.mmi
2023-06-09 12:23:51,209 ru.ru_gen Region 'select_chr_21_22' (control=False) has 2 contigs of which 2 are in the reference. There are 4 targets (including +/- strand) representing 3.03% of the reference. Reads will be unblocked when classed as single_off or multi_off; sequenced when classed as single_on or multi_on; and polled for more data when classed as no_map or no_seq.
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
Traceback (most recent call last):
  File "/home/nanoporeseq/miniconda3/envs/readfish/bin/readfish", line 8, in <module>
    sys.exit(main())
  File "/home/nanoporeseq/miniconda3/envs/readfish/lib/python3.8/site-packages/ru/cli.py", line 43, in main
    args.func(parser, args)
  File "/home/nanoporeseq/miniconda3/envs/readfish/lib/python3.8/site-packages/ru/ru_gen.py", line 506, in run
    simple_analysis(
  File "/home/nanoporeseq/miniconda3/envs/readfish/lib/python3.8/site-packages/ru/ru_gen.py", line 160, in simple_analysis
    caller = Caller(
  File "/home/nanoporeseq/miniconda3/envs/readfish/lib/python3.8/site-packages/ru/basecall.py", line 41, in __init__
    self.connect()
  File "/home/nanoporeseq/miniconda3/envs/readfish/lib/python3.8/site-packages/pyguppy_client_lib/pyclient.py", line 184, in connect
    raise exception_type(exception_message)
RuntimeError: Server response not formatted correctly: <result.bad_reply: 14>

This error is very similar to the one discussed (and resolved) in #188. I am following the same instructions that worked for that case, providing the path to readfish, but it is not working for me. These are the commands I am using:

which readfish

Which returns:

/home/nanoporeseq/miniconda3/envs/readfish/bin/readfish

So I am running:

/home/nanoporeseq/miniconda3/envs/readfish/bin/readfish targets --device MN23485 --experiment-name "RU Test basecall and map" --toml /home/nanoporeseq/Desktop/human_chr_selection.toml --log-file ru_test.log

Everything before step 5, including the Testing unblock response works perfectly. Just for the record, I am copying here the commands I used to generate the index of the human genome Homo_sapiens_assembly38.fasta.mmi using minimap2:

./minimap2 -x map-ont -d Homo_sapiens_assembly38.fasta.mmi test/Homo_sapiens_assembly38.fasta

Then I modified the toml file as follows:

[caller_settings]
config_name = "dna_r9.4.1_450bps_fast"
host = "127.0.0.1"
port = 5555

[conditions]
reference = "/home/nanoporeseq/Desktop/minimap2/Homo_sapiens_assembly38.fasta.mmi"

[conditions.0]
name = "select_chr_21_22"
control = false
min_chunks = 0
max_chunks = 12
targets = ["chr21", "chr22"]
single_on = "stop_receiving"
multi_on = "stop_receiving"
single_off = "unblock"
multi_off = "unblock"
no_seq = "proceed"
no_map = "proceed"

And validated it:

readfish validate /home/nanoporeseq/Desktop/human_chr_selection.toml

Everything seems okay, although this part of the message is slightly different from what is shown in https://github.com/LooseLab/readfish/tree/refactor:

Region 'select_chr_21_22' (control=False) has 2 contigs of which 2 are
in the reference. There are 4 targets (including +/- strand)
representing 3.03% of the reference. 

Here is the entire validation response:

😻 Looking good!
Generating experiment description - please be patient!
This experiment has 1 region on the flowcell

Using reference:
/home/nanoporeseq/Desktop/minimap2/Homo_sapiens_assembly38.fasta.mmi

Region 'select_chr_21_22' (control=False) has 2 contigs of which 2 are
in the reference. There are 4 targets (including +/- strand)
representing 3.03% of the reference. Reads will be unblocked when
classed as single_off or multi_off; sequenced when classed as
single_on or multi_on; and polled for more data when classed as no_map
or no_seq.

Does anyone know what could be causing this Connection error?

Thank you very much!

[Adoni5]

HI @diego-ds-souza

Could you check this comment on this issue? I think it's the same thing. #221 (comment)

Cheers
Rory