[Dorado] New Dorado Basecalling Workflow Terra

[fraser-combe]

🗑️ This dev branch should be deleted after merging to main.

🧠 Summary

A new Dorado Basecalling Workflow, a GPU-accelerated pipeline for basecalling Oxford Nanopore POD5 files. The workflow includes optional automatic model selection, SAM-to-BAM conversion, and demultiplexing into unique barcode fastq files, with outputs uploaded to a new user defined Terra table for further downstream analysis.

⚡ Impacted Workflows/Tasks

This is a new workflow that does not impact any other workflows

This PR may lead to different results in pre-existing outputs: No

This PR uses an element that could cause duplicate runs to have different results: No

🛠️ Changes

This PR introduces the following changes:

• New Workflow: Dorado Basecalling Workflow version 1.0.
• Optional Inputs: Added use_auto_model flag for automatic model selection.
• Manual and Auto Model Options: Supports both predefined models and automatic selection (sup, hac, fast).
• SAM-to-BAM Conversion: Integrated SAMTools task for efficient data handling.
• Demultiplexing: Added demux step to create barcode-specific FASTQ files.
• Terra Integration: Outputs transferred to Terra, with a table generated for downstream workflows.

⚙️ Algorithm

• New Tasks:
  1. Dorado Basecall: Converts POD5 files to SAM using GPU acceleration. Uses a new Dorado Staph-B Docker image v0.80
     https://github.com/StaPH-B/docker-builds/tree/master/dorado/0.8.0
  2. SAMTools Convert: Converts SAM files to BAM.
  3. Dorado Demultiplexing: Creates barcode-specific FASTQ files.
  4. File Transfer: Uploads FASTQ files to Terra.
  5. Terra Table Creation: Generates Terra table from the uploaded FASTQ files.

➡️ Inputs

• New Inputs:
  • use_auto_model (Boolean): Enables automatic model selection.
  • model_accuracy (String): Specifies model accuracy if using auto-selection (sup, hac, fast).
  • fastq_file_name (String): Prefix for output FASTQ files.
  • fastq_upload_path (String): Path to Terra for uploading FASTQ files.
  • kit_name (String): Specifies sequencing kit for adapter/barcode trimming.

⬅️ Outputs

• New Outputs:
  • basecalled_fastqs: Array of FASTQ files generated from basecalling.
  • demuxed_fastqs: Array of FASTQ files generated from demultiplexing.
  • logs: Logs generated during the demux step.
  • terra_table_tsv: TSV file for uploading to Terra.

🧪 Testing

• Tested locally with simulated POD5 inputs and GPU resources.
• Tested in Terra with multiple pod5 files - produced expected barcodes
  Test 1. With 9 Rabies pod5 files from 2 barcodes (manual model)
  -https://app.terra.bio/#workspaces/theiagen-training-workspaces/Theiagen_FCombe_sandbox/job_history/889322c2-19f0-4092-ac7f-4863e676b28a
  

Test 2. 24 pod5 files from 2 barcodes (manual model)
https://app.terra.bio/#workspaces/theiagen-training-workspaces/Theiagen_FCombe_sandbox/job_history/9bef28ea-82ba-4406-8545-f32de7e07e02

test 3. 24 files from 2 barcodes (auto mode)
https://app.terra.bio/#workspaces/theiagen-training-workspaces/Theiagen_FCombe_sandbox/job_history/cead789e-c737-4541-a6ed-d9b907493ee1

output terra table example

• Edge Case Handling: Verified workflow behavior with missing inputs and unsupported models.
• Terra Integration: Confirmed successful transfer and Terra table generation with sample data.

Suggested Scenarios for Reviewer to Test

1. Basecalling with Auto Model Selection: Run with the use_auto_model flag enabled.
2. Manual Model Input: Test with a specific dorado_model path and confirm outputs.
3. Demultiplexing: Verify barcode-specific FASTQ outputs.
4. Edge Case: Provide incomplete inputs (e.g., missing kit_name) to confirm error handling.
5. Terra Table Generation: Confirm Terra table creation and FASTQ uploads with valid inputs.

🔬 Final Developer Checklist

• The workflow/task has been tested and results, including file contents, are as anticipated
• The CI/CD has been adjusted and tests are passing (Theiagen developers)
• Code changes follow the style guide
• Documentation and/or workflow diagrams have been updated if applicable (Theiagen developers only)

🎯 Reviewer Checklist

• All changed results have been confirmed
• You have tested the PR appropriately (see the testing guide for more information)
• All code adheres to the style guide
• MD5 sums have been updated
• The PR author has addressed all comments
• The documentation has been updated

[fraser-combe]

outputs working and documentation updated, see ,https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/CDPH_Bioinformatics_Development/job_history/79417a5e-da8c-4fdc-aa61-f28de8490bba

[kapsakcj]

I will test 3 different workflows and report back:

1. using fast as dorado_model input string

• wf here: https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/CDPH_Bioinformatics_Development/job_history/c44f45c8-401f-4643-8492-dedcab07d143
• need to review outputs and logs ⚠️
• ✅ everything looks good. Correct dorado_model string was output, along with analysis date. FASTQ files copied to correct bucket location and Terra table updated accordingly (although the table was overwritten by the below workflow, as I expected)

2. using [email protected]

• wf here: https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/CDPH_Bioinformatics_Development/job_history/2028a37c-feaa-45b3-9ebf-b405c6ab3468
• successful, but need to review outputs and logs ⚠️
• ✅ everything looks good here as well!

3. using sup (as this will be the recommended input param for our users)

• wf here: https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/CDPH_Bioinformatics_Development/job_history/a9708f03-9732-483b-9c60-e786473defdd
• need to review outputs and logs ⚠️
• ✅ Everything looks good here as well. I'm assembling 4 samples with TheiaProk_ONT to confirm output FASTQs are valid and able to be processed. I expect it to pass like last time

EDIT: all of these wfs were run AFTER making the below commit 82a7962 bug fix

[kapsakcj]

TheiaProk_ONT ran successfully on the FASTQs produced by my test above with SUP dorado model 👍 https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/CDPH_Bioinformatics_Development/job_history/238d0f1f-fe13-4823-8846-b0774fb75e0c

More confirmation the FASTQs produced by this wf are valid for downstream processing