Nextflow fails to read the file on external drive M1 Mac

[bsaini7]

I granted 777 permissions to the files yet it is failing to read the file. Please advise how best I can have the nextflow program read the Plink files from the external drive.

[nebfield]

I'd recommend updating your workflow to the latest version we just published with nextflow pull pgscatalog/pgsc_calc

If you're using the docker profile, you'll need to set up some extra configuration to mount your external hard drive inside the docker containers.

Try changing:

pgsc_calc/nextflow.config

Line 120 in ddb19b3

docker.runOptions = ['-u $(id -u):$(id -g) ', platform].join(' ')

to

docker.runOptions      = ['-u $(id -u):$(id -g) ', platform, '-v /Volumes/LaCie/plink2'].join(' ')

This file should be in $HOME/.nextflow/assets/pgscatalog/pgsc_calc/ by default.

The workflow might be slow running on data from an external hard drive. It's best to use fast (internal) storage whenever possible.

[bsaini7]

Updated docker config:

docker.runOptions      = ['-u $(id -u):$(id -g) ', platform, '-v /Volumes/LaCie/plink2:/Volumes/LaCie/plink2'].join(' ')

[bsaini7]

Thanks @nebfield.

That did help. I had to update the suggested config slightly, -v /Volumes/LaCie/plink2:/Volumes/LaCie/plink2, to include the destination path for docker.

Unfortunately, I ran into the infamous 137 errors after that. I am using the suggested synthetic data, and there is several terra gigs of it. I tried with both bfiles set and pfiles set but got the same error status 137.

I might investigate the following plink command and why it is a memory hog. I wonder if it could be distributed on a Hadoop cluster with map reduce.

#!/bin/bash -euo pipefail
plink2 \
    --threads 2 \
    --memory 16384 \
    --missing vcols=fmissdosage,fmiss \
    --new-id-max-allele-len 100 missing --allow-extra-chr \
    --set-all-var-ids '@:#:$r:$a' \
    --var-id-multi @:# \
    --bfile synthetic_v1_chr-11  \
    --make-just-bim zs \
    --out GRCh38_cineca_11

# cross platform (mac, linux) method of preserving symlinks
cp -a synthetic_v1_chr-11.bed GRCh38_cineca_11.bed
cp -a synthetic_v1_chr-11.fam GRCh38_cineca_11.fam
gzip *.vmiss

cat <<-END_VERSIONS > versions.yml
PLINK2_RELABELBIM:
    plink2: $(plink2 --version 2>&1 | sed 's/^PLINK v//; s/ 64.*$//' )
END_VERSIONS

[nebfield]

HAPNEST is really big and I wouldn't expect it to run on a laptop unfortunately. Smaller datasets like the human genome diversity project or 1000 genomes do run on most normal laptops OK.

We tend to use the "small" HAPNEST synthetic data for development and testing. It's available to download from the same biostudies page if you'd like to experiment with it.

When we run 500,000 samples and 93,000,000 variants (UK Biobank) we deploy the calculator to a SLURM or LSF cluster. The largest process uses 64GB RAM and most plink2 processes use 16 or 32GB of RAM.

[bsaini7]

Thanks for highlighting the smaller dataset at HAPNEST. I just tried with it but keep running into the fails minimum matching threshold error. It is the same error I saw with my own raw data from 23andme, which then led me to find the bigger synthetic dataset.

Do you convert the small dataset to VCF files and then use them on an imputation server first?... or do you have a set of PGS IDs with which enough matching variants would exist to generate a valid report?

[nebfield]

For testing and development we reduce --min_overlap when working with synthetic data. It's a bad idea to do this with real data. On the small HAPNEST dataset PGS001229 should work if you set --min_overlap 0.5 to make a test report 😅

You're correct it's best if input genomes to the workflow have been imputed to get good variant match rates on real biological data.