Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VRS: delete listing with hadoop before import_vcf #409

Merged
merged 1 commit into from
Aug 28, 2023
Merged

VRS: delete listing with hadoop before import_vcf #409

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 2 additions & 15 deletions gnomad_qc/v4/annotations/vrs_annotation_batch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,23 +313,10 @@ def main(args):
batch_vrs.run()

logger.info(
"Batch jobs executed, preparing to read in sharded VCF from prior step."
" Preparing list of files first using Hail's hadoop_ls method."
"Batch jobs executed, importing annotated shards into Hail Table..."
)

annotated_file_dict = hl.utils.hadoop_ls(
f"gs://{working_bucket}/vrs-temp/annotated-shards/annotated.vcf/*"
)

annotated_file_list = [
annotated_file_item["path"] for annotated_file_item in annotated_file_dict
]

logger.info("File list created. Now reading in annotated shards.")

# Import all annotated shards
ht_annotated = hl.import_vcf(
annotated_file_list,
f"gs://{working_bucket}/vrs-temp/annotated-shards/annotated.vcf/*",
force_bgz=True,
reference_genome=assembly,
).rows()
Expand Down