Project: Pilot Large Data Support Service

[cmbz]

Overview

Pilot Harvard Dataverse large data support services using NESE tape resources for several datasets from Harvard affiliates.

Tasks

• Identify prospective data collections
• Confirm pilot participants
• Coordinate with pilot collection owners to estimate collection size and curation needs
• Create Harvard Dataverse collections
• Coordinate with NESE support staff for tape provisioning
• Coordinate with data collection owners, NESE support, and Dataverse team to upload data
• Coordinate with IQSS finance re. costs
• Assess pilot and propose workflow improvements

Process Development and Management

• Develop an intake process including basic and extended consultation process
• Develop and launch RT queue, including service attributes (will map to service offerings)
• Develop large data collection intake form @sbarbosadataverse
• Define what is included in the Large Data Technical Support service component
• Define what is included in the Large Data Administration service component
• Define what is included in the Large Data Monitoring service component
• End-user instructions for NESE data access, add to Dataverse documentation in Guide and HDV support website
• Document large data curation services process (see hdv-curation issues on large data)

Team

• @sbarbosadataverse
• @cmbz
• @qqmyers
• @landreev

Pilot Participants

School of Engineering and Applied Sciences (Harvard SEAS)

• https://github.com/IQSS/dataverse-HDV-Curation/issues/594

Terabyte-Scale Dataset for Partial Discharge 2 Detection in Covered Conductors via Contact 3 Galvanic Method

-https://db.cs.vsb.cz/datacolls/pdp2020/
-https://help.hmdc.harvard.edu/Ticket/Display.html?id=366158&results=56072ee681951385a6015bad30361ae4

The Oregon-Massachusetts Mammography Database (OMAMA-DB)

See also: https://fly.cs.umb.edu/omama/ and https://github.com/IQSS/dataverse-HDV-Curation/issues/443

• Size: 8T
• Contact: Daniel Haehn, SEAS & UMass Boston

GEOS-Chem 1 and 10 year Benchmark data

*recontact early next year - they lost staff and can't devote to this effort at this time
See also:
https://help.hmdc.harvard.edu/Ticket/Display.html?id=361614
https://github.com/IQSS/dataverse-HDV-Curation/issues/456

• Datasets
  • http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/
  • http://ftp.as.harvard.edu/gcgrid/geos-chem/10yr_benchmark
  • Size: about 1.6 TB:
    • 484G 10yr_benchmarks/
    • 484G total
    • 1.1T 1yr_benchmarks/
    • 1.1T total
  • Note: It would increase as they add new benchmark output. typically release model versions once per quarter. Prior to each release they would generate 1 or more benchmark runs for GCClassic and GCHP. The 10 year benchmarks are generated once per major version release which are less frequent.

DrivAerNet: A Parametric Car Dataset for Data-driven Aerodynamic Design and Graph-Based Drag Prediction

See also: https://github.com/IQSS/dataverse-HDV-Curation/issues/444

• moving their ‘streamlined’ dataset (<1TB) to the DeCoDE lab’s dataverse
• <1TBish
• File sizes: Largest: 1.6G and smallest: 400bytes,
• Total number of files: not sure, user quoted 10^6, but was fuzzy on that.
• File types: .tar.gz, txt, vtk, stl, .sh, data will be packed in .zip format
• By mid to end of May: the full 16TB.
• Author needs to repackage one dataset to meet packaging standards

Under discussion for support

-https://help.hmdc.harvard.edu/Ticket/Display.html?id=362556
-https://help.hmdc.harvard.edu/Ticket/Display.html?id=358471

Related RT Tickets

• #356558: Harvard Dataverse support request: Hosting of breast cancer database
  
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=361614

Issues

• Test the NESE-disk Globus configuration in production  dataverse.harvard.edu#305
• IQSS/10450 - Fixes for Globus download (transfer out) and related info messages dataverse#10451
• Prod. globus infrastructure, next phase: Work out and test the logistics of Globus downloads from NESE dataverse.harvard.edu#261
• Prod. globus infrastructure, next phase: Create documentation for data uploaders dataverse.harvard.edu#260

Related

• Service: Harvard Dataverse Repository Large Data Services #268
• GREI 5: HDV Task - Large Data Support #176
• Project: GEOS-Chem Proof-of-Concept #154
• Project: MOC Proof-of-Concept #105
• Start uploading production data to tape storage at NESE dataverse.harvard.edu#254
• Globus integration: further improvements/refactoring dataverse#10623

Resources

• Google Drive Folder

[cmbz]

Status: March 2024

• Our pilot for computing on large data was completed. See follow up: Epic: Operationalize Large Data and Compute Infrastructure #184 and continuing work here: Start uploading production data to tape storage at NESE dataverse.harvard.edu#254

Closed

• Project: MOC Proof-of-Concept #105

[cmbz]

Status: April 2024

Improvements made to Globus: "The PR fixes the issue with multifile Globus transfers out/downloads not working for draft datasets. It also improves handling of cases where ineligible files are selected for download or Globus transfer by only showing the download/transfer mechanisms that will work (on some files) given the files in the dataset and by improving the UI messages to indicate that files may not be eligible either because the user doesn't have permission (restricted or embargoed), or because the files can't be downloaded/transferred (i.e. files not in a Globus store when the user tries a Globus transfer or files in a Globus store that doesn't support normal downloads when the user selects download.)"

• IQSS/10450 - Fixes for Globus download (transfer out) and related info messages dataverse#10451

[sbarbosadataverse]

Status: May 2024

• Added new Process Development and Management section to this issue
• Tested Globus/NESE download for OMAMA dataset, download successful but found that guestbook is not collecting downloads, could be due to the dataset being in "draft"
• Leonid created a demo dataset for testing Globus access for non Harvard affiliates: https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/URDDBC
• Ceilyn revised documentation for large data services, budget planner, and internal HDV service components
• New section added to record groups interested, but not yet confirmed for the pilot

[cmbz]

Status: June 2024

• RT for large data support is created and populated (still require attributes we want to track)
• Started budget for Jax labs https://help.hmdc.harvard.edu/Ticket/Display.html?id=363736
• Started Globus instructions for users for OMAMA dataset

[sbarbosadataverse]

Status: July 2024

• Draft intake form started
• Instructions to download data from NESE complete
• OMAMA collection updated with NESE download instructions and checklist for curation completed (minus Invoice): https://github.com/IQSS/dataverse-HDV-Curation/issues/443
• Estimate for https://github.com/IQSS/dataverse-HDV-Curation/issues/444 completed and shared with Author
• Connected with Bob for Geos-Chem and received instructions on preparing the data for both NESE and s3 storage. Waiting on his details about the PDFs in the collection.

[cmbz]

Status: August 2024

• Related large data services updates in Service: Harvard Dataverse Repository Large Data Services #268 (comment)
• Pending

[sbarbosadataverse]

Status: September 2024

• Bertarelli is interested in 10TB of storage to NESE disk
• Related issue: Test the NESE-disk Globus configuration in production  dataverse.harvard.edu#305

[cmbz]

Status: October 2024

• Quote was prepared for Bertarelli Collections, standard pricing
• Testing and troubleshooting of Globus upload instructions performed by MIT researcher and Dataverse team
• Draft of Large Data Services intake Qualtrics form was developed and will be tested in November.

[cmbz]

Status: November 2024

• Published: DrivAerNet: A Parametric Car Dataset for Data-driven Aerodynamic Design and Graph-Based Drag Prediction. Discussed how to invoice MIT-content (As affiliates) and the invoice process for this collaboration. This dataset may need to rework some data packaging as they did not follow the required stamdards for one of their datasets.
• Our IT/RT Support team has completed testing the uploading and downloading via Globus and the scope of their work in assisting with these services has been entered into the services table, as requested.
• Large Data Services intake form was finalized

[sbarbosadataverse]

Status: December 2024

• Initiated Bertarelli large data collection support