Skip to content

Commit

Permalink
add docs for utils.download
Browse files Browse the repository at this point in the history
  • Loading branch information
zoghbi-a committed Oct 18, 2023
1 parent ef906c9 commit c31add4
Showing 1 changed file with 43 additions and 0 deletions.
43 changes: 43 additions & 0 deletions docs/utils/download.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
.. _pyvo-download:

**************************************************
Download Utilities (`pyvo.utils.download`)
**************************************************

The download utilities provide methods for downloading data once a link to it
it obtained. These can be considered an advanced version of `~pyvo.dal.Record.getdataset` that can handle
data from standard on-prem servers as well as cloud data. For now only AWS is supported.

There two methods with the same call signature: `http_download` and `aws_download`. The first handles
standard links from on-prem servers, while the second downloads data from the `Amazon S3 storage`_.



Example Usage
==============
```python
data_url = 'https://heasarc.gsfc.nasa.gov/FTP/chandra/data/byobsid/2/3052/primary/acisf03052N004_cntr_img2.jpg'
image_file = http_download(url=data_url)
s3_uri = 's3://nasa-heasarc/chandra/data/byobsid/2/3052/primary/acisf03052N004_cntr_img2.jpg'
image2_file = aws_download(uri=data_url)
```
or
```python
s3_key = 'chandra/data/byobsid/2/3052/primary/acisf03052N004_cntr_img2.jpg's
s3_bucket = 'nasa-heasarc'
image2_file = aws_download(bucket=s3_bucket, key=s3_key)
```

If the aws data requires authentication, a credential profile (e.g. `aws_user` profile in ``~/.aws/credentials``) can be passed
```python
image2_file = aws_download(bucket=s3_bucket, key=s3_key, aws_profile='aws_user')
```
A session (instance of ``boto3.session.Session``) can also be passed instead (see detials in `AWS session documentation`_).
```python
s3_session = boto3.session.Session(aws_access_key_id, aws_secret_access_key)
image2_file = aws_download(bucket=s3_bucket, key=s3_key, session=s3_session)
```

.. _Amazon S3 storage: https://aws.amazon.com/s3/
.. _AWS session documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

0 comments on commit c31add4

Please sign in to comment.