Skip to content

Commit

Permalink
Update S3 walkthrough with steps from #1224
Browse files Browse the repository at this point in the history
As amazon s3 depcrecated ACL functionality the usecase and a git annex initremote
parameter relied on, it broke.
Thanks to @NickleDave for the heads-up and the fixes, which should make
it work again given a recent enough git-annex
  • Loading branch information
adswa committed Jul 4, 2024
1 parent 324a910 commit 164eba2
Showing 1 changed file with 19 additions and 3 deletions.
22 changes: 19 additions & 3 deletions docs/basics/101-139-s3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Initialize the S3 special remote
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The steps below have been adapted from instructions provided on `git-annex documentation <https://git-annex.branchable.com/tips/public_Amazon_S3_remote>`_.
For more info on the S3 special remote, see `the s3 special remote manpage <https://git-annex.branchable.com/special_remotes/S3>`.

By initializing the special remote, what actually happens in the background
is that a :term:`sibling` is added to the DataLad dataset. This can be verified
Expand All @@ -178,7 +179,7 @@ it will be used again later.
$ BUCKET=sample-neurodata-public
$ git annex initremote public-s3 type=S3 encryption=none \
bucket=$BUCKET public=yes datacenter=EU autoenable=true
bucket=$BUCKET datacenter=EU autoenable=true
initremote public-s3 (checking bucket...) (creating bucket in EU...) ok
(recording state in git...)
Expand All @@ -188,7 +189,6 @@ The options used in this example include:
- ``type=S3``: the type of special remote (git-annex can work with many `special remote types <https://git-annex.branchable.com/special_remotes>`_)
- ``encryption=none``: no encryption (alternatively enable ``encryption=shared``, meaning files will be encrypted on S3, and anyone with a clone of the git repository will be able to download and decrypt them)
- ``bucket=$BUCKET``: the name of the bucket to be created on S3 (using the declared variable)
- ``public=yes``: Set to "yes" to allow public read access to files sent to the S3 remote
- ``datacenter=EU``: specify where the data will be located; here we set "EU" which is EU/Ireland a.k.a. ``eu-west-1`` (defaults to "US" if not specified)
- ``autoenable=true``: git-annex will attempt to enable the special remote when it is run in a new clone, implying that users won't have to run extra steps when installing the dataset with DataLad

Expand All @@ -209,6 +209,22 @@ to "Buckets" to see your newly created bucket. It should only have a single

A newly created public S3 bucket

By default, this bucket and its contents are not publicly accessible.
To make them public, switch to the "Permissions" tab in your buckets S3 console overview, and turn the option "Block all public access" off.

.. figure:: ../artwork/src/aws_s3_bucket_permissions.png

Bucket settings allow making the bucket public

.. find-out-more:: Info on public buckets created prior to April 2023

Amazon S3 buckets created after April 2023 had support for using ACLs for public read access to files.
This functionality has since been deprecated, and only remains for legacy buckets.
When dealing with an old S3 bucket using ACLs like that, it is possible to use the deprecated ``public`` parameter and set it to "yes".

- ``public=yes``: Set to "yes" to allow public read access to files sent to the S3 remote


Lastly, for git-annex to be able to download files from the bucket without requiring your
AWS credentials, it needs to know where to find the bucket. We do this by setting the bucket
URL, which takes a standard format incorporating the bucket name and location (see the code block below).
Expand All @@ -235,7 +251,7 @@ option. For consistency, we'll give the GitHub repository the same name as the d
.. code-block:: console
$ datalad create-sibling-github -d . neuro-data-s3 \
--publish-depends public-s3
--publish-depends public-s3 --access-protocol ssh
[INFO ] Configure additional publication dependency on "public-s3"
.: github(-) [https://github.com/jsheunis/sample-neuro-data.git (git)]
'https://github.com/jsheunis/sample-neuro-data.git' configured as sibling 'github' for Dataset(/Users/jsheunis/Documents/neuro-data-s3)
Expand Down

0 comments on commit 164eba2

Please sign in to comment.