Welcome to book Discussions!

[adswa]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[petralenzini]

Hi! I'm Petra. I've been trying to get HCP behavioral data to the NDA and am interested in anything that would make data harmonization easier, more robust, or altogether avoidable. Free ReproNim lectures and the friendly guidance of David Kennedy and Satra Ghosh brought me here. Background is in statistical genomics (where we programmed in SAS and shared programs via email) so I'm bit of a fish out of water. Learning the prerequisites quickly in my free time thanks to your high quality resources. Goal: solve a 'beginner' issue in 2020.

[adswa]

Hi Petra, its so nice to have you here! 🤗
Your use case sounds super interesting, and I'm glad the handbook is useful. Very happy to help with questions here or in the matrix channel :)

[kiwimarc]

Im trying to download some dataset with, but I only get the git annex version and not the full file.

My steps are:

1. datalad clone some_url
   • Output:
     Clone attempt: 0%| | 0.00/2.00 [00:00<?, ? Candidate locations/s]
     [INFO ] Detected a filesystem without fifo support.
     | Disabling ssh connection caching.
     [INFO ] Scanning for unlocked files (this may take some time)
     [INFO ] Remote origin not usable by git-annex; setting annex-ignore
     [INFO ] access to 1 dataset sibling s3-PRIVATE not auto-enabled, enable with:
     | datalad siblings -d "folder_structure" enable -s s3-PRIVATE
     install(ok): folder_structure (dataset)
2. datalad get \sub-718213\ses-01\anat\
   • Output:
     get(ok): sub-718211/ses-01/anat/sub-718211_ses-01_T1w.nii.gz (file) [from s3-PUBLIC...]
     get(ok): sub-718211/ses-01/anat/sub-718211_ses-01_T2w.nii.gz (file) [from s3-PUBLIC...]
     get(ok): sub-718211/ses-01/anat (directory)
     action summary:
     get (ok: 3)

From what I understand from the docs this should download the files in the folder I specified, but the files are still git annex files?

Is there anything thing I have missed?

[adswa]

Hi @kiwimarc!
(feel free to post such questions as an issue, as this discussions space isn't used much we might miss posts here more easily)

Your output looks exactly right, and indeed, after the clone and get you should have downloaded the file contents. Do you see an error opening the files when you process them?

In case the symlinks you might see on your system such as

(handbook) adina@muninn in ~/repos/data/ds003633 on git:master
❱ ls -lah sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz
lrwxrwxrwx 1 adina adina 143 Sep  8  2021 sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz -> ../../../.git/annex/objects/1x/9Z/MD5E-s9847733--7e6029603903a43ed5cb9e50d7624107.nii.gz/MD5E-s9847733--7e6029603903a43ed5cb9e50d7624107.nii.gz

lead you to conclude that the file content may not yet be present, the presence of the symlink wouldn't be indicative - an ls -l on files with and without content would look exactly the same. You could check with ls -L, which would work on files that have content retrieved, and fail on files that weren't yet datalad get'ed:

# file content is present
(handbook) adina@muninn in ~/repos/data/ds003633 on git:master
❱ ls -lah sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz
lrwxrwxrwx 1 adina adina 143 Sep  8  2021 sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz -> ../../../.git/annex/objects/1x/9Z/MD5E-s9847733--7e6029603903a43ed5cb9e50d7624107.nii.gz/MD5E-s9847733--7e6029603903a43ed5cb9e50d7624107.nii.gz
(handbook) adina@muninn in ~/repos/data/ds003633 on git:master
❱ ls -L sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz
sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz
# file content isn't present
(handbook) adina@muninn in ~/repos/data/ds003633 on git:master
❱ datalad drop sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz
drop(ok): sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz (file)
(handbook) adina@muninn in ~/repos/data/ds003633 on git:master
❱ ls -lah sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz                         
lrwxrwxrwx 1 adina adina 143 Sep  8  2021 sub-01/ses-movie/anat/sub-01_ses-movie_T1w.nii.gz -> ../../../.git/annex/objects/1x/9Z/MD5E-s9847733--7e6029603903a43ed5cb9e50d7624107.nii.gz/MD5E-s9847733--7e6029603903a43ed5cb9e50d7624107.nii.gz
(handbook) adina@muninn in ~/repos/data/ds003633 on git:master
❱ ls -L sub-01/ses-movie/meg/sub-01_ses-movie_task-movie_run-01_meg.ds/BadChannels
ls: cannot access 'sub-01/ses-movie/meg/sub-01_ses-movie_task-movie_run-01_meg.ds/BadChannels': No such file or directory

You can also query where file content would be by running git annex whereis <PATH>, and you can check how much file content is present locally by running datalad status --annex all.

If file content isn't present despite a successful get, can you point me to the dataset this has happened, so that I might try to reproduce that bug?

[kiwimarc]

I'm not using ls to see the file, but I'm using file and are getting the same output no matter what.
file sub-718211/ses-01/anat/sub-718211_ses-01_T1w.nii.gz
sub-718211/ses-01/anat/sub-718211_ses-01_T1w.nii.gz: symbolic link to ../../../.git/annex/objects/G1/4m/MD5E-s23444253--cc131312cf98a02cd73c2356e806d6e1.nii.gz/MD5E-s23444253--cc131312cf98a02cd73c2356e806d6e1.nii.gz

When I try to unzip the .gz file, by this command: gzip -d -k sub-718211/ses-01/anat/sub-718211_ses-01_T1w.nii.gz
I get the error: gzip: sub-718211/ses-01/anat/sub-718211_ses-01_T1w.nii.gz: Too many levels of symbolic links

datalad status --annex all returns: 353 annex'd files (45.4 MB/4.4 GB present/total size) nothing to save, working tree clean

This is the dataset Im trying to download: https://openneuro.org/datasets/ds003653/versions/1.0.0

[adswa]

Hey, this looks like the download worked completely fine. I also quickly checked the dataset and file, and I can open it with software such as FSLeyes:

In general, the error you see is gzip not wanting to act on symlinked files unless you explicitly instruct it to do so (see the exepted answer to this post). So in order to unzip the file as is, just run gunzip -f sub-718211/ses-01/anat/sub-718211_ses-01_T1w.nii.gz, or, in order to transform the file from a symlink into a regular file, run datalad unlock on the file first.

However, can I ask why you want to unzip the file? (is it a software that doesn't handle files in gzipped format?). Keep a few DataLad/git-annex peculiarities in mind when doing so: The files kept in git-annex (anything symlinked) are write-protected to prevent it from accidental modification. Proper file handling would always include an "unlock" step first to make the file writeable - however, unlocking double the amount of diskspace required to represent the file content (this is written up in chapter 3). Almost all major neuroimaging tools I'm aware of can ingest nii.gz natively, so its worthwhile to check if the software you're using does it out of the box, too. If you give me a pointer about the tool, I can also ask around in our institute with quite a breadth of software tool preferences for tips.

[kiwimarc]

datalad unlock worked!

The matlab tool SPM 12 doesn't work with nii.gz so all the files needs to be uncompressed as noted here

Thank you very much @adswa !