GIN as alternative to OSF?

[effigies]

Quick note, since OSF is apparently tricky: http://gin.g-node.org/

[oesteban]

Yes, OSF is becoming tricky :(

[oesteban]

It seems repositories must be created manually :(

http://handbook.datalad.org/en/latest/basics/101-139-gin.html#using-gin-as-a-data-source-behind-the-scenes

[yarikoptic]

FWIW, didn't try, but there is gogs (which is what gin is based on) end-point for creating new repos: https://github.com/gogs/docs-api/tree/master/Repositories#create . We just seems didn't bother creating some create-sibling-gogs (or -gin) yet at "datalad level".

[effigies]

Was able to create https://gin.g-node.org/markiewicz/tpl-MyTestTemplate with:

curl -v -H @tokens/gin.header https://gin.g-node.org/api/v1/user/repos -d name="tpl-MyTestTemplate"

tokens/gin.header is a file of the form

Authorization: token {ACCESS_TOKEN}

Where the token was created at: https://gin.g-node.org/user/settings/applications

[oesteban]

I can confirm that worked @effigies.

I've set up an organization templateflow at GIN, @yarikoptic please let me know if you want me to add your username to it (if so, tell me your username over there, please).

I'll start today figuring out the transfer/addition of GIN - I may even send a PR to datalad to handle GIN siblings !

[oesteban]

@yarikoptic - I followed the handbook's instructions to add gin as a data source (https://handbook.datalad.org/en/latest/basics/101-139-gin.html#using-gin-as-a-data-source-behind-the-scenes)

It seems that annexed files are stored at GIN too - e.g., https://gin.g-node.org/templateflow/tpl-MNI152NLin6Asym

However, if you datalad install that dataset and then try to datalad get, some resources fail with:

[ERROR  ] not available; (Note that these git remotes have annex-ignore set: gin origin) [get(/home/oesteban/datalad/templateflow/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-02_desc-b0_mask.nii.gz)]
get(error): /home/oesteban/datalad/templateflow/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-02_desc-b0_mask.nii.gz (file) [not available; (Note that these git remotes have annex-ignore set: gin origin)]

Another interesting thing that happened is that resources that did download, were pulled [from web...] (which I guess should say [from gin...] if I had set up the sibling correctly.

What am I missing?

[yarikoptic]

Make sure that you don't have .git suffix in the url for gin, ie https://gin.g-node.org/templateflow/tpl-MNI152NLin6Asym is ok, https://gin.g-node.org/templateflow/tpl-MNI152NLin6Asym.git is not.

Edit: ref datalad/datalad#5144 (comment) and there in

[oesteban]

Indeed that fixed the issue.

I'm now trying to configure the sibling correctly for tpl-MNI152NLin2009cAsym, which is one of the first datasets where I configured GIN:

$ datalad siblings configure -d . --name gin-update --pushurl [email protected]:/templateflow/tpl-MNI152NLin2009cAsym.git --url https://gin.g-node.org/templateflow/tpl-MNI152NLin2009cAsym --as-common-datasrc gin
CommandError: 'git -c diff.ignoreSubmodules=none annex initremote gin type=git location=https://gin.g-node.org/templateflow/tpl-MNI152NLin2009cAsym autoenable=true -c annex.dotfiles=true -c 'remote.gin-update.annex-ssh-options=-o ControlMaster=auto -S /Users/oesteban/Library/Caches/datalad/sockets/40d58c92' -c annex.retry=3 -c 'remote.gin-update.annex-ssh-options=-o ControlMaster=auto -S /Users/oesteban/Library/Caches/datalad/sockets/40d58c92'' failed with exitcode 1 under /Users/oesteban/tmp/templateflow/tpl-MNI152NLin2009cAsym
git-annex: There is already a special remote named "gin". (Use enableremote to enable an existing special remote.)

So, the question is - once you've made the mistake of setting the HTTP URL with .git suffix, how do you fix it so that, when you datalad clone (or git clone in this case, which is done automatically by the GitHub action), the proper remote is set up?

[oesteban]

@effigies - have you found the API section where creating repos under an organization is explained?

[effigies]

It's POST /org/:org/repos. Need to play with whether one of these orgs should be the name of the org or if it should be part of the repo name in the post data.

[effigies]

See https://github.com/gogs/docs-api/tree/master/Repositories#create that Yarik posted.

[oesteban]

curl -v -H @tokens/gin.header https://gin.g-node.org/api/v1/org/:templateflow/repos -d name="tpl-MyTestTemplate"

I get unauthorized with the above. The token corresponds to nipreps-admin, who is an admin of the org.

Something obvious I'm not seeing ? (thanks much for the help!)

[oesteban]

curl -v -H @tokens/gin.header https://gin.g-node.org/api/v1/org/:templateflow/repos -d name="tpl-testrepo"

but now with the right token, gives me

{"message":"Organization does not exist","url":"https://github.com/gogs/docs-api"}

which is confusing and worrisome (that URL pointing at gogs)

EDIT (and SOLVED) - scratch the above, this worked out:

curl -v -H @tokens/gin.header https://gin.g-node.org/api/v1/org/templateflow/repos -d name="tpl-testrepo"

@yarikoptic - the question of how to fix the remote name when the repo is checked out is still standing :(

[yarikoptic]

FWIW It is GIN days seems to me. @mih is also working on create-sibling-gogs datalad/datalad#5935

which is confusing and worrisome (that URL pointing at gogs)

IMHO it is ok, since gin is just a fork of gogs, and I guess all of the API end points are coming from gogs

So, the question is - once you've made the mistake of setting the HTTP URL with .git suffix, how do you fix it so that, when you datalad clone (or git clone in this case, which is done automatically by the GitHub action), the proper remote is set up?

if to fix within what git-annex knows about that remote, one way (using the case of recent similar question -- lnnrtwttkhn/highspeed-analysis#2 (comment))

• git remote remove gin (to remove it from .git/config)
• git annex enableremote gin location=https://gin.g-node.org/lnnrtwttkhn/highspeed-decoding - which would fix up remote.log entry for it with the new location within git-annex branch and add that remote back to known to git.

[oesteban]

Okay, I think I got it. I had to checkout git-annex then run the two commands you wrote down, commit, push and checkout master back.

[oesteban]

Cool, all datasets are now mirrored at GIN. I'll figure out next how to make it the default mode of distribution.