Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues when conf.py invokes git #17

Open
ntolia opened this issue Jun 3, 2020 · 11 comments
Open

Issues when conf.py invokes git #17

ntolia opened this issue Jun 3, 2020 · 11 comments
Labels
type:task An atomic feature / thing to be implemented

Comments

@ntolia
Copy link

ntolia commented Jun 3, 2020

I have been trying to use sphinx-multiversion but I am running into what seems like a blocking issue because my conf.py calls git internally. Basically, similar to https://protips.readthedocs.io/git-tag-version.html, we use git describe to get the version of the docs we should be building.

However, given what sphinx-multiversion does to copy a tree into /tmp, that fails as the tree copy isn't a real git repo and the conf.py git commands fail.

Any suggestions on how to address this?

@Holzhaus
Copy link
Collaborator

Holzhaus commented Jun 3, 2020

I don't think it's possible at the moment. Maybe it's feasible to make sphinx-multiversion set some environment variables with the actual git repository path and branch name when evaluating the config, and in your conf.py you check if these are set and use them instead. Would this solve your issue?

@ntolia
Copy link
Author

ntolia commented Jun 3, 2020

@Holzhaus Any reason why we couldn't just do a full tree copy in the copy_tree function and use git checkout to switch between branches?

@Holzhaus
Copy link
Collaborator

Holzhaus commented Jun 3, 2020

I think for big repos this would be slow and very memory intensive (all repositories are checked out at once). It also could lead to all kinds of issues when the working tree is not clean. Originally I tried to use a local git clone but that also had problems with remote branches.

@ntolia
Copy link
Author

ntolia commented Jun 3, 2020

@Holzhaus Is it possible to have this as an option then? Would be happy to test and contribute too.

@ntolia
Copy link
Author

ntolia commented Jun 4, 2020

Using the below change gets me past the issue highlighted here. It also does a shallow clone to be efficient. However, I have now run into another problem. We use a plugin (sphinx_substitution_extensions) that recently had a breaking change. So, I can either get old versions to work or new versions. Will have to figure out what to do here.

diff --git a/sphinx_multiversion/git.py b/sphinx_multiversion/git.py
index 8a02f24..9f34fbb 100644
--- a/sphinx_multiversion/git.py
+++ b/sphinx_multiversion/git.py
@@ -146,3 +146,23 @@ def copy_tree(src, dst, reference, sourcepath="."):
         fp.seek(0)
         with tarfile.TarFile(fileobj=fp) as tarfp:
             tarfp.extractall(dst)
+
+
+def shallow_clone(src, dst, reference, sourcepath="."):
+    with tempfile.SpooledTemporaryFile() as fp:
+        cmd = (
+            "git",
+            "clone",
+            "--quiet",
+            "--depth",
+            "1",
+            "-c",
+            "advice.detachedHead=false",
+            "-b",
+            reference.name,
+            "--",
+            src,
+            dst
+        )
+        subprocess.check_call(cmd, stdout=fp)
+        fp.seek(0)
diff --git a/sphinx_multiversion/main.py b/sphinx_multiversion/main.py
index fb84714..a53a252 100644
--- a/sphinx_multiversion/main.py
+++ b/sphinx_multiversion/main.py
@@ -129,7 +129,7 @@ def main(argv=None):
             # Clone Git repo
             repopath = os.path.join(tmp, gitref.commit)
             try:
-                git.copy_tree(gitroot.as_uri(), repopath, gitref)
+                git.shallow_clone(gitroot.as_uri(), repopath, gitref)
             except (OSError, subprocess.CalledProcessError):
                 logger.error(
                     "Failed to copy git tree for %s to %s",

@dolfinus
Copy link

Hello.

I've just faced with the same issue while building docs within docker container.

There is a conf.py script which fetches list of tags and passes it to Read The Docs template.
If I run sphinx-multiversion on locak machine, ir works perfectly, but in case of docker container it fail due to lack of .git folder in tmp dir.

My suggestion is to provide additional script arguments for managing the way of making repo copies:

  1. If argument is not provided, behavior remains unchanged - script uses git archive. Large repositories are happy because .git dir is not copied.
  2. If arguments is provided, git clone is used. If someone is facing just the same issue with accessing .git folder, he/she will just pass this arguments and conf.py will start working as expected.

@Holzhaus
Copy link
Collaborator

Holzhaus commented Aug 6, 2020

I'll try to come up with a PoC PR that will use git clone. By default it will use shallow clone, but that will be configurable.

@BurningEnlightenment
Copy link
Member

I think this should be solvable with git worktree.

@BurningEnlightenment BurningEnlightenment added the type:task An atomic feature / thing to be implemented label Oct 6, 2024
@BurningEnlightenment BurningEnlightenment moved this from Backlog to Ready in multiversion worklog Oct 6, 2024
@Holzhaus
Copy link
Collaborator

Holzhaus commented Oct 6, 2024

I think this should be solvable with git worktree.

Hmm, git worktree modifies the original repo, and I think it's not possible to have the same branch checked out twice, so if the user already uses git worktree this will lead to errors. I think a shallow git clone from a local directory will be the more reliable solution.

@BurningEnlightenment
Copy link
Member

Hmm, git worktree modifies the original repo, and I think it's not possible to have the same branch checked out twice, so if the user already uses git worktree this will lead to errors.

You can have multiple worktrees with the same commit / ref, but you do need to specify --force. The only issue would be that the user would have to do the same while sphinx-multiversion is working.

@BurningEnlightenment
Copy link
Member

git worktree modifies the original repo

I mainly had CI workflows in mind where this usually isn't an issue. On a second thought it might be possible to combine the git worktree -f strategy with an opt-out git clone --mirror, i.e. add a worktree to the bare mirror repository. This way we wouldn't cause any overhead for CI scenarios while keeping dev repositories pristine. I might give that a shot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:task An atomic feature / thing to be implemented
Projects
Status: Ready
Development

No branches or pull requests

4 participants