Need help working with submodules

[SymbionicNigel]

This question is about

• Installation
• Initializing / Cloning
• Alternate files
• Jinja templates
• Encryption
• Bootstrap
• Hooks
• Other

Folder structure

Base of Project Repository/
├─ .git/ # standard .git folder for full repository
├─ .gitmodules
├─ .secrets/ (submodule) 
│  ├─ .git # submodules use a .git  file
├─ dotfile-utils/ (submodule) # YADM Utility repository
│  ├─ scritps/
│  ├─ config/
│  ├─ yadm/ (submodule) # yadm-dev/yadm

Describe your question

I am working on creating a repository to store tools which I can use alongside yadm to trivialize the process of using submodules and some of the other features integrated with yadm. When testing my configuration with a configured yadm status command I get No such file or directory errors.

I add my utility repo as a submodule in any other project repository (./dotfile-utils/ in the above folder structure) and then with some of the included scripts, install yadm and add another submodule to store dotfiles in for the parent project. Once the dotfiles submodule has been added at ./.secrets/ I have not been able to configure the yadm command to properly run without error messages in the console on the submodule in ./.secrets.

At this point I am trying to run the yadm status command as configured below with my terminal session located at the base of the project repository:

DATA_DIR="$(pwd)/.secrets"
REPO_DIR="$(pwd)/.git/modules/.secrets" # Location of .git information for the submodule
YADM_DIR="$(pwd)/dotfile-utils/config"
yadm --yadm-data $DATA_DIR --yadm-dir $YADM_DIR --yadm-repo $REPO_DIR status

This will correctly run git status command on the ./.secrets/ directory but then when it comes to running the cd_work() function the script is not able to process Alternates and Perms. The string being passed to cd_work() is ../../../.secrets which would be correct if the script was being run from the location the submodules .git information was being stored. That relative path leads to a nonexistent folder above the project directory. This value seems to be set by the call in configure_paths() reading git config core.worktree.

Before this I had not included the --yadm-repo $REPO_DIR argument and I was not able to run any git commands as the checks in require_repo() do not pass. The value read from $YADM_REPO by default points to a .git file for the submodule and not a .git directory like it expects.

Am I not configuring the command or structuring the folders properly to get a configuration like this to work? I appreciate any help in getting past this roadblock and if I need to create an example repository I will do so.

TL;DR:
When trying to work with yadm in a project with submodules, I am unable to successfully run the yadm status command without errors.

[amabilee]

Have you verified if that core.worktree and other relevant configurations are correctly set in your .git/config files for both the main repository and the submodules?

You can check with this:
git config -f .git/modules/.secrets/config core.worktree

and make sure you have initialized your submodules.

[SymbionicNigel]

The output of git config -f .git/modules/.secrets/config core.worktree is ../../../.secrets

Currently I am running the following commands in a script to create and add the submodule through git

SUBMODULE_DIR="./.secrets"
echo "# $SUBMODULE_NAME" >> $SUBMODULE_NAME/README.md
git -C $SUBMODULE_DIR init
git -C $SUBMODULE_DIR add README.md
git -C $SUBMODULE_DIR commit -m "feat: initial commit"
git -C $SUBMODULE_DIR branch -M $BRANCH_NAME
git -C $SUBMODULE_DIR remote add origin $REMOTE_URL
git -C $SUBMODULE_DIR push -u origin $BRANCH_NAME
# Remove newly created repository to make room for submodule
rm -rf "$SUBMODULE_DIR"
git submodule add $REMOTE_URL

Would I need to run some of these commands through yadm or do I need to run yadm --yadm-data $DATA_DIR --yadm-dir $YADM_DIR --yadm-repo $REPO_DIR submodule update --recursive --init after these commands?

[SymbionicNigel]

I tried running the command I had just posted and got the error message fatal: /usr/lib/git-core/git-submodule cannot be used without a working tree. before seeing the No such file or directory messages that I was shown previously when running commands.

I will look into what command or function is throwing that error.

[SymbionicNigel]

Looking at #414, would I need to run yadm --yadm-data $DATA_DIR --yadm-dir $YADM_DIR --yadm-repo $REPO_DIR init -w $DATA_DIR instead of the call of git -C $SUBMODULE_DIR init?

[amabilee]

I tried running the command I had just posted and got the error message fatal: /usr/lib/git-core/git-submodule cannot be used without a working tree. before seeing the No such file or directory messages that I was shown previously when running commands.

I will look into what command or function is throwing that error.

This generally happens when Git operations are being attempted in a context where the working directory is not properly set.

Looking at #414, would I need to run yadm --yadm-data $DATA_DIR --yadm-dir $YADM_DIR --yadm-repo $REPO_DIR init -w $DATA_DIR instead of the call of git -C $SUBMODULE_DIR init?

yadm init -w command sets up the submodule repository with yadm's configurations and ensures the working directory is correctly initialized.

I think this should properly initialize your submodule, configure yadm, and run the status command without errors.

# Variables
SUBMODULE_NAME="secrets"
SUBMODULE_DIR="./.secrets"
REMOTE_URL="<your-remote-url>"
BRANCH_NAME="main"

# Initialize main repository if not already done
git init

# Add the submodule
git submodule add $REMOTE_URL $SUBMODULE_DIR
git submodule init
git submodule update --recursive --init

# Initialize yadm with the correct paths for managing the submodule
yadm --yadm-data "$SUBMODULE_DIR" --yadm-dir "$(pwd)/dotfile-utils/config" --yadm-repo "$(pwd)/.git/modules/$SUBMODULE_NAME" init -w "$SUBMODULE_DIR"

# Add README.md and commit it to the submodule
echo "# $SUBMODULE_NAME" >> $SUBMODULE_NAME/README.md
git -C $SUBMODULE_DIR add README.md
git -C $SUBMODULE_DIR commit -m "feat: initial commit"
git -C $SUBMODULE_DIR branch -M $BRANCH_NAME
git -C $SUBMODULE_DIR remote add origin $REMOTE_URL
git -C $SUBMODULE_DIR push -u origin $BRANCH_NAME

# Ensure the core.worktree is correctly set for the submodule
git config -f .git/modules/$SUBMODULE_NAME/config core.worktree "$(pwd)/.secrets"

# Run yadm commands to update the submodule and check status
DATA_DIR="$(pwd)/.secrets"
REPO_DIR="$(pwd)/.git/modules/$SUBMODULE_NAME"
YADM_DIR="$(pwd)/dotfile-utils/config"

yadm --yadm-data $DATA_DIR --yadm-dir $YADM_DIR --yadm-repo $REPO_DIR submodule update --recursive --init

# Check yadm status
yadm --yadm-data $DATA_DIR --yadm-dir $YADM_DIR --yadm-repo $REPO_DIR status

1.Ensure you have the necessary permissions to execute these commands and that your environment variables are correctly set.
2.Confirm that relative paths used are correct and make sense in your directory structure
3. Verify that the core.worktree configuration and paths are correctly set in your submodule.

If errors persist, check yadm --help for any updated flags or confirm the yadm version, as newer releases might have more support for submodules.