west update: remotes are not imported from submanifests

[leocencetti]

We are migrating to west to manage our (private) project dependencies/submodules, but we are facing some difficulties integrating west in our GitLab CI workflow: we need to use HTTPS remotes for CI pipelines (for authentication purposes), but we would like to keep using SSH locally.

It would be nice if remotes could be overridden way as projects, e.g. with a 00-ci.yml sub-manifest, included only when needed. For example:

# west.yml
manifest:
    version: "0.10"
    remotes:
        - name: origin
          url-base: [email protected]:group
    defaults:
        remote: oi-origin
    projects:
        - name: moduleA
          path: modules/moduleA
          repo-path: moduleA.git
self:
    import: submanifests 

# submanifests/00-ci.yml
manifest:
    version: "0.10"
    remotes:
        - name: origin
          url-base: https://gitlab.com/group

In this example, the repo-path for moduleA would resolve to:

• [email protected]:group/moduleA.git for local workflows
• https://gitlab.com/group/moduleA.git for CI workflows

At the current stage, including the 00-ci.yml file has no effect on the remote's base-url (always resolved to SSH).
The only solutions we could find are:

1. Re-declaring all the projects within the 00-ci.yml manifest, with a non-negligible duplication cost.
2. Perform a manual substitution (sed) of the remote's URL before running west update

[marc-hb]

git config --global [email protected]:group.insteadOf https://gitlab.com/group

[leocencetti]

@marc-hb Thank you for the suggestion!
Wouldn't it be more appropriate to handle this within west instead of monkey patching either git (as you suggest) or the manifest (as we were doing)?

I am new to west, so forgive my candor: could the logic that is used for projects be extended to remotes?
This would keep the manifest resolution consistent (overrides possible for both projects and remotes, but in theory extensible to any key)

[marc-hb]

I think "some prefer https: but others prefer ssh:" is not the most common and valid requirement. Why can't your CI use ssh? That's why I thought about "monkey patching".

On the other hand, I could imagine a more common and similar requirement: mirroring. For either performance or access control. For that use case adding a layer of indirection for remotes would be useful too but my preference would be to keep the manifests "clean" and rather use something at the west config level.

This would keep the manifest resolution consistent (overrides possible for both projects and remotes, but in theory extensible to any key)

I don't find that overriding remotes would be "consistent" with overriding projects. Overriding projects changes the code that gets built, that's why it involves some manifest changes. Because the purpose of the manifests is to define which code you get. But overriding remotes does not change the code at all. So you'd have a manifest change without any code change, it does not feel "consistent". If you compare with submodules for instance, these two are handled differently in totally different places.

[marc-hb]

But overriding remotes does not change the code at all. [...] If you compare with submodules ...

Actually, overriding remotes does not change SHA1s but if you use branches then overriding remotes can change the code. See related:

• scripts: build meta file workspace status extension zephyr#40085

Considering how messy this can get, "monkey patching" at the git level seems quite appropriate... what problem do you foresee with it?

[mbolivar-nordic]

Hi and thanks for filing an issue.

The decision not to inherit remotes is a deliberate design choice in west. The reason it is done this way is that you can specify a project's fetch URL in the following different ways: 1. by explicit remote, 2. by default remote, 3. by explicit url. Reference material is here: https://docs.zephyrproject.org/latest/guides/west/manifest.html#projects

I thought about how to treat remotes as first class objects you can inherit from sub-manifests that you import. It gets really messy and I couldn't come up with a clean set of rules for it. So the decision I made is that when you import a project, its fetch URL is completely resolved at import time. You can override the project definition in your parent manifest, but then you have to override everything about it. This leads to a bit more typing for sure, but I think the resulting mental model is cleaner.

[leocencetti]

@marc-hb @mbolivar-nordic thank you for your explanation.

Why can't your CI use ssh?

@marc-hb : Within the GitLab CI framework, private dependencies (submodules) must be referenced with a relative URL to the base "repo-path". Then GitLab pre-appends a token-authenticated base URL (HTTPS) to the relative submodule paths (see here), which has the form: https://gitlab-ci-token:<CI_JOB_TOKEN>@gitlab.com/<path_to_base_repo>.
SSH would be possible (in theory) but it requires storing the private key either in the repo or in the docker image, which is a no-go for us.

when you import a project, its fetch URL is completely resolved at import time. You can override the project definition in your parent manifest, but then you have to override everything about it. This leads to a bit more typing for sure, but I think the resulting mental model is cleaner.

@mbolivar-nordic : that was our deduction as well. Our main problem with this was the resulting duplication and its associated risk of divergence of revisions (e.g. caused by negligence).

In any case, we'll stick to the proposed solution of overriding git URLs.

[marc-hb]

Thanks for sharing Gitlab info, appreciated.

SSH would be possible (in theory) but it requires storing the private key either in the repo or in the docker image, which is a no-go for us.

This sounds like Gitlab secrets don't support ssh keys yet, correct? I had a quick look at the docs and I also found https://gitlab.com/gitlab-org/gitlab/-/issues/235506 but I'm still not 100% sure.

[mbolivar-nordic]

In any case, we'll stick to the proposed solution of overriding git URLs.

Since you have a solution and we have no plans to make remotes importable, I'm going to close this issue. Thanks again for taking the time to file it.

[marc-hb]

Thanks to recent #615 I just discovered that 2018's and still open #28 discussed the same topic