Handling dependencies across different tools for the different gz software and versions

[j-rivero]

The software dependencies for our packages are mainly coded for building in CMakeList.txt. When creating building and CI environments the code is hosted in different places:

• release-tools: dependencies_archive.bash and
• upstream software repositories: configure.bat (for Windows), CI configurations
• -release repositories: debian dep information
• homebrew-simulation repo: brew dependencies in each Formula

First feeling shared with @mjcarroll is that rosdep would be a bit too much and to get full support of it we would probably need to implement part of ROS specific stuff like package.xml.

[j-rivero]

Example of places to update when bumping a major release in ignition:

• gazebodistro files Use msgs6/fuel-tools5/physics3/gazebo4/sdf10 in ignition-dome gazebodistro#5
• update release-tools to compile branches i.e: https://github.com/ignition-tooling/release-tools/blob/93093d3d1f50e006458b2ea2edc13c20cc676577/jenkins-scripts/docker/ign_sensors-compilation.bash#L31-L38
• Formulas for brew
• CMakeLists.txt in every ignition repository
• Github actions in every ignition repository
• release tools DSL to generate nightlies Implement dome view and move nightly to dome #241

[chapulina]

I don't think anyone would disagree that we should have a standard way of defining the dependencies for each version of each package. I believe there are a couple of axes to be discussed:

• declarative vs imperative
• distributed vs centralized

Existing approaches

• packages.apt for Linux apt dependencies (declarative, distributed)

  • GitHub Actions uses it
  • Jenkins uses it since Install dependencies defined on packages.apt file in source code  #317
  • Our source-build documentation uses it

• -release repos for Linux dependencies when packaging (declarative, distributed)

  • Example for ign-gazebo6-release

• Homebrew formulae: macOS dependencies (declarative, centralized)

  • Used for macOS CI and packaging
  • Example for ign-gazebo6

• dependencies_archive.sh for Linux dependencies (imperative, centralized)

  • was used for Ignition before packages.apt
  • still used for Gazebo classic

• gazebodistro for colcon source builds (declarative, centralized)

  • mainly used by Ignition
  • used by tooling such as the pr-collection-labeler

• gzdev for Linux stable/pre-release/nightly repositories (declarative, centralized)

  • Used by GitHub actions
  • Used by Jenkins Linux

declarative vs imperative

I believe we should get rid of any imperative approaches. dependencies_archive.sh is tough to debug because a developer must follow the entire control flow to understand what dependencies came out at the end. It's also confusing for newcomers when they're looking at CI build logs and the entire dependencies archive is printed every time - what does that mean, what is actually meant to be installed for this build?

Proposal: declarative

distributed vs centralized

I believe we've had different degrees of success with our existing distributed and centralized approaches. I'll speak to the advantages of a distributed approach because that's what I've personally seen the best results with.

Development perspective

I believe the experience with packages.apt has been largely successful for developers. Developers introducing a new dependency can make that change as part of their source code changes and get immediate results from CI. No need to open a separate branch on a different repository, then have to manually configure and trigger jobs, or use features like ci_matching_branch.

In that situation, the centralized approach only holds back developers who are working on source code changes and don't care about how the infrastructure works. The distributed approach empowers them to iterate more quickly.

Maintenance perspective

Changing the dependencies of Ignition libraries, especially Ignition dependencies, currently involves a lot of work. The specific tasks are listed on ISSUE_TEMPLATE/bump_libraries.md, and one can look at issues like #574 to see the large number of pull requests that are necessary to update dependencies. That process has various problems, such as:

• Lots of PRs need to be open
  • they're so many and so particularly different that we had to create a script to help with that
  • it was so hard to trigger CI for all these branches that we had to implement the ci_matching_branch feature
• Then all PRs need to be merged in sync

I'd love to simplify that, and I believe that a distributed approach would help here, because our source code is distributed. We can't get rid of a PR to the source code. Then why not leverage that PR to also update as much infrastructure as possible? Imagine if we could configure colcon dependencies, macOS dependencies, release dependencies, enable nightlies... All with a single PR? That can be iterated faster, is less error-prone, it's easier for a newcomer to perform, it's easier to be kept in sync... To me, that's the dream.

Proposal: distribute all the things!

[scpeters]

I don't think anyone would disagree that we should have a standard way of defining the dependencies for each version of each package. I believe there are a couple of axes to be discussed:

• declarative vs imperative
• distributed vs centralized

Existing approaches

• packages.apt for Linux apt dependencies (declarative, distributed)

  • GitHub Actions uses it
  • Jenkins uses it since Install dependencies defined on packages.apt file in source code  #317
  • Our source-build documentation uses it

• -release repos for Linux dependencies when packaging (declarative, distributed)

  • Example for ign-gazebo6-release

• Homebrew formulae: macOS dependencies (declarative, centralized)

  • Used for macOS CI and packaging
  • Example for ign-gazebo6

• dependencies_archive.sh for Linux dependencies (imperative, centralized)

  • was used for Ignition before packages.apt
  • still used for Gazebo classic

• gazebodistro for colcon source builds (declarative, centralized)

  • mainly used by Ignition
  • used by tooling such as the pr-collection-labeler

• gzdev for Linux stable/pre-release/nightly repositories (declarative, centralized)

  • Used by GitHub actions
  • Used by Jenkins Linux

declarative vs imperative

I believe we should get rid of any imperative approaches. dependencies_archive.sh is tough to debug because a developer must follow the entire control flow to understand what dependencies came out at the end. It's also confusing for newcomers when they're looking at CI build logs and the entire dependencies archive is printed every time - what does that mean, what is actually meant to be installed for this build?

Proposal: declarative

distributed vs centralized

I believe we've had different degrees of success with our existing distributed and centralized approaches. I'll speak to the advantages of a distributed approach because that's what I've personally seen the best results with.

Development perspective

I believe the experience with packages.apt has been largely successful for developers. Developers introducing a new dependency can make that change as part of their source code changes and get immediate results from CI. No need to open a separate branch on a different repository, then have to manually configure and trigger jobs, or use features like ci_matching_branch.

In that situation, the centralized approach only holds back developers who are working on source code changes and don't care about how the infrastructure works. The distributed approach empowers them to iterate more quickly.

Maintenance perspective

Changing the dependencies of Ignition libraries, especially Ignition dependencies, currently involves a lot of work. The specific tasks are listed on ISSUE_TEMPLATE/bump_libraries.md, and one can look at issues like #574 to see the large number of pull requests that are necessary to update dependencies. That process has various problems, such as:

• Lots of PRs need to be open

  • they're so many and so particularly different that we had to create a script to help with that
  • it was so hard to trigger CI for all these branches that we had to implement the ci_matching_branch feature

• Then all PRs need to be merged in sync

I'd love to simplify that, and I believe that a distributed approach would help here, because our source code is distributed. We can't get rid of a PR to the source code. Then why not leverage that PR to also update as much infrastructure as possible? Imagine if we could configure colcon dependencies, macOS dependencies, release dependencies, enable nightlies... All with a single PR? That can be iterated faster, is less error-prone, it's easier for a newcomer to perform, it's easier to be kept in sync... To me, that's the dream.

Proposal: distribute all the things!

I don't have a problem with some centralization if the data is simple enough. I think gzdev and gazebodistro are pretty useful, but rosdistro is huge and scary to me because of how much it stores. Storing dependency names for different major versions of software on different versions of an operating system feels complex enough for me that it's much better to distribute that data in the packages.apt files.

A caveat: if we stored all our dependency information in package.xml and then auto-generated the content of gazebodistro or even the homebrew-simulation dependencies, I think that would be nice (similar to how bloom autogenerates the -release debian metadata).

So I would take it on a case-by-case basis and not throw out any of our tools just because they are centralized. But I definitely prefer packages.apt files over the _DEPENDENCIES variables in dependencies_archive.sh.

[j-rivero]

The process of using de-centralized packages.apt is ongoing: #705 #707 #704

[azeey]

We now have package.xml files in Harmonic packages, so we can move toward using rosdep for dependency installation.