Speed up CI with a two-tier system built on bors

[epage]

clap tests a lot of system combinations

Pros

• Ensure we don't break clap for any of our users

Cons

• This makes it easy to max out our number of parallel runners, slowing down CI results for PRs
• While Github offers hosting for free, we should be mindful of the cost and be respectful with the resources we use

However, with bors, we can get the best of both worlds. As a merge queue, bors does one extra CI check before commits make it to master. We can rely on that to implement a two-tier CI system

• Checks that run on pull requests
• Checks that run on branches, like master, staging, and trying

If a contributor suspects their change would impact a case not covered for pull requests, they can comment bors try on their PR (I'm assuming try is available to contributors, its unclear who has access rights to each command).

So this means for each push to a PR, we can run a fraction of the jobs, completing much more quickly while still verifying nothing will make master red by testing during the merge proces. If it would, it gets kicked out and the PR author gets that feedback.

Proposed implementation:

• Split ci.yml into pr.yml and branch.yml
• pr.yml would not include
  • wasm
  • i686 procs (clap generally doesn't have proc-specific logic)
  • macos-latest (close enough to ubuntu-latest for what clap does)
  • rust beta and nightly (changes are unlikely to break with upcoming Rust versions but instead with MSRV)

Additional speed ups

• (Maybe) Feature combinations be run within the same job (reuse dependency builds / reduce startup times)
• Try https://github.com/Swatinem/rust-cache

[pksunkara]

The issue with doing it is that we don't have enough PRs to actually have the merge queue efficiently reducing CI time. I am not using bors right now even though we have it working because of that frequency.

[pksunkara]

It is a matter of PR frequency. We have 58 jobs. You want to keep 5 on PR and the rest of 53 on master. So, if the PR frequency is 1, the all 58 would run anyway because bors merges that PR.

For us to get PR frequency of at least 2, we would need to approve 3 open PRs in quick succession which happened once in the last 2 years when me and another maintainer were working on our own stuff coinciding with an external contributor working on his own.

[epage]

It sounds like you are talking about relying on bors to batch merges, saving on how many jobs bors runs? That isn't what I'm worried about. Bors/master builds are asynchronous to user processes and users generally don't care when they complete.

What I'm trying to do is

• Reduce time between PR being posted and CI signing off.
• Reduce overall system load, assuming people are posting multiple updates to their PR (fairly common)

With a side benefit of making the UI clearer. Finding failures in the long list is not friendly and sometimes its unclear what the difference was for why one job failed and another passed because names get cut off, etc.

[pksunkara]

Reduce overall system load, assuming people are posting multiple updates to their PR (fairly common)

That is taken care of with github action's concurrency key.

The other reasons are not okay but IMHO I don't think they are as important compared to external contribution experience where an environment/scenario fails after their PR is merged and now they need to create another PR to fix that.

[epage]

The other reasons are not okay

Could you clarify by what you mean here?

I don't think they are as important compared to external contribution experience where an environment/scenario fails after their PR is merged and now they need to create another PR to fix that.

When bors fails to merge, isn't the PR left open and they just update it, like they would a normal CI failure?

[pksunkara]

Could you clarify by what you mean here?

My bad, I meant to type "The other reasons are okay".

When bors fails to merge, isn't the PR left open and they just update it, like they would a normal CI failure?

You are right. Let me do this setup as you recommended the coming weekend.

[pksunkara]

https://www.reddit.com/r/rust/comments/prhz1u/continuous_integration_with_github_actions_and/hdkbpjl/

[epage]

I personally disagree with some of the approaches matklad takes. I understand the convenience of running everything through cargo test but

• Each of these platform-agnostic checks now runs on all platforms, raising your CI times
• A lot of these checks he moves to cargo test are outside of the inner loop for development while cargo test is in that inner loop and having to workaround failures that don't matter for the inner loop is frustrating.