Staging with E2E environment

[devenv]

Hey,
Our current workflow is:

• PR is open, approved, and passes tests (GH actions)
• PR gets merged to main and deployed to staging env
• Once deployed E2E tests run (on staging)
• Dev runs a script that checks that images on staging all passed E2E and pushes them to prod

How would your rearrange this workflow for Mergify?

[jd]

What would you Mergify to automate for you there?

If it's part 1, you can write a rule to merge automatically. I'd go even further and implement a merge queue if you don't have one.

From what you describe, it's unclear if staging is a branch or just an env. If it's a branch, I'm not sure how/why it's different from main?

The last part where "dev runs a script" could probably be something like a mix of Mergify and GitHub Actions, I think I miss details on what's done exactly under the hood.

[devenv]

Thanks for the answer!
Staging is a just-before-prod env, always aligned with prod on main branch.
We use it to run e2e tests and if needed manually check flows on a similar to prod env.
I'm trying to automate, well, everything :) The issue is that in Mergify flow the lifecycle end on merge, but our PR lifecycle has extra steps and until they are finished the PR should remain in the queue.

I think there are two ways to align ourselves to Mergify:

• move e2e tests to before merge (and use speculative checks), so that merge would be the last step, also I'm checking if it's feasible cost-wise
• make staging based on a branch with its own queue, not sure if it's possible (i.e. dev branch -> mergify to staging -> if e2e pass automerge to prod, not sure what's the workflow if they don't pass)

Thanks!

[jd]

Yes, that's indeed a good idea. You are correct that Mergify operates around PR only, not branches.

An idea is not to merge code that does not work and pass tests. Ideally, e2e tests should run on the PR before the merge.

What you can do is something like this:

1. PR gets regular tests being run
2. When PR is ready to be merged (regular test passes + approve), you put it in the merge queue
3. Mergify creates a new branch with the base branch (that is up-to-date) + this PR and that branch gets the regular tests + the e2e tests. This is a good improvement for testing if you don't have that already.
4. If it works, the PR is merged and can be deployed.

You can use speculative checks to do that for multiple PRs at the same time.

If staging is mostly about having a copy of prod, you can have a script that resets staging to prod regularly or whenever you want.
If staging is about testing manually your code before merging it, that's also possible. You can have Mergify copy a PR to a staging branch and merging it automatically.

[devenv]

Thanks! This sounds like a much better direction, is this (mindblowing) workflow possible?:

• PR tests pass and it's approved
• it gets into the queue by @mergify queue
• a merge point is created to check if tests including e2e pass
• if the PR is tagged #manual-check it also goes to staging (can we pause here until dev aproves?)
• if the merge point passes checks it goes to prod

[devenv]

I see we can pause at any time by just adding a check-success=manual 🤔

[jd]

You could pause by having a CI only triggered when approved indeed. I imagine you could trigger a GHA based on a label or something like that.

[devenv]

I think I worked out the most parts, it's basically:

queue_rules:
  - name: default
    merge_conditions:
      - "check-success=e2e"

and GH actions:

name: CI
on:
  pull_request:
    branches: [ "main" ]
  workflow_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Tests
        run: sh test.sh

and

name: E2E
on:
  push:
    branches: [ "mergify/merge-queue/*" ]
  workflow_dispatch:
jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        parallelism: [1]
    steps:
      - uses: actions/checkout@v3
      - name: E2E
        run: sh e2e.sh

So the workflow:

• PR pass the GH checks (tests and approvals)
• @mergifyio queue
• a branch is created mergify/merge-queue/... with e2e check
• mergify merges once e2e pass