ci(e2e-tests): add e2e-tests workflow for CD apps

[JoakimSM]

With continuous delivery we should potentially be testing our apps against more than one backend version. We have set up Cypress testing against all applicable backend versions in the Capture App (minor versions, e.g. 2.38, 2.39 and so on). I have generalised our workflow to make it reusable for other apps. Currently, we are testing against live instances in the Capture app and the workflow reflects this. We will have to adapt some parts of the workflow to make it work with fixtures (if that is indeed something we want).

Please have a look at our routine in the Capture app to familiarise yourself with the idea.

You should follow the instructions in e2e-tests.yml to set it up (found at the top: set secrets, environment variables and required status checks).

The workflow should work for CD apps that have not specified a max version, but also those that have. When there is no max version specified, the workflow will look up the version of the dev instance to figure out the range.

You might notice that the workflow uses two new Github actions that I have created:
dhis2/action-supported-legacy-versions
dhis2/action-instance-version

They are separate actions because we also reuse the latter in our verify-app workflow (we should probably also update the verify-app workflow here at some point)

[ismay]

I've taken a look at this and was thinking that we might be able to simplify this by slightly altering the approach. I'm wondering if we could also solve this by:

1. Hardcoding the matrix for dhis2 versions to test against in our e2e workflow.
2. Automate the update of these versions with a central action, github app or other mechanism. Say send a PR to tagged repos to update these versions after a release (or slightly prior, whatever we want)

That means that the syntax for the workflows for our apps can remain very simple and we move the complexity for keeping these versions up to date to a central location instead of having it in all our repos.

[ismay]

So if I understood Austin's intentions correctly yesterday he does not want to go with anything centralized. So then I assume we want to go in the direction of using an action to deduce the supported versions for the job matrix.

This PR introduces a couple other concepts besides that as well, I was wondering if we could simplify that a little. The labels for example, we talked about using draft pr status for that instead. If we do that we could use the following syntax to enable e2e testing:

jobs:
  build:
    if: github.event.pull_request.draft == false

I think that would already allow us to simplify things a fair bit. Also, as Hendrik said, usually we don't want to skip ci tests, so we could remove that piece of functionality as well. ci skip tags work natively with actions, so that is always an option devs will have. Unless you see a usecase for specifically allowing devs to opt out of e2e testing on ci, let me know if you do.

Also, for cancelling previous runs, to speed things up, we could use:

concurrency:
  group: ${{ github.workflow}}-${{ github.ref }}
  cancel-in-progress: true

For the containers I feel it's fine to hardcode the amount we want. In my opinion the simplicity of writing:

      matrix:
        # run 3 copies of the current job in parallel
        containers: [1, 2, 3]

outweighs the convenience and maintenance burden of having it deduced from an env var.

There's a couple other details I think we could take a look at, but let me know what you think of the above first.

[JoakimSM]

@ismay Thanks for the input. Updated with your suggestions above (like we also discussed on slack)

[jenniferarnesen]

Curious is checkout is actually needed in this job?

[JoakimSM]

@jenniferarnesen yes, it is needed for the custom action dhis2/action-supported-legacy-versions@v1. I think it makes sense to keep it here (and not do this internally in the action), the same way as cypress-io/github-action@v4 requires you to checkout before using that action.

[JoakimSM]

I can move the checkout one step down though (makes more sense)

[jenniferarnesen]

Is there a container for node 16? It seems this would conflict with line 75.

I noticed that some of my line-listing-app tests are failing because they use javascript that is not supported in node 14 but is supported in node 16, and I'm wondering if this might be why

[jenniferarnesen]

I'm also wondering what container is for? I removed this (line-listing-app) and the tests ran fine. Also, I have been getting occasional errors that look like this:
Error: EACCES: permission denied, open '/github/home/.cache/Cypress/8.7.0/binary_state.json'

[JoakimSM]

Updated to use docker image cypress/browsers:node16.17.0-chrome106. Removed unnecessary node step.

[jenniferarnesen]

For the template we probably shouldn't have a real record key. Maybe use this instead:
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

[jenniferarnesen]

Should the e2e tests run on forks? The e2e job in the verify-app workflow doesn't allow it:
if: '!github.event.push.repository.fork'
https://github.com/dhis2/workflows/blob/master/ci/dhis2-verify-app.yml#L88

[JoakimSM]

Makes sense. Disabled for forks.

[JoakimSM]

Summary of approaches

1.1 Use label "e2e-tests" to trigger the test run. We can optionally rerun tests on push when the label is present (A new commit will «invalidate» the previous results).

Cons: Adding a different label will also trigger the workflow. Skipping the test step when the label is something else than ‘e2e-tests’ will result in the test steps disappearing from the PR status checks (I.e. if you previously had a successful run these will disappear when you add a new label). You can readd the ‘e2e-tests’ label to rerun the tests of course, but only the status checks for the last added label will ever appear in the PR (so it gets even worse in a hypothetical example where we are having a different label that also adds status checks, only the status checks for one of the labels will ever appear in the PR status check list).

1.2 Use a checkbox in the body description to trigger the test run (basically this: https://glebbahmutov.com/blog/re-run-the-tests-with-checkbox/). We can decide if we want to rerun tests on push when the checkbox is already checked (A new commit will «invalidate» the previous results though).

Cons: Basically the same problem as for labels. If we change the body description of the PR the workflow will trigger. If the checkbox is already checked (and unchanged after the edit) and we skip the test run based on this, the status checks will disappear. Hypothetically, if we had multiple checkboxes doing different tasks that should be visible in the status checks and we only run the tasks corresponding to the user interaction, only the tasks for the latest user interaction would show in the status checks.

2. Execute tests on push when the PR is «ready for review».

Pros: works as you expect. When the PR is not in draft, the tests will execute on each new push.

Cons: Not as explicit as the other solutions (when moving your PR to «ready for review» the tests will implicitly run). Can result in more test runs than the other solutions.

Less explored, but should work in theory:

3. Run the test workflow on push only, but use labels or checkboxes as a condition in the workflow to run the tests. I.e., add label ‘e2e-tests’ and wait for a new commit before actually executing the test run.

Cons: need to push new commit before test execution is triggered.

3.1 Build on 3 and automatically push a dummy commit when the label is added / checkbox is checked. Cons: Automatically pushing a dummy commit could be annoying depending on your merge strategy.

4. Run the test workflow on push and on «pr ready for review», but use labels or checkboxes as a condition in the workflow to run the tests. Flip the PR ready state when the label is added / checkbox is checked in a separate workflow.

Cons: If we have other workflows utilising the ready state in some way, we will mess those up. Also, the flipping will be listed in the PR conversation list.

Ideas explored, but met a dead end:

- Have a separate workflow for label added / checkbox checked. This workflow will trigger the actual test workflow using «workflow_dispatch» (through Github CI / API). We get separation between the workflows and the workflow doing the actual testing will only execute when we add the correct label or click the checkbox (i.e. tests will not execute when we add an unrelated label or do an unrelated body description change). Met a dead end because the results from the actual test workflow doesn’t show up in the status checks (https://stackoverflow.com/questions/63288356/how-to-link-a-github-actions-manual-run-to-a-pr)

- Use idea from 1.1/1.2, but look for previous result when an unrelated label is added / an unrelated edit was made (the workflow will run, but can we still get the result of the last run and use that?). Gets complicated/impossible to make a good implementation when you need to look at the «in progress» (cancelled) results. There is no way to determine from the log what the actual action was (can’t see that the workflow ran because the «e2e-tests» label was added).