From f5ea554c96fda85ecfd3160d8a21733497c2eab9 Mon Sep 17 00:00:00 2001 From: Rohan Sharma <117426013+RS-labhub@users.noreply.github.com> Date: Thu, 11 Apr 2024 10:09:38 +0530 Subject: [PATCH] Updated e2e-tests README.md (#26732) --- e2e-tests/README.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/e2e-tests/README.md b/e2e-tests/README.md index d55289a3bd2..0a5787c6432 100644 --- a/e2e-tests/README.md +++ b/e2e-tests/README.md @@ -10,47 +10,47 @@ Please refer to the [dedicated developer documentation](https://developers.matte ##### For pipeline debugging -The E2E testing pipeline's scripts depend on the following tools being installed on your system: `docker`, `docker-compose`, `make`, `git`, `jq`, `node` and some common utilities (`coreutils`, `findutils`, `bash`, `awk`, `sed`, `grep`) +The E2E testing pipeline's scripts depend on the following tools being installed on your system: `docker`, `docker-compose`, `make`, `git`, `jq`, `node`, and some common utilities (`coreutils`, `findutils`, `bash`, `awk`, `sed`, `grep`) Instructions, tl;dr: create a local branch with your E2E test changes, then open a PR to the `mattermost-server` repo targeting the `master` branch (so that CI will produce the image that docker-compose needs), then run `make` in this directory. Instructions, detailed: 1. (optional, undefined variables are set to sane defaults) Create the `.ci/env` file, and populate it with the variables you need out of the following list: * `SERVER`: either `self-hosted` (default) or `cloud`. - * `CWS_URL` (mandatory when `SERVER=cloud`, only used in such case): when spinning up a cloud-like test server that communicates with a test instance of customer web server. + * `CWS_URL` (mandatory when `SERVER=cloud`, only used in such case): when spinning up a cloud-like test server that communicates with a test instance of a customer web server. * `TEST`: either `cypress` (default), `playwright`, or `none` (to avoid creating the cypress/playwright sidecar containers, e.g. if you only want to launch a server instance) - * `ENABLED_DOCKER_SERVICES`: a space-separated list of services to start alongside the server. Default to `postgres inbucket`, for smoke tests purposes and for lightweight and faster start up time. Depending on the test requirement being worked on, you may want to override as needed, as such: + * `ENABLED_DOCKER_SERVICES`: a space-separated list of services to start alongside the server. Default to `postgres inbucket`, for smoke test purposes and for lightweight and faster start-up time. Depending on the test requirement being worked on, you may want to override as needed, as such: - Cypress full tests require all services to be running: `postgres inbucket minio openldap elasticsearch keycloak`. - Cypress smoke tests require only the following: `postgres inbucket`. - - Playwright full tests require only the folliwing: `postgres inbucket`. - * The following variables, which will be passed over to the server container: `MM_LICENSE` (no enterprise features will be available if this is unset; required when `SERVER=cloud`), and the exploded `MM_ENV` (a comma-separated list of env var specifications) + - Playwright full tests require only the following: `postgres inbucket`. + * The following variables, will be passed over to the server container: `MM_LICENSE` (no enterprise features will be available if this is unset; required when `SERVER=cloud`), and the exploded `MM_ENV` (a comma-separated list of env var specifications) * The following variables, which will be passed over to the cypress container: `BRANCH`, `BUILD_ID`, `CI_BASE_URL`, `BROWSER`, `AUTOMATION_DASHBOARD_URL` and `AUTOMATION_DASHBOARD_TOKEN` - * The `SERVER_IMAGE` variable can also be set, if you want to select a custom mattermost-server image. If not specified, the value of the `SERVER_IMAGE_DEFAULT` variable defined in file `.ci/.e2erc` is used. - * The `TEST_FILTER` variable can also be set, to customize which tests you want cypress to run. If not specified, only the smoke tests will run. Please check the `e2e-tests/cypress/run_tests.js` file for details about its format. -2. (optional) `make start-dashboard && make generate-test-cycle`: start the automation-dashboard in the background, and initiate a test cycle on it, for the given `BUILD_ID` + * The `SERVER_IMAGE` variable can also be set if you want to select a custom mattermost-server image. If not specified, the value of the `SERVER_IMAGE_DEFAULT` variable defined in file `.ci/.e2erc` is used. + * The `TEST_FILTER` variable can also be set, to customize which tests you want Cypress to run. If not specified, only the smoke tests will run. Please check the `e2e-tests/cypress/run_tests.js` file for details about its format. +2. (optional) `make start-dashboard && make generate-test-cycle`: start the automation dashboard in the background, and initiate a test cycle on it, for the given `BUILD_ID` * NB: the `BUILD_ID` value should stay the same across the `make generate-test-cycle` command, and the subsequent `make` (see next step). If you need to initiate a new test cycle on the same dashboard, you'll need to change the `BUILD_ID` value and rerun both `make generate-test-cycle` and `make`. * Note that part of the dashboard functionality assumes the `BUILD_ID` to have a certain format (see [here](https://github.com/saturninoabril/automation-dashboard/blob/175891781bf1072c162c58c6ec0abfc5bcb3520e/lib/common_utils.ts#L3-L23) for details). This is not relevant for local running, but it's important to note in the testing pipelines. * This also automatically sets the `AUTOMATION_DASHBOARD_URL` and `AUTOMATION_DASHBOARD_TOKEN` variables for the cypress container * Note that if you run the dashboard locally, but also specify other `AUTOMATION_DASHBOARD_*` variables in your `.ci/env` file, the latter variables will take precedence. - * The dashboard is used for orchestrating specs with parallel test run and is typically used in CI. + * The dashboard is used for orchestrating specs with parallel test runs and is typically used in CI. * Only Cypress is currently using the dashboard; Playwright is not. 3. `make`: start and prepare the server, then run the Cypress smoke tests - * You can track the progress of the run in the `http://localhost:4000/cycles` dashboard, if you launched it locally + * You can track the progress of the run in the `http://localhost:4000/cycles` dashboard if you launched it locally * When running with `SERVER=cloud`, this will automatically create a cloud customer against the specified `CWS_URL` service, and delete that user after the run is complete. * If you want to run the Playwright tests instead of the Cypress ones, you can run `TEST=playwright make` * If you just want to run a local server instance, without any further testing, you can run `TEST=none make` - * If you're using the automation dashboard, you have the option of sharding the E2E test run: you can launch the `make` command in parallel on different machiness (NB: you must use the same `BUILD_ID` and `BRANCH` values that you used for `make generate-test-cycle`) to distribute running the test cases across them. When doing this, you should also set on each machine the `CI_BASE_URL` variable to a value that uniquely identifies the instance where `make` is running. + * If you're using the automation dashboard, you have the option of sharding the E2E test run: you can launch the `make` command in parallel on different machines (NB: you must use the same `BUILD_ID` and `BRANCH` values that you used for `make generate-test-cycle`) to distribute running the test cases across them. When doing this, you should also set on each machine the `CI_BASE_URL` variable to a value that uniquely identifies the instance where `make` is running. 4. `make stop`: tears down the server (and the dashboard, if running) - * This also implicitly runs `make clean`, which also remove any generated environment or docker-compose files. + * This also implicitly runs `make clean`, which also removes any generated environment or docker-compose files. Notes: - Setting a variable in `.ci/env` is functionally equivalent to exporting variables in your current shell's environment, before invoking the makefile. -- The `.ci/.env.*` files are auto generated by the pipeline scripts, and aren't meant to be modified manually. The only file you should edit to control the containers' environment is `.ci/env`, as specified in the instructions above. -- All of the variables in `.ci/env` must be set before the `make generate-server` command is run (or, if using the dashboard, before the `make generate-test-cycle` command). Modifying that file afterwards has no effect, because the containers' env files are generated in that step. +- The `.ci/.env.*` files are auto-generated by the pipeline scripts and aren't meant to be modified manually. The only file you should edit to control the containers' environment is `.ci/env`, as specified in the instructions above. +- All of the variables in `.ci/env` must be set before the `make generate-server` command is run (or, if using the dashboard, before the `make generate-test-cycle` command). Modifying that file afterward has no effect because the containers' env files are generated in that step. - If you restart the dashboard at any point, you must also restart the server containers, so that it picks up the new IP of the dashboard from the newly generated `.env.dashboard` file - If new variables need to be passed to any of the containers, here are the general principles to follow when deciding where to populate it: * If their value is fixed (e.g. a static server configuration), these may be simply added to the `docker_compose_generator.sh` file, to the appropriate container. - * If you need to introduce variables that you want to control from `.ci/env`: you need to update the scripts under the `.ci/` dir, and configure them to write the new variables' values over to the appropriate `.env.*` file. In particular, avoid defining variables that depend on other variables within the docker-compose override files: this is to ensure uniformity in their availability, and simplifies the question of what container has access to which variable considerably. + * If you need to introduce variables that you want to control from `.ci/env`: you need to update the scripts under the `.ci/` dir and configure them to write the new variables' values over to the appropriate `.env.*` file. In particular, avoid defining variables that depend on other variables within the docker-compose override files: this is to ensure uniformity in their availability and simplifies the question of what container has access to which variable considerably. * Exceptions are of course accepted wherever it makes sense (e.g. if you need to group variables based on some common functionality) - The `publish-report` Make target is meant for internal usage. Usage and variables are documented in the respective scripts.