Skip to content

Latest commit

 

History

History
105 lines (74 loc) · 6.7 KB

README.md

File metadata and controls

105 lines (74 loc) · 6.7 KB
Raw

Overview

This is the repository for managing Gen3 integration tests. The code is written in Python, and the following tools/frameworks are used:

  • poetry (package management docs)
  • pytest (testing framework docs)
  • requests (tool for making HTTP requests docs)
  • playwright (tool for automating web applications docs)
  • gen3sdk-python (SDK for handling common Gen3 tasks docs)
  • xdist (parallel test execution docs)
  • allure (tool for visualizing test results docs)

Running tests

Setup

Set up prerequisites

Checkout and switch directory

Checkout this repo and switch to gen3-integration-tests directory. This is the root directory for integration tests.

Create ~/.gen3 directory

The integration tests look for API keys in this location. Make sure you created this directory.

Create .env file

Switch to gen3-integration-tests directory and create a .env file. The code is designed to fetch environment variables set in this file.

Create output and install dependencies

The output directory is used to store the markdown report that is also generated along with the allure report.

Switch to gen3-code-vigil/gen3-integration-tests and run the commands:

mkdir output
poetry install

Set environment variable GEN3_INSTANCE_TYPE

Gen3 instances can be run on an admin VM using cloud-automation or using helm, and the mechanisms for running admin tasks vary based on the instance type.

Since some integration tests run admin tasks, we must set this variable for executing tests correctly in the .env file. The following values are accepted:

  • ADMINVM_REMOTE (for instances hosted on a remote admin VM using cloud-automation)
  • HELM_LOCAL (for instances hosted locally using helm)

Set up test users

The code supports running test steps as different users. This code can provide insights into the set up process.

The test users required to run the tests are listed here.

The API keys for these users must be saved to ~/.gen3 directory before running tests. Please find the instructions for each GEN3_INSTANCE_TYPE here

Set up test user permissions

User permissions required for the tests to pass are documented here. The tests attempt to run usersync before starting, so if usersync is correctly set up with this configuration there is nothing more to do. If that is not the case please make sure to run usersync or useryaml with this configuration before running the tests.

Set up test data

Guppy

We run guppy tests with fixed ES data to enable data validation consistently. Before running guppy tests we must ensure the indices are created with the required data. We can use one of the setup scripts located here depending on the type of Gen3 instance being tested.

Run tests and review results

Read these docs for specific information on how to run tests for each GEN3_INSTANCE_TYPE.

The report can be viewed by running allure serve allure-results

-n auto comes from python-xdist. auto distributes tests across all available CPUs. We can set to to a smaller value to use only some of the cores.

Test classes / suites run in parallel using the --dist loadscope. We implemented custom scheduling for grouping tests across test suites which is explained here

Markers and -m flag can be used to specify what tests should or should not run. For example, -m wip selects only tests with marker wip and -m not wip skips tests with marker wip.

-k flag can be used to run specific test suites (test suite name is the class name), e.g.

  • -k TestHomePage runs only the test_homepage.py
  • -k "TestHomePage or TestETL" runs test_homepage.py

Read more about marking tests here

Writing tests

Design principles

  • The test suites must be independent and idempotent. This is essential since we run test classes in parallel by using xdist (loadscope).
  • All tests should be able to run anywhere (locally / CI) without changing test code.
  • Debugging must be done locally, not in CI pipeline.
  • Documentation is essential. Code is incomplete without it.
  • Avoid hard waits. Test should wait for application state, not otherwise.
  • Tag tests appropriately using markers. Ensure that the markers are added here
  • Add test steps as docstrings in the test for understanding the purpose of the test easily.
  • Mark in-progress tests with marker wip to prevent CI pipeline from breaking when incomplete test code is pushed to repo.
  • Ensure that privileged information is not logged since the tests run in Github Actions and the logs are public.

Code structure

The test code is organized into several directories for ease of maintenance:

  • tests: Contains tests written in pytest
  • test_data: Contains test data needed for integration tests.
  • pages: Contains endpoint definitions, locators, and methods specific to each page in the portal, with a separate module for each page.
  • services: Contains endpoints and methods specific to each service, with a separate module for each service.
  • utils: Provides utility and helper functions used across tests.
  • scripts: Includes standalone helper scripts used for setting up the test environment.

conftest.py controls the test flow.

The integration tests perform Gen3 operations and admin tasks, e.g., etl, metadata-aggregate-sync as part of the test flow. The code for handling these is at utils/gen3_admin_tasks. Read this doc for more information.

Code used for running integration tests in CI at CTDS is at gen3-code-vigil/gen3-integration-tests/gen3_ci. Jenkins is used for setting up the test environments and interacting with them.

  • jenkins-jobs directory contains the groovy scripts used by the jenkins jobs that perform Gen3 admin tasks.
  • scripts directory contains python scripts used in the github actions workflow.

Tests are organized into test suites using classes as explained here.