Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CI workflows for E2E tests #13722

Closed

Conversation

lukaszcl
Copy link
Collaborator

@lukaszcl lukaszcl commented Jun 28, 2024

This PR introduces a Reusable GitHub Workflow for E2E tests, centralizing all the necessary logic to run various types of tests—such as Docker and K8s remote runner—within a single workflow. This approach allows other E2E test workflows to utilize this central workflow at different stages of the software development process, whether it's for every commit, within the merge queue, nightly, or on-demand.

The workflow sources the tests to run from the .github/e2e-tests.yml file, where each test or group of tests is defined with specific parameters: test-cmd to dictate how to execute the test, runs-on to specify the required GitHub runner, and a workflows parameter that outlines when each test should be executed (e.g., on every pull request or nightly).

  # Example of 1 runner for all tests in integration-tests/smoke/ocr_test.go
  - id: smoke/ocr_test.go:* 
    path: integration-tests/smoke/ocr_test.go
    test-type: docker
    test-inputs:
       chainlinkImage: '{{ env.QA_CHAINLINK_IMAGE }}'
       .. # other test inputs if required by a test
    runs-on: ubuntu-latest
    workflows:
      - Run PR E2E Tests
      - Run Nightly E2E Tests
    test-cmd: cd integration-tests/ && go test smoke/ocr_test.go -timeout 30m -count=1 -json
    pyroscope-env: ci-smoke-ocr-evm-simulated-nightly

The run-e2e-tests-reusable-workflow.yml workflow can be utilized in various ways:

  • Specify Test IDs: Provide comma-separated test IDs in the test_ids input to run specific tests. For example, see the run-selected-e2e-tests.yml workflow.
  • Run Tests by Workflow: Use the test_workflow input to run all tests associated with a particular workflow. For example, see the run-nightly-e2e-tests.yml workflow.
  • Custom Test List: Input a list of tests with custom settings using the test_list input. For example, see the run-automation-ondemand-e2e-tests.yml workflow.

To learn more about new CI Workflows for E2E Tests, please visit https://smartcontract-it.atlassian.net/wiki/spaces/TT/pages/815497220/CI+Workflows+for+E2E+Tests

Note: This update does not affect existing workflows for now. Changes to integrate existing workflows with this new setup will be addressed in separate PRs. More details are provided below.


Full scope of this PR:

  1. Add reusable workflow to run E2E tests: run-e2e-tests-reusable-workflow.yml
  2. Add a workflow to run any E2E test from .github/e2e-tests.yml on demand: run-selected-e2e-tests.yml
    gh workflow run run-selected-e2e-tests.yml --ref ${branch_name} -f chainlink_version=develop -f test_ids="smoke/ocr_test.go:*,soak/ocr_test.go:^TestOCRv1Soak$"
    
  3. Add a workflow to run tests nightly: run-nightly-e2e-tests.yml. Tests to run nightly have "Run Nightly E2E Tests" in workflows field in .github/e2e-tests.yml file. Example build.
  4. Add run-automation-ondemand-e2e-tests.yml that works the same as automation-ondemand-tests.yml but uses the reusable workflow. Once it's tested we can remove the old workflow. run-automation-ondemand-e2e-tests.yml is an example how to run tests with custom test inputs
  5. Add go tool to detect dead-tests (tests that are not part of any CI workflow). Learn more here. This is disabled by default for now
  6. Add go tool to create default test config override (./.github/actions/create-default-e2e-config-override) and use it instead of the old bash script
  7. Add jobs to reusable workflow to easily build required Chainlink Node image and Chainlink Plugins image. Required versions can be set using require_chainlink_image_versions_in_qa_ecr and require_chainlink_plugin_versions_in_qa_ecr workflow inputs

Not in the scope:

  1. Refactoring existing workflows to utilise the new Reusable E2E Tests Workflow will be addressed in separate PRs
  2. Allow to pass full test_config_base64_override as workflow input. This will be enabled once TestConfig does not have any secrets. Related ticket https://smartcontract-it.atlassian.net/browse/TT-1283
  3. Add Slack notifications for workflows that use the E2E tests reusable workflow. This will be addressed in

@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 602d740 to 0fbedb7 Compare July 1, 2024 11:12
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 0fbedb7 to fe07752 Compare July 1, 2024 11:16
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from fe07752 to d6fe746 Compare July 1, 2024 11:18
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from d6fe746 to 6fbd4f5 Compare July 1, 2024 11:22
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 6fbd4f5 to 3550114 Compare July 1, 2024 11:25
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 3550114 to afad909 Compare July 1, 2024 11:41
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from afad909 to 66531cd Compare July 1, 2024 11:55
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 66531cd to 8108ec6 Compare July 1, 2024 11:56
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 8108ec6 to 3dd4e8b Compare July 1, 2024 12:00
@lukaszcl lukaszcl force-pushed the TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core branch from 3dd4e8b to 2d0ae2b Compare July 1, 2024 12:01
@lukaszcl lukaszcl closed this Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant