Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): persist acceptance tests #5090

Merged
merged 1 commit into from
Mar 11, 2024

Conversation

thisislawatts
Copy link
Member

@thisislawatts thisislawatts commented Mar 5, 2024

Pull Request Submission

Please check the boxes once done.

The pull request must:

  • Reviewer Documentation
    • follow CONTRIBUTING rules
    • be accompanied by a detailed description of the changes
    • contain a risk assessment of the change (Low | Medium | High) with regards to breaking existing functionality. A change e.g. of an underlying language plugin can completely break the functionality for that language, but appearing as only a version change in the dependencies.
    • highlight breaking API if applicable
    • contain a link to the automatic tests that cover the updated functionality.
    • contain testing instructions in case that the reviewer wants to manual verify as well, to add to the manual testing done by the author.
    • link to the link to the PR for the User-facing documentation
  • User facing Documentation
    • update any relevant documentation in gitbook by submitting a gitbook PR, and including the PR link here
    • ensure that the message of the final single commit is descriptive and prefixed with either feat: or fix: , others might be used in rare occasions as well, if there is no need to document the changes in the release notes. The changes or fixes should be described in detail in the commit message for the changelog & release notes.
  • Testing
    • Changes, removals and additions to functionality must be covered by acceptance / integration tests or smoke tests - either already existing ones, or new ones, created by the author of the PR.

What does this PR do?

Continuing the work introduced through #5087, adopts junit reporter for acceptance test suites so we can collect information and gain a better understanding of how these perform. Once we have this data we can make a more informed pitch on where to spend our time to reduce CircleCI duration.

It is worth noting that there is some delay between test reports being uploaded and being represented in the CircleCI insights tool.

Where should the reviewer start?

Validate that test reports are being uploaded, heres a direct link through to the results for acceptance-tests linux arm64
https://app.circleci.com/pipelines/github/snyk/cli/21083/workflows/be568491-f0a0-45f0-8611-9576ea668b20/jobs/314068/tests

Each of the acceptance test jobs in this run successfully exited after uploading their test report
https://app.circleci.com/pipelines/github/snyk/cli/21083/workflows/be568491-f0a0-45f0-8611-9576ea668b20

How should this be manually tested?

You could deliberately introduce a test failure and verify that it is highlighted under the Tests tab in CircleCI UI.

@thisislawatts thisislawatts force-pushed the chore/ci-persist-acceptance-tests branch 12 times, most recently from 0bcaaff to 856d847 Compare March 8, 2024 12:32
@@ -983,13 +983,19 @@ jobs:
command: npm install
- run:
name: Running acceptance tests
no_output_timeout: 30m
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed as switching to junit reporter means there is no output generated, as our test suite currently takes >10 mins to run it will be terminated by CircleCI.

@thisislawatts thisislawatts marked this pull request as ready for review March 8, 2024 13:56
@thisislawatts thisislawatts requested a review from a team as a code owner March 8, 2024 13:56
@thisislawatts thisislawatts force-pushed the chore/ci-persist-acceptance-tests branch 3 times, most recently from f3a0ce0 to c42fc01 Compare March 11, 2024 08:39
@thisislawatts thisislawatts enabled auto-merge (squash) March 11, 2024 08:39
@thisislawatts thisislawatts force-pushed the chore/ci-persist-acceptance-tests branch from c42fc01 to 0cb1816 Compare March 11, 2024 08:43
@thisislawatts thisislawatts merged commit af382ea into main Mar 11, 2024
14 checks passed
@thisislawatts thisislawatts deleted the chore/ci-persist-acceptance-tests branch March 11, 2024 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants