Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v3.0.5]Allure-Report not marking tests as flaky (allure-playwright) #1199

Open
PatatN7 opened this issue Nov 28, 2024 · 0 comments
Open

[v3.0.5]Allure-Report not marking tests as flaky (allure-playwright) #1199

PatatN7 opened this issue Nov 28, 2024 · 0 comments

Comments

@PatatN7
Copy link

PatatN7 commented Nov 28, 2024

Describe the bug
When a tests failed within the last 5 test runs, the test is not marked as flaky. None of the conditions specified in the documentations mark a test as flaky (https://allurereport.org/docs/test-stability/)

To Reproduce
Steps to reproduce the behavior:

  1. Run a test and make sure it passes
  2. Generate allure report
  3. Clean out allure-results folder
  4. Run a test and make sure it fails
  5. Copy history folder from allure-report to allure-results
  6. Generate allure report
  7. Clean out allure-results folder
  8. Run a test and make sure it passes
  9. Copy history folder from allure-report to allure-results
  10. Generate allure report
  11. Repeat steps 7-10 until you have a total of 5 test runs

Expected behavior
Expected the test that failed in the last 5 test runs to be marked as flaky.

Screenshots
This is what my trend looks like where multiple tests failed within the last 5 test runs
image

But none of the tests were marked as flaky
image

Desktop :

  • OS: Windows 11
  • Browser: Chrome
  • Version: 131.0.6778.86

Additional context
The tests are also set to retry 3 times when failed

After the tests have completed, it does show in the CLI that there are flaky tests, but it is not shown in the allure-report.
image

My playwright configuration is also set to only allure-playwright as the reporter

Command used to run tests: npx playwright test JobLogin.spec.ts -g "Job login and logout"
Command used to generate reports: ** allure generate --clean**

@PatatN7 PatatN7 changed the title Allure-Report not marking tests as flaky (allure-playwright) [v3.0.5]Allure-Report not marking tests as flaky (allure-playwright) Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant