"Any success" strategy for tests results

[austyh]

Hey Allure Team, I am from MarathonLabs.
Marathon preserves the history of test launches on mobile devices. In cases where a test is flaky, we can execute it simultaneously on multiple devices. It suffices for at least one of the tests to be successful, even if the others fail. The challenge lies in the fact that the Allure report necessitates the last test in time to be successful. Could you incorporate support for any success in some format?

[baev]

Currently, Allure works the way so you can keep allure-results folder between the test runs, and the Allure Report will only show the latest results per test. It's very useful for local development & debugging, and I would like to keep this feature for Allure Report users.

Of course, It would be nice to have some run-id property in results from the start and consider all the other results (with different run-id) as history, not retries. However, that way it adds additional complications for distributed testing environments (you'll have to share the same ID between multiple nodes/processes). I doubt we can do it for all the tools and frameworks we support.

The current implementation allows users to use distributed runs without an additional centralised state and then simply generate the report, specifying multiple allure-results (or even merge allure-results before the report generation.

If we simply change the way retries work (e.g. by providing additional options for commandline, such as allure generate --retries-policy=any), we may break the common pattern people use reports locally. Allure will produce different results for the reports with different retries policies (all green locally, but failed tests on CI and vice versa).

The other way to implement it is to store the information about the retries policy in the results itself. For me, that is the way to go — when you design a test you think how the results should work for you. Some frameworks, like, TestNG and JUnit 5, have a repeated tests feature — such tests run multiple times with the same set of parameters to detect flaky (or the opposite, to avoid flaky failures). For TestNg, you even can specify the minimum required success rate for test to pass.

This way, we need to somehow determine, are the results from the same run, or different ones. We can implement it by specifying some retry-batch-id or any other label, which will be provided by framework integration.

Then, we'll need the policy itself, something like retry-success-rate (any, >0, <=100, with default to 100). The policy will be applied only if retry-batch-id is specified and for results with the same retry-batch-id.

Although, all that sounds a bit complicated. And we may have some performance-related issues in Allure TestOps with the solution I described above.