You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running tests and updating the rolling benchmark have been simplified. Adding tests to the test repo is straightforward. Getting new tests to be compared with the SUT out on the CI server, however, is difficult. The way that nrtest is written only matching tests will be compared between two benchmarks with differing tests. So for new tests to run the benchmark archives need to be updated with the new results, otherwise, comparison won't occur.
Option 1) is to build the SUT and generate the benchmarks locally. Then update the latest release with the new benchmark archives. The downside is that this can fail if there are differences between the local and CI build environments.
Option 2) is to get the benchmark executable from the CI and run it locally to generate the new results and update the benchmark archive. Appveyor CI has been configured to save a zip archive of the epanet-solver install package as a build artifact for this purpose.
The text was updated successfully, but these errors were encountered:
Running tests and updating the rolling benchmark have been simplified. Adding tests to the test repo is straightforward. Getting new tests to be compared with the SUT out on the CI server, however, is difficult. The way that nrtest is written only matching tests will be compared between two benchmarks with differing tests. So for new tests to run the benchmark archives need to be updated with the new results, otherwise, comparison won't occur.
Option 1) is to build the SUT and generate the benchmarks locally. Then update the latest release with the new benchmark archives. The downside is that this can fail if there are differences between the local and CI build environments.
Option 2) is to get the benchmark executable from the CI and run it locally to generate the new results and update the benchmark archive. Appveyor CI has been configured to save a zip archive of the epanet-solver install package as a build artifact for this purpose.
The text was updated successfully, but these errors were encountered: