Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting new tests to run on CI needs to be simplified #1

Open
michaeltryby opened this issue Sep 10, 2019 · 0 comments
Open

Getting new tests to run on CI needs to be simplified #1

michaeltryby opened this issue Sep 10, 2019 · 0 comments

Comments

@michaeltryby
Copy link
Collaborator

michaeltryby commented Sep 10, 2019

Running tests and updating the rolling benchmark have been simplified. Adding tests to the test repo is straightforward. Getting new tests to be compared with the SUT out on the CI server, however, is difficult. The way that nrtest is written only matching tests will be compared between two benchmarks with differing tests. So for new tests to run the benchmark archives need to be updated with the new results, otherwise, comparison won't occur.

Option 1) is to build the SUT and generate the benchmarks locally. Then update the latest release with the new benchmark archives. The downside is that this can fail if there are differences between the local and CI build environments.

Option 2) is to get the benchmark executable from the CI and run it locally to generate the new results and update the benchmark archive. Appveyor CI has been configured to save a zip archive of the epanet-solver install package as a build artifact for this purpose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant