ci: serialise integration tests and quarantine problematic tests #1149
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Integration tests are incredibly flakey currently. Perhaps this is related to the parallel execution of the tests. Certainly the fact that the tests are randomly distributed across threads doesn't help with figuring out what works and what doesn't.
dimaqq
approved these changes
Oct 9, 2024
I can see the value in that. My thinking is that this separate job is not really intended as a permanent quarantine zone, but rather we'll eventually fix the model and cross model tests, and then rename the job to 'Integration - model' or something. We'd keep the separate job for easier debugging and faster test runtime. I think 'Quarantined Integration Tests' works as the name for now since contributors will understand when it fails. A you say, if we go in the direction of wanting to quarantine individual tests in the future we could add a pytest marker in the future. |
This was referenced Oct 9, 2024
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tests in
integration/test_model.pyseem to be flaky even when run serially. All tests inintegration/test_crossmodel.pyare currently skipped, except one which used to be skipped, and is currently flaky even when run serially.This PR:
test_model.pythat seem to always fail currently, whether run in serial or in parallel, following ci: serial integration tests with those that always fail disabled #1145As a bonus feature, this split of the tests into two runners with
-n 1seems to be faster than the original method of running all the integration tests in a single runner with-n auto(which worked out to be 4 processes on github).