Merge pull request #1149 from james-garner-canonical/24.10/ci/quarant…

…ine-and-serialise

#1149

Tests in `integration/test_model.py` seem to be flaky even when run serially. All tests in `integration/test_crossmodel.py` are currently skipped, except one which used to be skipped, and is currently flaky even when run serially.

This PR:
* Serialises all integration tests following #1143
* Skips two tests from `test_model.py` that seem to always fail currently, whether run in serial or in parallel, following #1145
* Moves the flaky tests noted above into a separate job, so that the job running the remaining integration tests will hopefully have a shot at succeeding

As a bonus feature, this split of the tests into two runners with `-n 1` seems to be faster than the original method of running all the integration tests in a single runner with `-n auto` (which worked out to be 4 processes on github).

.github/workflows/test.yaml

@@ -121,3 +121,29 @@ jobs:
       - name: Run integration
         # Force one single concurrent test
         run: tox -e integration
+
+  integration-quarantine:
+    name: Quarantined Integration Tests
+    needs: [lint, unit-tests]
+    timeout-minutes: 150
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python:
+          - "3.10"
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v3
+      - name: Setup operator environment
+        uses: charmed-kubernetes/actions-operator@main
+        with:
+          provider: lxd
+          juju-channel: 3.4/stable
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Install dependencies
+        run: pip install tox
+      - name: Run integration
+        run: tox -e integration-quarantine

tests/integration/test_model.py

@@ -309,6 +309,7 @@ async def test_deploy_local_bundle_with_overlay_multi():
 
 @base.bootstrapped
 @pytest.mark.bundle
+@pytest.mark.skip("Always fails -- investigate bundle charms")
 async def test_deploy_bundle_with_overlay_as_argument():
     async with base.CleanModel() as model:
         overlay_path = OVERLAYS_DIR / 'test-overlay.yaml'
@@ -342,6 +343,7 @@ async def test_deploy_bundle_with_multi_overlay_as_argument():
 
 @base.bootstrapped
 @pytest.mark.bundle
+@pytest.mark.skip("Always fails -- investigate bundle charms")
 async def test_deploy_bundle_with_multiple_overlays_with_include_files():
     async with base.CleanModel() as model:
         bundle_yaml_path = TESTS_DIR / 'integration' / 'bundle' / 'bundle.yaml'

tox.ini

@@ -63,7 +63,27 @@ envdir = {toxworkdir}/py3
 commands =
     pip install urllib3<2
     pip install pylxd
-    python -m pytest --tb native -ra -v -n auto -k 'integration' -m 'not serial' {posargs}
+    python -m pytest \
+        --tb native \
+        -ra -v -n 1 \
+        -k 'integration' \
+        --ignore {toxinidir}/tests/integration/test_crossmodel.py \
+        --ignore {toxinidir}/tests/integration/test_model.py \
+        -m 'not serial' \
+        {posargs}
+
+[testenv:integration-quarantine]
+envdir = {toxworkdir}/py3
+commands =
+    pip install urllib3<2
+    pip install pylxd
+    python -m pytest \
+        --tb native \
+        -ra -v -n 1 \
+        -m 'not serial' \
+        {posargs} \
+        {toxinidir}/tests/integration/test_crossmodel.py \
+        {toxinidir}/tests/integration/test_model.py
 
 [testenv:unit]
 envdir = {toxworkdir}/py3
@@ -74,6 +94,8 @@ commands =
 
 [testenv:serial]
 # tests that can't be run in parallel
+# there's one test marked with 'serial'
+# it doesn't get run in CI
 envdir = {toxworkdir}/py3
 commands =
     pip install urllib3<2