test: separate unit and integration tests

Since integration tests are more expensive, we separated them to run
only after unit tests have succeeded. We also don't need to repeat them
for every python version, so in CI they will only run in Python 3.12.

.changes/unreleased/Test-20240621-171539.yaml

@@ -0,0 +1,3 @@
+kind: Test
+body: Separate tests into unit and integration tests
+time: 2024-06-21T17:15:39.021747+02:00

.github/workflows/code-quality.yaml

@@ -19,8 +19,11 @@ jobs:
       - name: fetch server schema
         run: "hatch run dev:fetch-schema"
 
-      - name: tests
-        run: "hatch run test:run"
+      - name: unit tests
+        run: "hatch run test:unit"
+
+      - name: integration tests
+        run: "hatch run test.py3.12:integration"
         env:
           SL_HOST: ${{ secrets.TEST_HOST }}
           SL_ENV_ID: ${{ secrets.TEST_ENV_ID }}

CONTRIBUTING.md

@@ -43,9 +43,13 @@ We use [changie](https://changie.dev/) to manage our [`CHANGELOG.md`](./CHANGELO
 
 ### Running tests
 
-Run tests by using `hatch run test:run`. This will run all our tests in all supported Python versions. Make sure they pass before you submit a PR.
+Run tests by using `hatch run test:all`. This will run all our tests in all supported Python versions. Make sure they pass before you submit a PR.
 
-The test suite requires a `tests/server_schema.gql` file to exist, since it checks that all GraphQL queries from the SDK will work against the server. That file must contain the schema of the GraphQL API, and you can obtain it by running our introspection script with `hatch run dev:fetch-schema`.
+If you want to run unit test separately, run `hatch run test:unit`. For integration tests, run `hatch run test.py3.12:integration` to avoid running slow tests in all python versions.
+
+The unit test suite requires a `tests/server_schema.gql` file to exist, since it checks that all GraphQL queries from the SDK will work against the server. That file must contain the schema of the GraphQL API, and you can obtain it by running our introspection script with `hatch run dev:fetch-schema`.
+
+The integration test suite requires an actual Semantic Layer account. Make sure you have `SL_HOST`, `SL_TOKEN` and `SL_ENV_ID` set as environment variables before running.
 
 
 ### Committing changes

pyproject.toml

@@ -72,7 +72,10 @@ features = [
   "dev-test"
 ]
 [tool.hatch.envs.test.scripts]
-run = "pytest --server-schema tests/server_schema.gql"
+all = "pytest --server-schema tests/server_schema.gql"
+unit = "pytest --ignore tests/integration/ --server-schema tests/server_schema.gql"
+integration = "pytest tests/integration/ --server-schema tests/server_schema.gql"
+
 [[tool.hatch.envs.test.matrix]]
 python = ["3.9", "3.10", "3.11", "3.12"]