switch to pytest for testing (#164)

.circleci/config.yml

@@ -1,7 +1,7 @@
 version: 2.1
 
 orbs:
-  collection-testing: maxhoesel-ansible/ansible-collection-testing@0.4.0
+  collection-testing: maxhoesel-ansible/ansible-collection-testing@0.5.3
 
 filters: &semver-tagged
   tags:
@@ -10,28 +10,44 @@ filters: &semver-tagged
   branches:
     ignore: /.*/
 
+executors:
+  pytest:
+    machine:
+      image: ubuntu-2204:current
+    resource_class: large
+
+jobs:
+  test:
+    parameters:
+      parallelism:
+        description: Number of parallel runners
+        type: integer
+      ansible-version:
+        description: Version of Ansible to use for testing
+        type: string
+      node-python-version:
+        description: Version of python to use for module tests
+        type: string
+    executor: pytest
+    parallelism: << parameters.parallelism >>
+    steps:
+      - collection-testing/pytest:
+          pytest-args: >
+            --ansible-version << parameters.ansible-version >>
+            --node-python-version << parameters.node-python-version >>
+
 workflows:
   ci:
     jobs:
+      - test:
+          name: Test (ansible-<< matrix.ansible-version >>)
+          parallelism: 3
+          matrix:
+            parameters:
+              ansible-version: ["2.15", "2.14"]
+              node-python-version: ["3.6"]
       - collection-testing/pre-commit-lint:
           name: Lint
-      #- collection-testing/antsibull-docs:
-      #    name: Generate Docs
-      #- collection-testing/run-tox-environments:
-      #    name: Test Modules
-      #    match-environments: \-test\-
-      #    parallelism: 3
-      #    resource-class: medium
-      #    retries: 1
-      #    retry-delay: 60
-      - collection-testing/run-tox-environments:
-          name: Test Roles
-          match-environments: roles
-          # number of scenarios * ansible versions to test
-          parallelism: 12
-          resource-class: large
-          retries: 1
-          retry-delay: 60
       - collection-testing/publish-github:
           name: Publish Release to GitHub
           context: collection-publishing

.config/molecule/config.yml

@@ -3,17 +3,10 @@ dependency:
   name: galaxy
 
 driver:
-  name: podman
+  name: docker
 
 provisioner:
   name: ansible
-  env: {}
-    # Cannot enable pipelining for podman at this time:
-    # https://github.com/ansible-community/molecule-podman/issues/2
-    #ANSIBLE_PIPELINING: false
-  inventory:
-    group_vars:
-      all: {}
 
 scenario:
   test_sequence:

.gitignore

@@ -145,5 +145,12 @@ dmypy.json
 
 # Ansible collection archives
 *.tar.gz
-# Adding this to gitignore breaks ansible-test. so let's not do that
-#ansible_collections/
+docs/ansible_collections/
+
+# Don't commit ephermal integration config
+tests/integration/integration_config.yml
+collections
+
+# CI files
+results
+split-tests

CONTRIBUTING.md

@@ -9,9 +9,8 @@ Note that by contributing to this collection, you agree with the code of conduct
 
 Prerequisites:
 
-- A recent version of Python supported by `ansible-core` (see [here](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#control-node-requirements))
-- For role tests: `podman` 4 or newer set up as shown [below](#setting-up-podman) (note that Docker will *not* work for role tests!)
-- For plugin tests (TBD): A recent version of Docker
+- A recent version of Python supported by the current release of `ansible-core` (see [here](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#control-node-requirements))
+- Docker (for running Tests)
 
 Steps:
 
@@ -27,7 +26,7 @@ Steps:
 ### Plugins (and Modules)
 
 There are currently no plugins in this collection.
-In the future, we may look into creating plugins for accessing the Pterodactyl API.
+In the future, we may add modules to wrap around the pterodactyl API.
 
 ### Roles
 
@@ -41,74 +40,53 @@ Some general guidelines:
 - Keep the configuration for the user simple and try to provide sensible defaults where possible
 - Try to avoid using complex data structures as role variables/parameters, use simple values that can be composed easily instead.
 - Make sure to document any role variables in both the `README.md` and in the `meta/argument_specs.yml` file.
-  Te latter is used to generate role documentation programmatically.
+  The latter is used to generate role documentation programmatically.
 
 ## Testing Changes
 
-We aim to test all of the components in this collection as thoroughly as possible.
-We currently test all components using the following testing matrix:
+We aim to test every part of this collection as thoroughly as reasonable to ensure correct behavior.
+We use `pytest` to run all of our tests, both for plugins and roles.
+If you set up the test environment as described in [the Getting Started guide](#getting-started), you should be able to see all available tests:
 
-- Ansible version: The three most recent major releases (such as 6,7,8), with a compatible host Python
-- Node Python: The minimum supported Python version (see `tox.ini`, used for plugi tests)
-- Pterodactyl Version: The most recently released version of Panel+Wings
-- For each entry in this matrix, we test all roles on all of their supported platforms, as well as all modules/plugins
+`pytest --co`
 
-To make testing all these permutations easier, we use `tox`to manage test scenarios.
-If you set up the test environment as described in [the Getting Started guide](#getting-started), you should be able to run `tox -l` from the collection root:
+You can run these tests using `pytest` and limit execution to specific test with `pytest -k 'test_pattern'` (or just use your editors testing plugin).
+Please note that running the full test suite executes all molecule scenarios and may take **up to an hour** to complete.
 
-This should print a list of test environments (scenarios) that you can run.
-To run a specific environment, just use `tox -e {environment}`.
-You can also use substitution syntax to run multiple environments at once:
+### Testing different App Versions
 
-```bash
-# Will run the py3-ansible7-roles-pterodactyl_panel_selfsign environment
-tox -e py3-ansible7-roles-pterodactyl_panel_selfsign
+When you run the collection tests using `ptytest`, they are executed with the current stable Ansible version in `requirements.txt`.
+To ensure that this collection remains backwards-compatible, we also test against older versions of Ansible.
+Our testing Matrix currently looks like this:
 
-# Will run XXX-roles-pterodactyl_panel_selfsign environment with ansible versions 6,7 and 8
-# Note the single quotes around the string!
-tox -e 'py3-ansible{6,7,8}-roles-pterodactyl_panel_selfsign'
-```
-
-### Plugins (and Modules)
-
-TBD
-
-### Roles
-
-We use the `tox-ansible` plugin (v1) to integrate molecule scenarios into tox.
-You can run these scenarios using `tox -e`, just like for module tests.
-
-Molecule itself runs the subject role against several containers to verify its functionality across target systems.
-Since some roles involve the management of systemd services, we need a container runtime with good systemd support,
-something which `docker` sadly doesn't offer on [modern linux distros](https://gist.github.com/pinkeen/bba0a6790fec96d6c8de84bd824ad933).
+| Component | Module Tests | Role Tests | Versions |
+|-----------|--------------|------------|----------|
+| `ansible-core` | ✅ | ✅ | Three most recent releases (e.g. `2.13`, `2.14`, `2.15`) |
+| Node Python Version | ✅ | ❌ | Collection-supported Python version (see [README](./README.md))
 
-Instead, we use `podman`, a daemonless, rootless container runtime developed by RedHat to be (mostly) compatible to docker, but with better support for certain features such as systemd.
-See below for setup instructions.
-Once podman is installed and running, you can use `tox` to run the molecule scenarios.
+All possible permutations are automatically tested in CI.
+You can change the tested versions locally by supplying additional arguments to `pytest`:
 
-#### Setting up Podman
-
-1. Ensure that you have the following packages installed:
-    - `podman` version 4+ (as it comes with the new netvark networking stack)
-    - `aardvark-dns`, a plugin for netvark which provides DNS between containers in the same network
-    - **NOTE:** If you previously used an older (`<=3.x`) version of `podman`, you will have to migrate to the new networking stack fist. This can be done with `podman system reset`
-2. Ensure that your user has a subuid/subguid configuration associated with them so that you can run rootless containers
-    - Check the `/etc/subuid` and `/etc/subgid` files for entries corresponding to your username.
-    - If there is nothing, you can add them like so: `sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 <USERNAME>` (make sure that the range is not  already taken by another user in `/etc/subuid`/`/etc/subgid`).
-3. Once you have applied your changes, run `podman system migrate` to force `podman` to pick up the new configuration.
-
-That's it! Podman should now be working! To test it, you can run a container just like with docker: `podman run --rm -it ubuntu bash`
+```
+$ pytest --help
+# truncated output
+Custom options:
+  --ansible-version=ANSIBLE_VERSION
+                        Version of ansible to use for tests, in the format '2.xx'. Default: see requirements.txt
+  --node-python-version=NODE_PYTHON_VERSION
+                        Python version to test Ansible modules with, in the format '3.x'. Default: '3.6'
+```
 
 ## Writing Tests
 
 Any new new component or change to an existing one should be covered by tests to ensure that the code works, and that it keeps working into the future.
 This section will help you in adding your own tests to this collection.
 
-#### Plugins
+### Plugins
 
 TBD
 
-#### Roles
+### Roles
 
 There are tons of good guides online for how to write tests using molecule.
 Alternatively, you can always look at the existing molecule scenarios in this collection
@@ -124,18 +102,14 @@ some_role/
       converge.yml
       molecule.yml
       prepare.yml
-      requirements.txt # --> symlink to /tests/roles/requirements.txt
       verify.yml
     another-scenario/
       ...
   tasks
   ...
 ```
 
-The `requirements.txt` symlink is used by `tox-ansible` when running tests via `tox` to install a specific, known-good version of `molecule` and the `molecule-podman` driver.
-That way, all roles and scenarios in this collection can use the same version of `molecule`.
-
-The [root molecule config](./.config/molecule/config.yml) contains the basic settings for molecule, such as driver setup and the step utility versions.
+The [root molecule config](./.config/molecule/config.yml) contains the basic settings for molecule, such as driver setup.
 In addition, your roles molecule scenario must define a set of platforms to test on, as well as any inventory configuration that you may need.
 To get started you can copy the `molecule.yml` configuration from an existing role, then adjust it to suit your needs.
 
@@ -145,17 +119,19 @@ TBD
 
 ## Maintainer information
 
-### Updating Tested Versions
+### Bumping supported ansible-core versions
+
+1. Update the versions in the [CI config](./.circleci/config.yml)
+
+### Bumping node python version
 
-- Pterodactyl and ansible-core (for plugins): Add the new core version to the following places in `tox.ini`:
-    - The `envlist` in `tox.ini`
-    - each `testenv:xxx` section header that deals with plugin tests
-    - Set the correct environment variable in the main `testenv` section
-- ansible (for roles): Change the version string in the `ansible` section in `tox.ini`
+1. Update the version in [`tests/conftest.py`](./tests/conftest.py)
+2. Update the version in the [CI config](./.circleci/config.yml)
 
 ### Versioning and Releases
 
+- Releases are automatically drafted by `release-drafter`, with a changelog generated from PR labels
 - When merging a pull request, make sure to select an appropriate label (pr-bugfix, pr-feature, etc.).
-  Release-drafter will automatically update the draft release changelog and the galaxy.yml version will be bumped if needed.
-- Once a draft release is actually published, collection packages will be added to the release and ansible-galaxy automatically.
-- If you need to manually bump the collection version, run the `update-version` script and adjust the test versions if required.
+  Release-drafter will automatically update the draft release changelog and a PR will be opened with bumped collection versions.
+- Once a draft release is actually published, collection packages will be published to the release and ansible-galaxy automatically.
+- If you need to manually bump the collection version, run the `update-version` script

README.md

@@ -15,7 +15,7 @@ for v1.0 becomes available.
 
 ### Dependencies
 
-- A recent version of ansible. We test against the current and the 3 previous major releases
+- A recent version of ansible. We test against the current and the previous major release
 - Python 3.6 or newer on remote hosts and the controller
 
 ### Install via ansible-galaxy

galaxy.yml

@@ -18,7 +18,6 @@ build_ignore:
 - .readthedocs.yaml
 - pyproject.toml
 - requirements.txt
-- tox.ini
 - '**/requirements.txt'
 dependencies:
   community.crypto: '>=1.0.0'

pyproject.toml

@@ -4,20 +4,23 @@ ignore = "E402"
 
 # Pylint settings
 [tool.pylint.'NESSAGES CONTROL']
-disable=[
+disable = [
     "logging-fstring-interpolation", # See https://github.com/PyCQA/pylint/issues/2354
     "missing-module-docstring",
     "missing-class-docstring",
-    "missing-function-docstring", # also triggers on abstract functions
-    "consider-using-enumerate", # not for simple loops
+    "missing-function-docstring",    # also triggers on abstract functions
+    "consider-using-enumerate",      # not for simple loops
     "consider-using-dict-items",
     "no-else-return",
     "invalid-name",
     "too-many-boolean-expressions",
     "too-few-public-methods",
-    'wrong-import-position', # Ansible module best practices
+    'wrong-import-position',         # Ansible module best practices
     "I",
 ]
 
 [tool.pylint.'FORMAT']
-max-line-length=120
+max-line-length = 120
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

requirements.txt

@@ -2,19 +2,20 @@
 # Includes utilities, CLI helpers and so on
 
 # Linting & Formatting
-ansible-lint==6.21.1
-pylint==3.0.2
+ansible-lint==6.20.3
+pylint==3.0.1
 autopep8==2.0.4
-pre-commit==3.5.0
+pre-commit==3.4.0
 
-# Test scenario runner
-tox==3.28.0
+# Testing libraries
+pytest==7.4.2
+pytest-virtualenv==1.7.0
+docker==6.1.3
 
 # Utility packages used in scripts
 pyyaml==6.0.1
 packaging==23.2
 
-# Also include a version of ansible-core for IDE hints and such.
-# Note that this version is independent of the one used in tests, see tox.ini
-# and /tests
+# Also include a version of ansible-core for IDE hints
+# and as the default version used in tests
 ansible-core==2.15.5

roles/pterodactyl_panel/README.md

@@ -89,7 +89,6 @@ Other versions are supported on a best-effort basis.
 ### Panel Settings
 
 Prefix for all variables: `pterodactyl_panel_`
-[Title](../../../ansible-collection-smallstep/.config)
 | Name | Description | Required | Default |
 |------|-------------|:--------:|---------|
 | `app_key` | base64 encoded app key for the panel, as generated by `php artisan key:generate`. You can generate a key with this command: `echo "base64:$(openssl rand -base64 32)"` | X | undefined |