From cbd0ac6d5c079ef0b1553e4dafa6cdee9b095692 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Lesimple?= Date: Tue, 3 Oct 2023 12:26:06 +0000 Subject: [PATCH] doc: add documentation on how to write integration tests --- doc/sphinx/development/setup.rst | 18 +-- doc/sphinx/development/tests.rst | 191 ++++++++++++++++++++++++++----- doc/sphinx/index.rst | 5 +- 3 files changed, 178 insertions(+), 36 deletions(-) diff --git a/doc/sphinx/development/setup.rst b/doc/sphinx/development/setup.rst index b4fdb95d2..72cc4c10b 100644 --- a/doc/sphinx/development/setup.rst +++ b/doc/sphinx/development/setup.rst @@ -15,7 +15,7 @@ access the files, and potentially modify them (such as for ``perltidy``). The supported tools are as follows: -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1 Usage: ./docker/devenv/run-tool.sh COMMAND [OPTIONS] @@ -47,21 +47,21 @@ is automatically checked by ``perlcritic``, ``perltidy`` and ``shellcheck`` each If you previously cloned the repository with such a command: -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1 git clone https://github.com/ovh/the-bastion Then you can copy the provided :file:`pre-commit` script into your local :file:`.git` folder: -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1 cp contrib/git/pre-commit .git/hooks/pre-commit To verify that it works checkout a new test branch and add two dummy files like this: -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1-5 git checkout -B mybranch @@ -97,8 +97,8 @@ To verify that it works checkout a new test branch and add two dummy files like As you see, the checks are running before the commit is validated and abort it should any check fail. -Integration tests run -********************* +Running integration tests +************************* Using Docker ------------ @@ -110,7 +110,7 @@ and tests the return values against expected output. To test the current code, use the following script, which will run ``docker build`` and launch the tests: -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1 tests/functional/docker/docker_build_and_run_tests.sh @@ -120,7 +120,7 @@ You'll get a list of the supported targets by calling the command without argume For example, if you want to test it under Debian (which is a good default OS if you don't have any preference): -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1 tests/functional/docker/docker_build_and_run_tests.sh debian12 @@ -128,7 +128,7 @@ For example, if you want to test it under Debian (which is a good default OS if The full tests usually take 25 to 50 minutes to run, depending on your hardware specs. If you want to launch only a subset of the integration tests, you may specify it: -.. code-block:: shell +.. code-block:: none :emphasize-lines: 1 tests/functional/docker/docker_build_and_run_tests.sh debian12 --module=320-base.sh diff --git a/doc/sphinx/development/tests.rst b/doc/sphinx/development/tests.rst index db496062d..e1672128a 100644 --- a/doc/sphinx/development/tests.rst +++ b/doc/sphinx/development/tests.rst @@ -1,41 +1,180 @@ +Writing tests ============= -Running Tests -============= -Using Docker -============ +.. contents:: + +When modifying code, adding features or fixing bugs, you're expected to write one or more tests to ensure that +the feature your adding works correctly, or that the bug you've fixed doesn't come back. + +Integration tests modules live in the :file:`tests/functional/tests.d` folder. +You may either add a new file to test your feature, or modify an existing file. + +These modules are shell scripts, and are sourced by the main integration test engine. Having a look at one of +these modules will help you understand how they work, the :file:`tests/functional/tests.d/320-base.sh` is a good +example you might want to look at. + +Example +------- + +Here is a simple test taken from :file:`320-base.sh`: + +.. code-block:: none + :caption: a simple test + + success help2 $a0 --osh help + contain "OSH help" + json .error_code OK .command help .value null + +A complete reference of such commands can be found below, but let's explain this example in a few words: + +The command ``success`` implies that we're running a new test command, and that we expect it to work (we might +also want to test invalid commands and ensure they fail as they should). +The tester docker will connect to the target docker (that is running the bastion code) as a bastion user, and +run the ``--osh help`` command there. This is expected to exit with a code indicating success (0), +otherwise this test fails. + +The output of the command, once run on the bastion, should contain the text ``OSH help``, or the test will fail. + +In the JSON output (see :doc:`/using/api`) of this command, we expect to find the ``error_code`` field set to ``OK``, +the ``command`` field set to ``help``, and the ``value`` field set to ``null``, or the test will fail. + +Running just this test will yield the following output: + +.. code-block:: none + :caption: a simple test output + + 00m04 [--] *** [0010/0021] 320-base::help2 (timeout --foreground 30 ssh -F /tmp/bastiontest.pgoA5h/ssh_config -i /tmp/bastiontest.pgoA5h/account0key1file user.5000@bastion_debian10_target -p 22 -- --json-greppable --osh help) + 00m05 [--] [ OK ] RETURN VALUE (0) + 00m05 [--] [ OK ] MUST CONTAIN (OSH help) + 00m05 [--] [ OK ] JSON VALUE (.error_code => OK) [ ] + 00m05 [--] [ OK ] JSON VALUE (.command => help) [ ] + 00m05 [--] [ OK ] JSON VALUE (.value => null) [ ] + +As you can see, this simple test actually checked 5 things: the return value, whether the output text contained +a given string, and 3 fields of the JSON output. + +Reference +--------- + +These are functions that are defined by the integration test engine and should be used in the test modules. + +Launch a test +************* + +run ++++ + +.. admonition:: syntax + :class: cmdusage + + - run + +This function runs a new test named ````, which will execute ```` on the tester docker. +Usually ```` will connect to the target docker (running the bastion code) using one of the test accounts, +and run a command there. + +A few accounts are preconfigured: + +- The main account ("account 0"): this one is guaranteed to always exist at all times, and is a bastion admin. + There are a few variables that can be referenced to use this account: + + - ``$a0`` is the ssh command-line to connect to the remote bastion as this account + - ``$account0`` is the account name, to be used in parameters of ``--osh`` commands where needed + +- A few secondary accounts that are created, deleted, modified during the tests: + + - ``$a1``, ``$a2`` and ``$a3`` are the ssh command-lines to connect to the remote bastion as these accounts + - ``$account1``, ``$account2`` and ``$account3`` are the accounts names + +- Another special non-bastion-account command exists: + + - ``$r0`` is the required command-line to directly connect to the remote docker on which the bastion code is running, + as root, with a bash shell. Only use this to modify the remote bastion files, such as config files, between tests + +A few examples follow: + +.. code-block:: none + :caption: running a few test commands + + run test1 $a0 --osh info + run test2 $a0 --osh accountInfo --account $account1 + run test3 $a1 --osh accountDelete --account $account2 + +Note that the ``run`` function just runs the given command, but doesn't check whether it exited normally, you'll +need other functions to verify this, see below. + +success ++++++++ + +.. admonition:: syntax + :class: cmdusage + + - success + +This function is exactly the same as the ``run`` command above, except that it expects the given ```` to +return a valid error code (zero). Most of the time, you should be using this instead of ``run``, except if you're +expecting the command to fail, in which case you should use ``run`` + ``retvalshouldbe``, see below. + +plgfail ++++++++ + +.. admonition:: syntax + :class: cmdusage + + - plgfail + +This function is exactly the same as the ``run`` command above, except that it expects the given ```` to +return an error code of 100, which is the standard exit value when an osh command fails. + +This function is equivalent to using ``run`` followed by ``retvalshouldbe 100`` (see below). + +Verify a test validity +********************** + +retvalshouldbe +++++++++++++++ + +.. admonition:: syntax + :class: cmdusage + + - retvalshouldbe + +Verify that the return value of a test launched right before with the ``run`` function is ````. +You should use this if you expect the previous test to return a non-zero value. -Functional tests use ``Docker`` to spawn an environment matching a bastion install. +Note that the ``success`` function is equivalent to using ``run`` followed by ``retvalshouldbe 0``. -One of the docker instances will be used as client, which will connect to the other instance -which is used as the bastion server. +contain ++++++++ -The client instance sends commands to the server instance and tests the return values against expected output. +.. admonition:: syntax + :class: cmdusage -To test the current code, put it on a machine with docker installed, and use the following script, -which will run docker build and launch the tests: + - contain + - contain REGEX - ``tests/functional/docker/docker_build_and_run_tests.sh `` +This function verifies that the output of the test contains a given ````. If you need to use a regex +to match the output, you can use the ``contain REGEX`` construction, followed by the regex. -Where target is one of the supported OSes. Currently only Linux targets are supported. -You'll get a list of the supported targets by calling the command without argument. +nocontain ++++++++++ -Without Docker -============== +.. admonition:: syntax + :class: cmdusage -You can however still test the code against a BSD (or any other OS) without using Docker, -by spawning a server under the target OS, and installing the bastion on it. + - nocontain + - nocontain REGEX -Then, from another machine, run: +This function does the exact opposite of the ``contain`` function just above, and ensure that a given text +or regex is NOT present in the output. - ``test/functional/launch_tests_on_instance.sh [outdir]`` +json +++++ -Where ``IP`` and ``port`` are the information needed to connect to the remote server to test, -``remote_user_name`` is the name of the account created on the remote bastion to use for the tests, -and ``ssh_key_path`` is the private SSH key path used to connect to the account. -The ``outdir`` parameter is optional, if you want to keep the raw output of each test. +.. admonition:: syntax + :class: cmdusage -This script is also the script used by the Docker client instance, -so you're sure to get the proper results even without using Docker. + - json [ ...] -Please do **NOT** run any of those tests on a production bastion! +This function checks the JSON API output of the test, and validates that it contains the correct value for each +specified field. The ```` entries must be valid `jq` filters. diff --git a/doc/sphinx/index.rst b/doc/sphinx/index.rst index 4fe4b6b0b..43a3aafdd 100644 --- a/doc/sphinx/index.rst +++ b/doc/sphinx/index.rst @@ -35,7 +35,9 @@ The third section focuses on the **USAGE** of the bastion, from the perspective The fourth section is about the proper **ADMINISTRATION** of the bastion itself. If you're about to be the person in charge of managing the bastion for your company, you want to read that one carefully! -The fifth section is the complete reference of all the **PLUGINS** that are the commands used to interact with the bastion accounts, groups, accesses, credentials, and more. +The fifth section is about **DEVELOPMENT** and how to write code for the bastion. If you'd like to contribute, this is the section to read! + +The sixth section is the complete reference of all the **PLUGINS** that are the commands used to interact with the bastion accounts, groups, accesses, credentials, and more. The unavoidable and iconic FAQ is also available under the **PRESENTATION** section. @@ -82,6 +84,7 @@ The unavoidable and iconic FAQ is also available under the **PRESENTATION** sect :caption: Development development/setup + development/tests .. _plugins: