Feature Request: Support or document methods of "black box" integration testing

[drmrd]

When wishing to run test scenarios against the full EVerest stack that require the monitoring of EVerest modules, the approach that appears publicly in everest-core tests is to dynamically define a new type of EVerest module at runtime using everestpy and run it as a standalone module alongside the rest of the stack. This custom "proble module" relies on everestpy to connect to the MQTT message broker and publish/subscribe to topics associated with other modules.

This approach has a number of limitations, some of which are as follows:

1. Required Knowledge of EVerest Tooling: The test in startup_tests.py looks readable on the surface, but in order to reuse it in new tests, a developer almost immediately requires a knowledge of each of the following (at a minimum):

   • everestpy from everest-framework (and pybind11 syntax, accordingly).
   • How the everest.testing submodule is really a copy of the everest-testing submodule from everest-utils, which to my knowledge isn't explicitly documented anywhere.
   • What a standalone module is and what features are/are not supported with them.

2. Not Testing a Production EVerest Stack: While unlikely, it's possible that introducing a misconfigured standalone module into EVerest results in a change to the behavior of other modules within EVerest.

3. Lack of Convenient Testing Abstractions: In its current implementation, the probe module is not general purpose, meaning that a bespoke probe module is defined for testing a specific behavior. Without a higher-level abstraction here, this pattern will likely result in quite a bit of boilerplate code in test suites and brittle tests.

4. Complexity: This seems like quite a few layers of abstraction simply to trigger a module command or obtain a variable value.

All of these issues can be addressed through a mix of improvements to the main EVerest documentation, providing documentation for everestpy, extending the documentation of everest-testing, extending everest-testing with some kind of general-purpose probe module, etc. That said, I'm curious why the community has adopted this approach over building out a testing framework that connects to MQTT directly, using a client like Paho.

Either way, from my own experience and conversations with other EVerest adopters, it seems like commentary on existing testing approaches would be a welcome contribution to the documentation.

[drmrd]

Communicating with modules over MQTT directly has other advantages, as well. In particular, it enables third parties to perform "black box" integration with EVerest, where EVerest is stood up in conjunction with other software (and hardware) to confirm expected behavior across a more connected set of services (e.g., SIL or HIL tests involving an EV, EVerest, and a CSMS).

I believe @shankari wanted to provide some real world examples from recent PKI testing efforts where this would come in handy.

[andistorm]

Hi @drmrd @shankari,
thanks for bringing this topic up!
There are various levels of Testing:

• Unit Testing
• Integration Testing (with probe module)
• Integration Testing (without probe module)

I agree we need more documentation regarding testing, including:

• explanation of testing-framework / everest-testing / everest.testing / probe_module
• everytpy / pybind
• pytest

This would adress your 1. point. I agree this knowledge is necessary to impelement Tests on Integration level (with probe module) I also guess it would adress your 3. point because I would say the probe module is not defined for specific testing and is general purpose. So don't agree on 3.
The probe module is abstracted away in some pytest fixtures in a way it provides a "handy" standalone module as possibility to inject inputs / ouputs on module layer. On this level it wouldn't make sense to speak mqtt directly, because you wouldn't need even more detailed knowledge on the EVerest framework. With probe module you nearly only need the knowledge you need to implement modules.

To point 4: If you are not familiar with pytest it looks quite complex, but the testing framework is "just" a collection of "handy"/useful fixtures (pytest) that are controlled by markers. Both can be reused.

To point 2:
I agee there are currently no integration tests on this level (integration tests without probe module). In my opinion it would make sense to also use pytest for these and start EVerest with the everest-core fixture, use external mqtt API of SIL modules (which also needs improvement in documentation) to control the simulation. For this it would probably a good idea to add fixtures like sil_controller which can be based on the already existing fixture connected_mqtt_client

So in summarization:

• We need imrpoved documentation
• testing framework should be extended to adress testing on integration level without probe module

Open questions:

• @shankari, you mentioned to dockerize this testing level, could you explain your idea in more detail?