Seeking implementation advice for a plugin to run tests inside a desktop app

[LKajan]

I would like to better integrate pytest into our development process. We develop plugins for a desktop application (QGIS) that has an embedded Python interpreter. At startup, the desktop app initializes part of its Python API, making some parts of the API usable only when Python is run inside the desktop app. Currently, we run pytest outside the desktop app and have mocked the APIs that the desktop app provides. We would like to run tests inside the desktop app to test the plugins in the actual environment, reducing the need for extensive mocking.

I’m considering writing a pytest plugin for this purpose. The plugin would start the desktop app, which would then run a bootstrap Python script to execute pytest. The plugin would also collect the test results and display them in the main process.

I’m seeking advice on how to implement this. Specifically, I’m unsure how to start the desktop app from the pytest plugin and how to collect the test results. I’m also uncertain if this is the best approach to run pytest inside the desktop app.

My initial idea is that a user would run pytest in their development environment as usual, with all the supported arguments. The pytest plugin would likely implement the pytest_runtestloop hook, where the necessary bootstrapping Python script is generated, and the desktop app is started with the bootstrap script as a startup script. The bootstrap script would run the collected tests and pass the test results back to the main process.

I’m looking for suggestions on the following questions:

• How to actually run the collected tests in the bootstrap script?
• Or should I collect the tests later in the desktop process?
• How to pass the test results back to the main process?

[RonnyPfannschmidt]

this one is very tricky - 3 possible choices are relatively easy to come up with as an idea - ease of implementation may be harrowing

a) run all if pytest inside the application - put a testreport to the outside
b) create an api that shares the mock apis but network communicates to the inside of the applciation (this can be a horror or a joy depending on the application under test)
c) create a execnet hack for pytest-xdist that enables to run a "-n 1" pytest execution with a pytest on the outside that starts the application and a normal pytest-xdist on the inside (this will require crude hacks and deep understanding of xdist/execnet - but its basically all about starting a execnet socketserver inside the application, then running xdist against it
)