Discussion: PHPUnit configuration

[mk-mxp]

While digging into the performance degradation caused by upgrades in #656 and the problems seen in exercism/php-test-runner#100 during the upgrades:

Should we introduce a well defined and binding (CI enforced) PHPUnit configuration for contributors and one less strict for the test-runner processing student code?

Currently we rely on the PHPUnit defaults, which differ largely between major versions 9, 10, 11 and 12. This, I think, was a choice made to reduce the burden of maintenance between different moving parts. Also "no configuration" is the most likely configuration of students using the CLI. But this seems to cause problems now.

There are options to limit the test runtime to a certain amount of time. There are options to set the output specifications. The option to fail when there are no test fails but other issues like deprecations. All can be specified as commandline options in shell scripts or in config files.

I think it is easier to fiddle with configuration files while looking for problems. And I think files are much easier to compare than commandline options in various shell scripts. But these also have to be maintained to keep them aligned to PHPUnit versions. And a config file needs to specify each and every option to be safe against changes in defaults. Maybe this would also yield useful deprecation or failure messages when the options change (e.g. are renamed due to semantic changes).

But with files there is a risk of differing substantially from the defaults a student would have on CLI. Other tracks (like JavaScript) provide a bunch of additional files with each exercise (stored in each exercise, which is many copies). We could use that for PHPUnit config files.

[mk-mxp]

We now have another test runner approach implemented and the need for this seems to be gone.