Return code 0 when 256 tests fail

[milandamen]

Describe the bug
When running tests with qx test and the amount of failing tests is 256, the resulting code will be 0.

To Reproduce
Steps to reproduce the behavior:

1. Have a collection of tests where exactly 256 tests fail.
2. Run the tests using qx test
3. Get the result code using echo $?

Expected behavior
Expect a non-zero result code, because not all tests passed.

Log

$ qx test
...
DONE testing 405 ok, 256 not ok, 0 skipped - [21942 ms]
$ echo $?
0

Desktop:

• OS: Kubuntu 20.04.3 LTS
• Browser: Chromium
• Version: Version 97.0.4692.71 (Official Build) snap (64-bit)

qx-lock.json

{
  "libraries": [
    {
      "library_name": "Commandline Testrunner for Qooxdoo Apps",
      "library_version": "1.0.12",
      "path": "qx_packages/qooxdoo_qxl_testtapper_v1_0_12",
      "uri": "qooxdoo/qxl.testtapper",
      "repo_name": "qooxdoo/qxl.testtapper",
      "repo_tag": "v1.0.12"
    },
    {
      "library_name": "logpane",
      "library_version": "1.0.3",
      "path": "qx_packages/qooxdoo_qxl_logpane_v1_0_3",
      "uri": "qooxdoo/qxl.logpane",
      "repo_name": "qooxdoo/qxl.logpane",
      "repo_tag": "v1.0.3"
    }
  ],
  "version": "2.1.0"
}

Qooxdoo framework version: 7.0.0-beta.6

[hkollmann]

This is a problem of bash - error code is only a byte. Any suggestions? Stop counting by 255?

[milandamen]

I think the exit code should not reflect how many tests failed, but what kind of "error" occurred.

I kind of like how Go's Ginkgo package does exit codes:

• 0 if every test passed and no other errors occurred.
• 1 if one or more test failed.
• 197 if every test passed but not all possible tests were run, for example when using --class or --method. This is useful because you do not want a CI to pass if not all tests were run.