Feature request: mull line coverage

[Warwolt]

(Writing this feature request after a twitter conversation with Alex Denisov)

User story:

"As a developer looking to verify the completeness of my unit tests, I want Mull to be able to produce the line coverage of the code under test, so that I can be sure that the 100% mutant kill rate is based on as many mutants as possible for a given source file."

Motivation

The basic idea is that, for a given source file module.c with a test suite module_test.cc there should be some set M of all mutants possible to generate with Mull. If the line coverage of module_test.cc is less than 100%, it is possible that the set of actually generated mutants M´ form a strict subset of the set of M, that is M' ⊂ M.

The kill rating is based on the number of kills to the total number of mutants, with the goal of getting a 100% kill rate. When the set of mutants being trialed with the test suite is the subset M', then the 100% kill rate represents less behavioural coverage than the 100% kill rate for the set M.

The goal of mutation testing is to measure to the extent that a test suite enforces the behaviour of the code under test, and the ability for the mutation tool to perform this task is strongly related to the line coverage of the test suite.

As an illustrative example, imagine that module.c contains 10 functions each with 10 possible mutants, but only 2 of the functions are fully line covered by module_test.cc. Even with a 100% mutant kill rate, we have only covered 20% of the possible behaviour that Mull would be able to detect, and therefore still have improvements to make that aren't reported.

My feature request therefore is to add a line coverage metric to Mull during mutation testing, and report this along with the unkilled mutants, to guide developers in gauging how much more work needs before a test suite can be considered done.

Knowing that there is a 100% line coverage and a 100% mutant kill rate would give strong arguments for the validity of the test suite being measured, and so would add I believe quite a lot of value to the already good Mull mutation tool.

[chgans]

I do exactly that, i build and run my tests with code coverage, and then I build and run them again with mutation. I end up with 2 metrics.
Are you asking for mull to be compatible with code coverage? so that (in my case) and only need 1 build and 1 run?

[Warwolt]

Yes essentially, I've seen other mutation testing tools able to tell the amount of coverage in the test report and imply that there isn't a true 100% kill rate until all code is covered.

Of course running a coverage separately is an easy work around, but this would be a real Nice-To-Have.

[AlexDenisov]

Hi @Warwolt, since we've moved away from JIT, Mull does not gather code coverage information on its own anymore: it can only rely on the coverage info coming from Clang (at least for the time being).

I implemented support for non-covered mutants here #831.
As far as I understand, this is not exactly what you are asking for, but I do not have any better suggestions at the moment.
You still have to use code coverage from Clang, but at least Mull won't say that the mutation coverage is 100% if the actual line coverage is not full.
Also, I suggest you try the Mutation Testing Elements report. There you will see not-covered mutants highlighted in a special way (see the screenshot).

If this solution satisfies your needs, then I suggest we mark this issue as done :)
Otherwise, I'm open to discussion on how we can improve it even further ;)

P.S. Currently, Mull only works on the level of "function coverage," i.e., if at least one line in the function is covered, it considers the whole function for mutation. This will be addressed as part of #790 and #791.

[Warwolt]

I opened this issue because I noticed that Mull could report a "100%" score even though you hadn't actually covered your test to the extent that Mull would be able to report due to the missing code coverage of the code under test. In the case that Mull doesn't report 100% score anymore unless you've fully killed all possible mutants and not just the applicable ones, then I think this issue can safely be closed, based on my concern.