Several XWiki commons modules report 0 mutation score

[vmassol]

When running pitest/descartes on xwiki-commons using a threshold to fail the build, I noticed several modules that generate 0% mutation score even though they have several tests. Namely:

• xwiki-commons-text
• xwiki-commons-cache-infinispan
• xwiki-commons-extension-handler-jar
• xwiki-commons-observation

For the first one xwiki-commons-text it can be explained by the fact that it uses JUnit5 tests. However the other ones are using JUnit4.

Also note that the target/test-classes directories correctly contain the test classes.

I've attached the zip of target/pit-reports for xwiki-commons-cache-infinispan at https://up1.xwikisas.com/#YBg00VTYn3l82nxtafX3yg in case it can help.

Here's the console output for this module:

...
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ xwiki-commons-cache-infinispan ---

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
objc[9680]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/bin/java (0x10ed7d4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x110df14e0). One of the two will be used. Which one is undefined.
Running org.xwiki.cache.infinispan.InfinispanCacheTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.331 sec - in org.xwiki.cache.infinispan.InfinispanCacheTest
Running org.xwiki.cache.infinispan.InfinispanConfigTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.557 sec - in org.xwiki.cache.infinispan.InfinispanConfigTest

Results :

Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
...
[INFO] --- pitest-maven:1.2.0:mutationCoverage (pitest-check) @ xwiki-commons-cache-infinispan ---
[INFO] Found plugin : JSON report plugin
[INFO] Found plugin : Default csv report plugin
[INFO] Found plugin : Default xml report plugin
[INFO] Found plugin : Default html report plugin
[INFO] Found plugin : Default limit mutations plugin
[INFO] Found plugin : Kotlin junk mutations filter
[INFO] Found shared classpath plugin : Engine for extreme mutation operators
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Adding fr.inria.stamp:descartes to SUT classpath
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-cache/xwiki-commons-cache-infinispan/target/classes
[INFO] Defaulting target classes to match packages in build directory
8:44:59 AM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
CP is :/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-cache/xwiki-commons-cache-infinispan/target/test-classes:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-cache/xwiki-commons-cache-infinispan/target/classes:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-cache/xwiki-commons-cache-api/target/xwiki-commons-cache-api-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-configuration/xwiki-commons-configuration-api/target/xwiki-commons-configuration-api-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-component/xwiki-commons-component-api/target/xwiki-commons-component-api-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-stability/target/xwiki-commons-stability-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-text/target/xwiki-commons-text-10.3-SNAPSHOT.jar:/Users/vmassol/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar:/Users/vmassol/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/Users/vmassol/.m2/repository/commons-beanutils/commons-beanutils/1.9.3/commons-beanutils-1.9.3.jar:/Users/vmassol/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-environment/xwiki-commons-environment-api/target/xwiki-commons-environment-api-10.3-SNAPSHOT.jar:/Users/vmassol/.m2/repository/commons-io/commons-io/2.6/commons-io-2.6.jar:/Users/vmassol/.m2/repository/org/infinispan/infinispan-core/8.2.6.Final/infinispan-core-8.2.6.Final.jar:/Users/vmassol/.m2/repository/org/infinispan/infinispan-commons/8.2.6.Final/infinispan-commons-8.2.6.Final.jar:/Users/vmassol/.m2/repository/org/jgroups/jgroups/3.6.7.Final/jgroups-3.6.7.Final.jar:/Users/vmassol/.m2/repository/org/jboss/marshalling/jboss-marshalling-osgi/2.0.0.Beta3/jboss-marshalling-osgi-2.0.0.Beta3.jar:/Users/vmassol/.m2/repository/org/jboss/logging/jboss-logging/3.3.2.Final/jboss-logging-3.3.2.Final.jar:/Users/vmassol/.m2/repository/javax/transaction/jta/1.1/jta-1.1.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-cache/xwiki-commons-cache-tests/target/xwiki-commons-cache-tests-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-tools/xwiki-commons-tool-test/xwiki-commons-tool-test-component/target/xwiki-commons-tool-test-component-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-tools/xwiki-commons-tool-test/xwiki-commons-tool-test-simple/target/xwiki-commons-tool-test-simple-10.3-SNAPSHOT.jar:/Users/vmassol/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/vmassol/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/vmassol/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.1.0/junit-jupiter-api-5.1.0.jar:/Users/vmassol/.m2/repository/org/apiguardian/apiguardian-api/1.0.0/apiguardian-api-1.0.0.jar:/Users/vmassol/.m2/repository/org/opentest4j/opentest4j/1.0.0/opentest4j-1.0.0.jar:/Users/vmassol/.m2/repository/org/junit/platform/junit-platform-commons/1.1.0/junit-platform-commons-1.1.0.jar:/Users/vmassol/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.1.0/junit-jupiter-engine-5.1.0.jar:/Users/vmassol/.m2/repository/org/junit/platform/junit-platform-engine/1.1.0/junit-platform-engine-1.1.0.jar:/Users/vmassol/.m2/repository/org/junit/platform/junit-platform-launcher/1.1.0/junit-platform-launcher-1.1.0.jar:/Users/vmassol/.m2/repository/org/junit/vintage/junit-vintage-engine/5.1.0/junit-vintage-engine-5.1.0.jar:/Users/vmassol/.m2/repository/org/hamcrest/hamcrest-library/1.3/hamcrest-library-1.3.jar:/Users/vmassol/.m2/repository/org/jmock/jmock/2.6.0/jmock-2.6.0.jar:/Users/vmassol/.m2/repository/org/jmock/jmock-junit4/2.6.0/jmock-junit4-2.6.0.jar:/Users/vmassol/.m2/repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.jar:/Users/vmassol/.m2/repository/net/bytebuddy/byte-buddy/1.7.9/byte-buddy-1.7.9.jar:/Users/vmassol/.m2/repository/net/bytebuddy/byte-buddy-agent/1.7.9/byte-buddy-agent-1.7.9.jar:/Users/vmassol/.m2/repository/org/objenesis/objenesis/2.6/objenesis-2.6.jar:/Users/vmassol/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar:/Users/vmassol/.m2/repository/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar:/Users/vmassol/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.25/jcl-over-slf4j-1.7.25.jar:/Users/vmassol/.m2/repository/org/slf4j/log4j-over-slf4j/1.7.25/log4j-over-slf4j-1.7.25.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-observation/xwiki-commons-observation-api/target/xwiki-commons-observation-api-10.3-SNAPSHOT.jar:/Users/vmassol/.m2/repository/org/apache/commons/commons-lang3/3.7/commons-lang3-3.7.jar:/Users/vmassol/.m2/repository/jmock/jmock/1.2.0/jmock-1.2.0.jar:/Users/vmassol/.m2/repository/jmock/jmock-cglib/1.2.0/jmock-cglib-1.2.0.jar:/Users/vmassol/.m2/repository/cglib/cglib-nodep/2.1_3/cglib-nodep-2.1_3.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-component/xwiki-commons-component-default/target/xwiki-commons-component-default-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-component/xwiki-commons-component-observation/target/xwiki-commons-component-observation-10.3-SNAPSHOT.jar:/Users/vmassol/dev/xwiki/xwiki-commons/xwiki-commons-core/xwiki-commons-context/target/xwiki-commons-context-10.3-SNAPSHOT.jar:/Users/vmassol/.m2/repository/fr/inria/stamp/descartes/0.2-SNAPSHOT/descartes-0.2-SNAPSHOT.jar:/Users/vmassol/.m2/repository/org/pitest/pitest/1.2.0/pitest-1.2.0.jar
8:44:59 AM PIT >> INFO : MINION : objc[9681]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/bin/java (0x107e304c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x108f02
8:44:59 AM PIT >> INFO : MINION : 4e0). One of the two will be used. Which one is undefined.

8:44:59 AM PIT >> INFO : Sending 0 test classes to minion
8:44:59 AM PIT >> INFO : Sent tests to minion
8:45:00 AM PIT >> INFO : MINION : 8:45:00 AM PIT >> INFO : Checking environment

8:45:00 AM PIT >> INFO : MINION : 8:45:00 AM PIT >> INFO : Found  0 tests

8:45:00 AM PIT >> INFO : MINION : 8:45:00 AM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

8:45:00 AM PIT >> INFO : MINION : 8:45:00 AM PIT >> INFO : 0 tests received

8:45:00 AM PIT >> INFO : Calculated coverage in 0 seconds.
8:45:00 AM PIT >> INFO : Created  4 mutation test units
8:45:00 AM PIT >> INFO : Completed in 0 seconds

So for some reasons it's not finding the tests.

[oscarlvp]

PITest seems to have several issues where some tests are not detected/run. It might be related to some of the following open issues: hcoles/pitest#426, hcoles/pitest#355 or even with custom runners such as hcoles/pitest#402.

[tmortagne]

The reason for xwiki-commons-cache-infinispan seems to be #71