No Unit Test Coverage or Mutation Coverage reported for Robolectric based Unit Tests

[im-not-josh]

Hello,

I am seeing an issue where, the Unit Test coverage and Mutation test coverage reported for each unit test using RobolectricTestRunner is 0.
Pitest runs to completion, there is no error. If i remove RobolectricTestRunner (and disable tests that need it) coverage is reported as expected.

I am using the latest version of pitest and robolectric, 1.7.3 and 4.7.3.

Setting excludeMockableAndroidJar = true or excludeMockableAndroidJar = false doesnt seem to make a difference here either.

Any ideas?

[koral--]

Could you share a project with a minimal reproducible example demonstrating this issue?

[im-not-josh]

Hey @koral--
Finally coming back to this, sorry for the delay.
https://github.com/im-not-josh/PlayGround
Clone this repo, run pitestDebug gradle task and you should see this:

The class NotCoveredLogic has the RobolectricTestrunner, which, if you comment out, then returns 100% mutation coverage, the same as CorrectlyCoveredLogic class

[im-not-josh]

Interestingly, if i add
excludeMockableAndroidJar = true
to the task config, i then see this

[koral--]

Thanks, will check it.

[im-not-josh]

Hey there @koral-- checking in with you, did you get a chance to check this issue over?

[koral--]

Hi, sorry for the delay but not yet.

[koral--]

It seems that robolectric perform some actions that cause pitest to not work correctly.
There is one issue reported related to Robolectric: hcoles/pitest#744 (comment)

I've reported this issue to upstream: hcoles/pitest#1065

[im-not-josh]

Thanks for the effort to date, hoping this can get some attention upstream

[hcoles]

I've taken a look at this and the issue looks to be that Roboelectric now loads the classes from a classloader other than the context one (this is a change from the last time I had to prod pitest to get the two working together some years ago).

This is issue is similar to one I fixed recently for QuarkusTest, but roboelectric appears to keep a pool of classloaders somewhere, which complicates things for pitest considerably.

I have some horrible hacking locally that looks to fix the issue for the simple case demonstrated in https://github.com/im-not-josh/PlayGround.

I'm concernted it may not work with something more complex that uses roboelectric for real. I don't do Android dev, so have no idea what that looks like.

If someone could create a slightly more complex project that makes more realistic use of roboeletric I'll see if the fix looks viable.

If there's anyway this can be made self contained (ie not rely on an android sdk being already configured on a machine) that would be helpful. At the moment I can't see how I could add an integration test to the build for this.

[im-not-josh]

Wow awesome job! I'm so happy to have progress on this so fast. Thank you!
I originally created that repo and it is fairly representative of what robolectric tests normally look like. The only things that might change in an enterprise app is increase in quantity of test classes and tests inside each class and sometimes a more complicated test which relies on one or more robolectric classes (like Context or Application class)
Do we have the option of creating a snapshot of your change I can test on an enterprise app? That would be a faster way to validate your change. Otherwise I can expand that sample app a little, it will just take a bit of time

[hcoles]

@im-not-josh The roboelectric test in the repo doesn't seem to actually do anything - it runs green with the standard junit runner. I presume a more representative test would mock out part of the android sdk in some way?

If you could update it so it does even a minimal amount of androidy stuff, that would be helpful.

I'll push a branch with my changes once I've tidied up some of the hacking.

[hcoles]

@im-not-josh The changes for this are in the branch

https://github.com/hcoles/pitest/tree/feature/roboelectric

If you'd like to try it out

mvn install

should take about 3:30, and produce pitest-1.0.0-SNAPSHOT into your local .m2 repo.

It would still be helpful if you could update https://github.com/im-not-josh/PlayGround to include some slightly more realistic roboelectric tests.

My main concern is that the bytecode transformation pitest is doing will clash with any transformation performed by roboelectric (which I think will not be performing any in the current example).

[hcoles]

@im-not-josh Have you had chance to update the repo or try out the branch? I'd prefer not to merge the changes into pitest until I have some confidence they fix the issue.

[im-not-josh]

Hey @hcoles
Thanks so much for the reminders and thanks so much for work done so far, I really appreciate it.
I did push 2 more test classes with 4 more super trivial tests in there. Its an over simplified idea of what we might normally do in an android project but it will at leas give you a better picture.

I updated the versions of pitest ( to match current prod releases, not your robo changes branch) also but it seems like that has caused some more issues.

The tests I added fail before mutation, stating the string resources cannot be found.
I tried to toggle the value of excludeMockableAndroidJar which is part of pi test for android plugin - setting to true allows mutation testing to pass, but it ignores the newly added class.

I will also some more robolectric based tests and android junit based tests with some more examples for you.

[im-not-josh]

More tests added, some test based on Room DB.

Normally in robolectric tests, we add this into our gradle config:
android {
testOptions {
unitTests {
includeAndroidResources = true
}
}
}

it allows resources like strings to be included in tests builds. This is set in the example app so the tests pass when running outside of mutation. Seems like that same property should also be applied in the mutation tests. That should resolve the no string resource found error. For now, those tests could be commented out to run to completion.

[im-not-josh]

Added yet more tests, more android sdk based this time. Bitmap stuff which is always fun

[im-not-josh]

Last update for the day today, I finally managed to get your snapshot working locally.
The good news is the original class with no reported coverage now works. The bad news is, the other tests I included dont work. The other tests I included actually report 0% unit test coverage as well - jacoco reports has varying levels, but bitmap utils is 74%, so at least, non 0% coverage.

[hcoles]

@im-not-josh Thanks for this. Unfortunately I won't be able to look into for a few weeks, but the updated example will be very helpful.

[cee-dee]

I'm also facing this issue, so I'd be interested in any updates :-)
Is there any news on this?