Robolectric sample project

[alixwar]

I added a sample project based on my own setup:

• Gradle plugin 2.x (latest released)
• Robolectric (latest released)

There is a Robolectric test in the sample project which executes one line of code but I'm not able to get any code coverage

Steps to reproduce:

• Run gradlew testDebug
• Analyze sample-robolectric\build\jacoco\junitPlatformTestDebug.exec (In IntelliJ you can open an .exec-file with ctrl+alt + F6), it shows coverage for all lines of code in de.mannodermaus.gradle.plugins.android_junit5.sample_robolectric.Foo except for the method junit4robolectric()

[alixwar]

I've cleaned up the sample now

[alixwar]

I improved the sample to include JUnit4 and JUnit5 tests where code coverage is reported correctly

[IgorGanapolsky]

Where is sample-robolectric and where is sample_robolectric? Are they the same module.

[alixwar]

Where is sample-robolectric and where is sample_robolectric? Are they the same module.

I'll take a look later tonight! I'm sure it's a mistake on my part. Should be the same

[alixwar]

@IgorGanapolsky I think it looks alright. The folder is called sample-robolectric (following the naming convention of the project in general) but the package name is sample_robolectric (dash not permitted). Do you have trouble building the project?

[IgorGanapolsky]

@alixwar You might want to fetch upstream and merge into your repo. Dependency versions were updated yesterday...

[alixwar]

@IgorGanapolsky Thanks for telling me, will do that now

[alixwar]

@IgorGanapolsky I think it should be fine now

[alixwar]

I opened the generated exec file and it's the same result. All tests (JUnit4, JUnit5, Mockito) generate coverage data except for the Robolectric test.

Side note: I managed to f*** up the merge so the changes are quite big. Don't look at the changes, rather follow the steps to reproduce after checking out the project.

[IgorGanapolsky]

Ok, it builds now. One question, what do the tests in sample-robolectric do - just generate coverage report? If that's the case, how to open .exe file on a Mac?

[alixwar]

@IgorGanapolsky sample-robolectric contains a source file, Foo.java with some methods (each method is tested by a separate test). The project contains four different types of tests. Jacoco has been configured so running these tests will generate an exec file (code coverage data file). This exec file proves that code coverage is not generated for the Robolectric test as there is no coverage data for the method that is tested by the Robolectric test.

How to open the .exec file on a Mac? I don't know, but your favourite IDE capable of Java code probably supports loading the file and displaying code coverage

[VenomVendor]

The whole formatting is screwed up from 4 spaces to 2 spaces, was it intentional?
Mostly in this commit
It's hard to find the actual changes now. #18

Ok, found the actul commit, 2-space formatting

[IgorGanapolsky]

It was probably formatted automatically by Android Studio upon commit.

[alixwar]

Yeah, I'm not sure why this fork has diverged from the upstreams repo (not a Git wizard) but the point is not to get this PR approved but rather to prove that code coverage is not generated properly even using the latest available version of dependencies

[mannodermaus]

Yeah, sorry about that, that reformatting is on me. Since your sample project is quite isolated from the remainder of the repository (i.e. its own module), it should be fairly easy to merge the upstream back into your fork. That being said, I also see this PR as a proof-of-concept for missing code coverage with Robolectric, hence why I haven't merged it in yet.

[joecks]

I ran the changes and figured out that it is still not possible to run roboloectric tests with junit5, thus using the robolectric test runner. Any suggestions how to solve that?

[alixwar]

I ran the changes and figured out that it is still not possible to run roboloectric tests with junit5, thus using the robolectric test runner. Any suggestions how to solve that?

@joecks
Weird. I was able to run the tests without any issues. The only problem for me was the code coverage. How do you run your tests? From an IDE or command line?

[joecks]

@alixwar Both. The issue is that there is actually no junit5 test here that is ran with the RobolectricTestRunner. So if I am running any Android related action like accessing the Context this test crash due to a not mocked exception

java.lang.RuntimeException: Method getMainLooper in android.os.Looper not mocked. See http://g.co/androidstudio/not-mocked for details.

example is

    @Test
    void junit5codeCoverage() {
        final int result = new Foo().junit5();

        Assert.assertEquals(Looper.getMainLooper().getThread(), Thread.currentThread());

        assertEquals(-1, result);
    }

So I am still not sure how to achieve a combination between robolectric an junit5, any suggestions?

[alixwar]

@joecks I was not able to reproduce your problem (Windows 10). This is the result after running the tests from command line:

D:\Repositories\android-junit5\sample-robolectric>..\gradlew clean testDebug
Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use --status for details

> Configure project :sample-robolectric
NDK is missing a "platforms" directory.
If you are using NDK, verify the ndk.dir is set to a valid NDK directory.  It is currently set to C:\Users\Alix\AppData\Local\Android\Sdk\ndk-bundle.
If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning.

The setTestClassesDir(File) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use the setTestClassesDirs(FileCollection) method instead.
The getTestClassesDir() method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use the getTestClassesDirs() method instead.
The ConfigurableReport.setDestination(Object) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use the method ConfigurableReport.setDestination(File) instead.

> Task :sample-robolectric:compileDebugUnitTestJavaWithJavac
warning: unknown enum constant Status.STABLE
  reason: class file for org.apiguardian.api.API$Status not found
warning: unknown enum constant Status.STABLE
2 warnings

> Task :sample-robolectric:junitPlatformTestDebug
WARNING: No manifest file found at .\AndroidManifest.xml.
Falling back to the Android OS resources only.
To remove this warning, annotate your test class with @Config(manifest=Config.NONE).

Test run finished after 2111 ms
[         6 containers found      ]
[         0 containers skipped    ]
[         6 containers started    ]
[         0 containers aborted    ]
[         6 containers successful ]
[         0 containers failed     ]
[         4 tests found           ]
[         0 tests skipped         ]
[         4 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         0 tests failed          ]

BUILD SUCCESSFUL in 11s
18 actionable tasks: 17 executed, 1 up-to-date

[joecks]

Well it is not an issue per se with your branch, but right now it is not very useful, because it misses the usage of Android classes with junit5 tests which I thought was the purpose of this exercise, but maybe I was wrong. You can see the issue by adding this line:

Assert.assertEquals(Looper.getMainLooper().getThread(), Thread.currentThread());

in your junit5codeCoverage test. Or do you mean that you have patched and tested this already? I

[joecks]

Btw I get this kind of error that is not very surprising:

Failures (1):
  JUnit Jupiter:FooJunit5Tests:junit5codeCoverage()
    MethodSource [className = 'de.mannodermaus.gradle.plugins.android_junit5.sample_robolectric.FooJunit5Tests', methodName = 'junit5codeCoverage', methodParameterTypes = '']
    => java.lang.RuntimeException: Method getMainLooper in android.os.Looper not mocked. See http://g.co/androidstudio/not-mocked for details.

[alixwar]

@joecks I think that you have misunderstood the purpose of this branch. The issue that I'm narrowing down is the lack of code coverage when running tests annotated @RunWith(RobolectricTestRunner.class)

The issue you experience sounds like a pure Robolectric runtime environment issue. Maybe @mannodermaus could comment on this?

[joecks]

Well maybe I have misunderstood the purpose indeed. I was just confused to see that @RunWith(RobolectricTestRunner.class) would not work with junit5, which I thought an robolectric-example module should cover.
But did you managed to run robolectric tests with junit5?

[alixwar]

@joecks

But did you managed to run robolectric tests with junit5?

Yes, I can run all the tests in sample-robolectric (as mentioned in the comment). It also works fine running the tests from IntelliJ IDEA (there I even get full code coverage data). The only issue is when I run the tests from command line and inspect the generated ".exec" file where the coverage data is missing for the Robolectric test

[joecks]

But the robolectric test are run with junit4 and not 5, or?

[alixwar]

But the robolectric test are run with junit4 and not 5, or?

JUnit5 using the vintage junit4 runner. To get pure JUnit5 support the Robolectric project needs to implement new extensions. But this is a good point that you have, it is not clear that I meant JUnit5 with the legacy runner

[joecks]

@mannodermaus do you have any inside on how to implement a junit5 robolectric test runner?

[mannodermaus]

You'd basically have to convert the RobolectricTestRunner into a JUnit 5 Extension in order to drive Robolectric tests with the Jupiter TestEngine. There are several extension points to hook into, and that should be enough to migrate over the implementation details of Robolectric. Unfortunately I don't have the insight into the internal architecture of the library, nor the resources to dedicate to these efforts - it would be best to raise awareness over at Alix' issue in their Github!

[erikhuizinga]

What's the status of this PR and, more importantly, the status of Android + JUnit 5 + Robolectric?

[mannodermaus]

This PR used to be a proof-of-concept integration of Robolectric against an outdated version of the android-junit5 plugin. It's up to the Robolectric team to provide full support for JUnit 5, which in turn is dependent on a good proposal for this issue on the JUnit 5 repo first. Until a real integration of the two is managed by the respective team, there won't be a canonical sample project in this repo. I'd also like to link this section from Danny's talk on JUnit 5 at this year's Droidcon, because it gives a great overview of the current state of this topic, too.

[vincent-paing]

After 5 years, Is the state of Roboletric still the same where it doesn't work with Junit5?

[mannodermaus]

@vincent-paing The main blocker has been resolved from JUnit 5's end in the meantime, but regardless Robolectric still does not work with it. I gave some context in a pinned issue that you can check out – it also has a link to the Robolectric feature request if you would like to read more or chime in.

With the disparity between the current state of this repo and the timestamp of this POC PR, I'm taking the liberty to also close it down. 🙇‍♂️ I'm crossing my fingers that there'll eventually be a time where Robolectric and JUnit 5 will finally be united in harmony.