Add UI Tests using Espresso Testing Framework #1900

simosathan9 · 2024-05-22T19:19:52Z

This PR introduces some UI tests using the espresso testing framework.

LoyaltyCardCreationTest class simulates the process of creating a loyalty card at Catima and does assertions on many field insertions.

LanguageChangeUITest class simulates the process of changing the language of the application. When a change is happening the test asserts that the text in the app matches the chosen language.

I also added the necessary dependencies on build.gradle.kts

To execute the tests you must have an open device or emulator and run the following command :
./gradlew connectedAndroidTest

Screenshot automation workflow using fastlane screengrab could be added through those tests to satisfy issue #712

TheLastProject

This looks pretty cool! Thanks for your MR!

I haven't had the time to look super thoroughly yet but I'm noticing some things that I find a bit weird on first pass.

I also think this isn't tested in GitHub Actions yet? That would be quite important to have to be honest as it would otherwise be way too easy for the test to accidentally break without anyone noticing.

TheLastProject · 2024-05-25T10:09:02Z

app/build.gradle.kts

@@ -94,6 +95,10 @@ dependencies {
    implementation("androidx.preference:preference:1.2.1")
    implementation("com.google.android.material:material:1.12.0")
    implementation("com.github.yalantis:ucrop:2.2.9")
+    implementation("androidx.tracing:tracing:1.2.0")


Is this dependency necessary? Given only new tests are being added, it seems weird to me to add a new runtime dependency.

You are probably right this dependency might not be necessary. While trying to integrate fastlane screengrab (unsuccessfully) within these tests I had to add some more dependencies and probably did not remove them.

I can see that the tests fail if you remove this, it seems to be a dependency of Espresso.

However, given it is an Espresso test dependency and not a runtime dependency, it seems to me this should be an androidTestImplementation instead of implementation so we don't ship it with the actual APKs users will install as it's not needed there.

So that's what I tried... and it didn't work. And then I researched more. And... it seems to be an AGP bug. Sigh: android/android-test#1755 (comment)

I decided to ask yet again for a way to track this issue as the previous request for that got no answer. It's definitely an ugly workaround, but it seems the only option currently: android/android-test#1755 (comment)

TheLastProject · 2024-05-25T10:13:27Z

app/build.gradle.kts

+    androidTestImplementation("androidx.test.ext:junit:1.2.0-beta01")
+    androidTestImplementation("androidx.test:rules:1.6.0-beta01")
+    androidTestImplementation("androidx.test.espresso:espresso-contrib:3.6.0-beta01")


I'm seeing some duplicates here. All three of these seem to also be listed under the "Espresso" header. So I think those could be removed here?

TheLastProject · 2024-05-25T10:19:57Z

app/src/androidTest/java/protect/card_locker/LoyaltyCardCreationTest.java

+    public GrantPermissionRule mGrantPermissionRule =
+            GrantPermissionRule.grant(
+                    "android.permission.CAMERA",
+                    "android.permission.WRITE_EXTERNAL_STORAGE",


Catima dosen't request write permission, this seems to cause a crash when testing locally:

java.lang.SecurityException: Error granting runtime permission at android.app.UiAutomation.grantRuntimePermissionAsUser(UiAutomation.java:1433) at android.app.UiAutomation.grantRuntimePermission(UiAutomation.java:1399) at androidx.test.runner.permission.UiAutomationPermissionGranter.requestPermissions(UiAutomationPermissionGranter.java:44) at androidx.test.rule.GrantPermissionRule$RequestPermissionStatement.evaluate(GrantPermissionRule.java:149) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at androidx.test.ext.junit.runners.AndroidJUnit4.run(AndroidJUnit4.java:162) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:68) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:59) at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:463) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2402) Caused by: java.lang.SecurityException: Package me.hackerchick.catima.debug has not requested permission android.permission.WRITE_EXTERNAL_STORAGE at android.os.Parcel.createExceptionOrNull(Parcel.java:3057) at android.os.Parcel.createException(Parcel.java:3041) at android.os.Parcel.readException(Parcel.java:3024) at android.os.Parcel.readException(Parcel.java:2966) at android.app.IUiAutomationConnection$Stub$Proxy.grantRuntimePermission(IUiAutomationConnection.java:677) at android.app.UiAutomation.grantRuntimePermissionAsUser(UiAutomation.java:1430) ... 31 more

TheLastProject · 2024-05-25T10:21:09Z

app/src/androidTest/java/protect/card_locker/LoyaltyCardCreationTest.java

+            GrantPermissionRule.grant(
+                    "android.permission.CAMERA",
+                    "android.permission.WRITE_EXTERNAL_STORAGE",
+                    "android.permission.READ_EXTERNAL_STORAGE");


I'm also not sure about READ_EXTERNAL_STORAGE, as that is also only needed up to maxSdkVersion 23. So Android 7 and up will never ask this. This line seems to fail on newer Android verisons too.

Even then, this test would never touch any code that requires this permission, even on Android 5 or 6, so it seems granting this permission should never be necessary.

Same as mentioned above I added them while trying to integrate the Screengrab to create an automated screenshot workflow. Sorry for the inconvenience, I will remove everything unnecessary in the upcoming commits 😅

TheLastProject · 2024-05-25T10:31:19Z

app/src/androidTest/java/protect/card_locker/LoyaltyCardCreationTest.java

+    @Test
+    public void loyaltyCardCreationTest() {
+        ViewInteraction textView = onView(
+                allOf(withId(R.id.add_card_instruction), withText("Click the + plus button to add a card, or import from the ⋮ menu."),


I'm not sure how valuable it is to test the exact strings. It seems pretty safe her to just refer to the noGiftCards string directly. I don't think we need to test if Android will take the correct text string from the resources, I would hope the Android team tests their own core code like context.getString functions :)

I also changed my emulator's device language to Dutch and it seems to crash here now, so that seems to also support this makes the test overly fragile and causes failures under normal conditions.

TheLastProject · 2024-05-25T10:36:00Z

app/src/androidTest/java/protect/card_locker/LanguageChangeUITest.java

+        overflowMenuButton.perform(click());
+
+        ViewInteraction materialTextView = onView(
+                allOf(withId(androidx.recyclerview.R.id.title), withText("Settings"),


This seems to cause a test failure if the device isn't set to English. For the rest of the test it makes sense to check explicit strings as you are changing the language, but the default Catima setting is to use the system language which may not always be English. So it seems this should just use the settings string in strings.xml

app/src/androidTest/java/protect/card_locker/LanguageChangeUITest.java

TheLastProject · 2024-05-25T10:40:13Z

app/build.gradle.kts

@@ -28,6 +28,7 @@ android {
        multiDexEnabled = true

        resourceConfigurations += listOf("ar", "bg", "bn", "bn-rIN", "bs", "cs", "da", "de", "el-rGR", "en", "eo", "es", "es-rAR", "fi", "fr", "he-rIL", "hi", "hr", "hu", "in-rID", "is", "it", "ja", "ko", "lt", "lv", "nb-rNO", "nl", "oc", "pl", "pt-rPT", "ro-rRO", "ru", "sk", "sl", "sv", "tr", "uk", "vi", "zh-rCN", "zh-rTW")
+        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"


I'm thinking this might also need the following change as documented on https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner to ensure that running the "LoyaltyCardCreationTest" twice in a row won't cause failures but I haven't been able to test this yet as the LoyaltyCardCreationTest currently fails for me no matter what 😅

Suggested change

testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"

testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"

testInstrumentationRunnerArguments clearPackageData: 'true'

simosathan9 · 2024-05-30T11:37:45Z

I applied most of the changes proposed above regarding the duplicate dependencies and the way that tests point specific strings. I did not remove the dependency androidx.tracing:tracing:1.2.0 because the tests were failing when this dependency was missing and I also removed the unnecessary permissions android.permission.WRITE_EXTERNAL_STORAGE and android.permission.READ_EXTERNAL_STORAGE . Please let me know if any further changes are needed.

TheLastProject

Looking a lot better! I still noticed a few issues upon review and I can confirm that the androidx:tracing:tracing issue is indeed a bug in AGP and you're using the suggested workaround. Frustrating bug, but well, nothing we can do there :)

TheLastProject · 2024-06-01T08:13:44Z

app/build.gradle.kts

@@ -28,6 +28,8 @@ android {
        multiDexEnabled = true

        resourceConfigurations += listOf("ar", "bg", "bn", "bn-rIN", "bs", "cs", "da", "de", "el-rGR", "en", "eo", "es", "es-rAR", "fi", "fr", "he-rIL", "hi", "hr", "hu", "in-rID", "is", "it", "ja", "ko", "lt", "lv", "nb-rNO", "nl", "oc", "pl", "pt-rPT", "ro-rRO", "ru", "sk", "sl", "sv", "tr", "uk", "vi", "zh-rCN", "zh-rTW")
+        testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
+        testInstrumentationRunnerArguments(mapOf("clearPackageData" to "true"))


This version seems to be deprecated:

'testInstrumentationRunnerArguments(Map<String, String>): Unit' is deprecated. Replaced by testInstrumentationRunnerArguments property

The suggestion I gave you earlier, which I took directly from Google's documentation, is probably the right one (see https://github.com/CatimaLoyalty/Android/pull/1900/files#r1614554689 and https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner):

Suggested change

testInstrumentationRunnerArguments(mapOf("clearPackageData" to "true"))

testInstrumentationRunnerArguments clearPackageData: 'true'

I tried the syntax that you proposed and was also mentioned in the docs but some errors occurred. I think it is because the proposed syntax is in Groovy DSL, while it should be in Kotlin DSL, so I adjust it accordingly.

TheLastProject · 2024-06-01T08:18:03Z

app/build.gradle.kts

@@ -94,6 +95,10 @@ dependencies {
    implementation("androidx.preference:preference:1.2.1")
    implementation("com.google.android.material:material:1.12.0")
    implementation("com.github.yalantis:ucrop:2.2.9")
+    implementation("androidx.tracing:tracing:1.2.0")


I can see that the tests fail if you remove this, it seems to be a dependency of Espresso.

However, given it is an Espresso test dependency and not a runtime dependency, it seems to me this should be an androidTestImplementation instead of implementation so we don't ship it with the actual APKs users will install as it's not needed there.

So that's what I tried... and it didn't work. And then I researched more. And... it seems to be an AGP bug. Sigh: android/android-test#1755 (comment)

I decided to ask yet again for a way to track this issue as the previous request for that got no answer. It's definitely an ugly workaround, but it seems the only option currently: android/android-test#1755 (comment)

TheLastProject · 2024-06-01T08:30:49Z

app/src/androidTest/java/protect/card_locker/LanguageChangeUITest.java

+                        childAtPosition(
+                                withId(android.R.id.list_container),
+                                0)));
+        recyclerView.perform(actionOnItemAtPosition(4, click()));


I just realized this text will break whenever we add a new language. That seems unnecessarily fragile. Can we instead make the test pick the tested language by string name?

Yes indeed adding new languages would make the tests fail. I will make the necessary changes to pick the tested language by string name.

TheLastProject · 2024-06-01T08:40:37Z

app/src/androidTest/java/protect/card_locker/LanguageChangeUITest.java

+                                                0)),
+                                1),
+                        isDisplayed()));
+        appCompatImageButton.perform(click());


The test fails for me here when testing on an Android 14 emulator:

androidx.test.espresso.NoMatchingViewException: No views in hierarchy found matching: (view.getContentDescription() is "Navigate up" and Child at position 1 in parent (view.getId() is <2131296920> and Child at position 0 in parent view.getClass().getName() matches: is "com.google.android.material.appbar.AppBarLayout") and (view has effective visibility <VISIBLE> and view.getGlobalVisibleRect() to return non-empty rectangle))

I've attached the full view-hierarchy:
view-hierarchy-1.txt

This seems weird to me because I am also running the tests on an Android 14 emulator, but I will examine this further.

Changed the way LanguageChangeUITest chooses a new language in order to check that the system has indeed change the language to the specified one.

obfusk · 2024-06-05T20:26:15Z

app/src/androidTest/java/protect/card_locker/LanguageChangeUITest.java

+        int position = 0;
+        for (String locale : availableLocales) {
+            if (locale.equals(selectedLanguage)) {
+                break;
+            }
+            position++;
+        }
+        return position;


Suggested change

int position = 0;

for (String locale : availableLocales) {

if (locale.equals(selectedLanguage)) {

break;

}

position++;

}

return position;

return Arrays.asList(availableLocales).indexOf(selectedLanguage);

Thank you for your suggestion! I've integrated your changes into the PR.

Co-authored-by: FC (Fay) Stegerman <[email protected]>

simosathan9 added 4 commits May 22, 2024 21:56

Add dependencies for espresso tests

72ed403

Language Change UI Test

ed1b5ba

Card Creation UI Test

65f98a6

Merge branch 'CatimaLoyalty:main' into main

81e2103

TheLastProject requested changes May 25, 2024

View reviewed changes

simosathan9 added 2 commits May 30, 2024 12:06

Merge branch 'CatimaLoyalty:main' into main

b49ea56

Apply changes suggested in PR review

150332f

TheLastProject reviewed Jun 1, 2024

View reviewed changes

simosathan9 added 3 commits June 4, 2024 21:45

Update testInstrumentationRunnerArguments in build.gradle.kts

d329c02

Change language selection process on LanguageChangeUITest.

34a588c

Changed the way LanguageChangeUITest chooses a new language in order to check that the system has indeed change the language to the specified one.

Change android.yml to run Espresso tests during CI

99cbace

obfusk reviewed Jun 5, 2024

View reviewed changes

simosathan9 and others added 4 commits June 6, 2024 23:03

Refactor locale position lookup to use Arrays.asList().indexOf()

f3e6bcc

Co-authored-by: FC (Fay) Stegerman <[email protected]>

Add import needed for the proposed refactoring

3831392

Remove Espresso from android.yml

4bc5a72

Merge branch 'CatimaLoyalty:main' into main

8ea7f02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UI Tests using Espresso Testing Framework #1900

Add UI Tests using Espresso Testing Framework #1900

simosathan9 commented May 22, 2024 •

edited

Loading

TheLastProject left a comment

TheLastProject May 25, 2024

simosathan9 May 25, 2024

TheLastProject Jun 1, 2024

TheLastProject May 25, 2024

TheLastProject May 25, 2024

TheLastProject May 25, 2024

simosathan9 May 25, 2024

TheLastProject May 25, 2024

TheLastProject May 25, 2024

TheLastProject May 25, 2024

simosathan9 commented May 30, 2024

TheLastProject left a comment

TheLastProject Jun 1, 2024

simosathan9 Jun 4, 2024

TheLastProject Jun 1, 2024

TheLastProject Jun 1, 2024

simosathan9 Jun 3, 2024

TheLastProject Jun 1, 2024

simosathan9 Jun 3, 2024

obfusk Jun 5, 2024

simosathan9 Jun 6, 2024

	testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
	testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
	testInstrumentationRunnerArguments clearPackageData: 'true'

	testInstrumentationRunnerArguments(mapOf("clearPackageData" to "true"))
	testInstrumentationRunnerArguments clearPackageData: 'true'

Add UI Tests using Espresso Testing Framework #1900

Are you sure you want to change the base?

Add UI Tests using Espresso Testing Framework #1900

Conversation

simosathan9 commented May 22, 2024 • edited Loading

TheLastProject left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simosathan9 commented May 30, 2024

TheLastProject left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simosathan9 commented May 22, 2024 •

edited

Loading