End to End testing

Overview

End-to-End tests test the app from an end user’s experience by simulating the real user scenario and validating the system under test and its components for integration and data integrity.

These tests play a major role in publishing the app. It gives confidence in the final application or a feature when it's finished and hence these tests should run on a real device or a virtual device to make sure that our code interacts with the Android environment as expected.

End-to-End tests in Oppia-android are written using UiAutomator. These tests are written under the instrumentation module and don’t have Gradle support.

instrumentation/ -- android test binaries for each test suite.
`-- src -- AndroidManifest.xml to use the TestApplication from the java sourceset
    |-- java
    |   `-- org
    |       `-- oppia
    |           `-- android
    |               `-- instrumentation
    |                   `-- application -- Test application and modules
    `-- javatests
        `-- org
            `-- oppia
                `-- android
                    `-- instrumentation -- Test suites for each part of the app

These tests are run using Bazel and ADB. Each test suite tests a particular part of the app, each Testsuite has its kt_android_library, android_binary and android_instrumentation_test.

Note: android_instrumentation_test target is not supported yet (#3617 for details).

How it works.

The android_binary of a test suite generates a test apk with the same name as the class of the Test suite. his test apk is installed along with the original apk in the emulator. Now to run the test we use the adb am instrument command with this the adb shell opens the device or emulator, runs the tests and gives the test results.

How to run an End-to-End test

Prerequisites:

1. Setup Bazel for Oppia.
2. Add adb to the environment (platform-tools) i.e, add the following line to the .bashrc or the file path according to the Operating System.

   export PATH=/home/<username>/Android/Sdk/platform-tools:$PATH
   

3. Download and install test-services-1.1.0.apk and orchestrator-1.1.0.apk in emulator.

   adb install -r test-services-1.1.0.apk && adb install -r orchestrator-1.1.0.apk
   

4. java version 8 (Optional, only for uiautomatorviewer)

   java -version
   

   output: openjdk version "1.8.0_292" OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10) OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

Steps to run the tests

1. Build the BaseTest android_binary from the instrumentation module

   bazel build :oppia_test && bazel build //instrumentation/src/javatests/org/oppia/android/instrumentation/player:ExplorationPlayerTestBinary
   

2. install the oppia_test.apk and the ExplorationPlayerTest.apk

   adb install -r bazel-bin/oppia_test.apk && adb install -r bazel-bin/instrumentation/src/javatests/org/oppia/android/instrumentation/player/ExplorationPlayerTestBinary.apk
   

3. Run the instrumentation tests using am instrument command

   adb shell 'CLASSPATH=$(pm path androidx.test.services) app_process / \
   androidx.test.services.shellexecutor.ShellMain am instrument -w -e clearPackageData true \
   -e targetInstrumentation org.oppia.android.app.instrumentation/androidx.test.runner.AndroidJUnitRunner \
   androidx.test.orchestrator/.AndroidTestOrchestrator'
   

How to create a new test suite

Each test suite uses the macro oppia_instrumentation_test which generates the necessary targets required for each test suite.