docs(android): update benchmarks section in README

measure-sh · Nov 28, 2024 · ccd7a5f · ccd7a5f
1 parent 980cb79
commit ccd7a5f
Show file tree

Hide file tree

Showing 5 changed files with 88 additions and 81 deletions.
diff --git a/android/docs/internal-documentation.md b/android/docs/internal-documentation.md
@@ -5,43 +5,39 @@
     * [Periodic batching and export](#periodic-batching-and-export)
     * [Exceptions and ANRs export](#exceptions-and-anrs-export)
 * [Thread management](#thread-management)
-* [Configuration](#configuration)
 * [Testing](#testing)
 
 # Storage
 
 Measure primarily uses **SQLite database** to store events. However, it also uses the file system to
-store parts of large events, like exceptions, attachments, etc. Deciding which events are directly
-stored in the database and which are stored in the file system is based on the size of the event.
-Even though SQLite can store large blobs, due to the cursor window size limit on Android, it can
-lead to
-a [TransactionTooLargeException](https://developer.android.com/reference/android/os/TransactionTooLargeException)
-if a query exceeds the limit, and it makes working with large blobs cumbersome.
+store certain large events, like exceptions, attachments, etc to
+avoid [TransactionTooLargeException](https://developer.android.com/reference/android/os/TransactionTooLargeException).
 
-Sqlite database is configured with the following settings:
+SQLite database is configured with the following settings:
 
 * [journal_mode](https://sqlite.org/pragma.html#pragma_journal_mode): WAL
 * [foreign_keys](https://sqlite.org/pragma.html#pragma_foreign_keys): ON
 
-Events are written to the database & file storage (if needed) as soon as they are received by
-the `Event Processor`. This can be improved in future by adding a queue which batches the inserts.
+Events are written to the database & file storage (if needed) as soon as they are received. This can be 
+improved in future by adding a queue which batches the inserts. However, as WAL mode enabled, this optimization
+has been ignored for now.
 
 # Batching & export
 
-Measure exports events to the server in batches. All events for sessions that contain a crash are
-exported. All non-crashed sessions are exported by default, a sampling rate can be applied to
-non-crashed sessions to reduce the number of sessions exported. See [README](../README.md) for 
-more details about configuring sampling rate.
+Events are sent to the server in batches at regular intervals (30s) while the app is in foreground and when the app 
+goes to background.   
 
 * [Periodic batching and export](#periodic-batching-and-export)
 * [Exceptions and ANRs export](#exceptions-and-anrs-export)
 
 ## Periodic batching and export
 
 Measure creates and sends one batch at a time to the server at a regular interval of 30 seconds.
-Batching is done to reduce the number of network calls and to reduce the battery consumption while
+Batching is done to reduce the number of network calls and battery consumption while
 also ensuring that the events are sent to the server without too much delay.
 
+Only **one** batch of events is sent to the server at a time to reduce memory usage and optimize batch size.
+
 The following algorithm is used to periodically batch and send events to the server:
 
 ```mermaid
@@ -54,7 +50,7 @@ flowchart TD
     CL[[Cold Launch]] --> D0
     AB[[App Background]] --> D0
     D1{existing batches available?}
-    D1 -->|Yes\neach batch\nsequentially| Export
+    D1 -->|Yes| Export
     D2{last batch created recently?}
     D1 -->|No| D2
     D2 -->|No| CNB[Create new batch]
@@ -88,10 +84,8 @@ In worse case scenarios the following must be ensured:
 ## Exceptions and ANRs export
 
 All events except for exceptions and ANRs are sent to the server in batches, periodically, as shown
-above. Exceptions
-and ANRs however, are sent to the server immediately as soon as they are received. This is done to
-ensure that clients
-can be notified of issues as soon as possible.
+above. Exceptions and ANRs however, are attempted to be sent as soon as they occur. This is done to
+ensure that clients can be notified of issues as soon as possible.
 
 # Thread management
 
@@ -100,65 +94,47 @@ tasks. This makes it easy to manage the lifecycle of executors and also to provi
 tune the number of threads used for various tasks.
 
 The following executors are used:
+
 1. IO Executor: Used for all long running operations like writing to the database, reading from the database,
    writing to the file system, etc.
 2. Export Executor: Used for exporting events to the server over the network.
-3. Default Executor: Used for short running tasks that need to be run in background like processing 
-events, etc.
+3. Default Executor: Used for short running tasks that need to be run in background like processing
+   events, etc.
 
-All executors are configured to be single-threaded and internally use a scheduled executor service 
+All executors are configured to be single-threaded and internally use a scheduled executor service
 with unbounded queue, which can be tuned in the future.
 
-
-# Configuration
-
-The SDK is configured using the `MeasureConfig` object. The client can pass in a custom config at
-the time of initialization. These configurations can help in enabling/disabling features, preventing
-sensitive information from being sent or modifying the behavior of the SDK.
-
-Any configuration change made to `MeasureConfig` is a public API change and must also result in
-updating the documentation.
-
-See [README](../../docs/android/configuration-options.md) for more details about the 
-available configurations.
-
-## Applying configs
-
-Configs which modify events, like removing fields or decision to drop events are all centralized in
-the `EventTransformer` which makes it easy to keep these modifications in one place instead of
-scattering them throughout the codebase.
-
-However, some configs modify the behavior of collection itself, like `screenshotMaskColor`which
-changes
-the color of the mask applied to the screenshot. These configs are applied at the time of collection
-itself.
-
 # Testing
 
-The SDK is tested using both unit tests and integration tests. Certain unit tests which require 
+The SDK is tested using both unit tests and integration tests. Certain unit tests which require
 Android framework classes are run using Robolectric. The integration tests are run using Espresso
 and UI Automator.
 
 To run unit tests, use the following command:
+
 ```shell
 ./gradlew :measure:test
 ```
 
 To run integration tests (requires a device), use the following command:
+
 ```shell
 ./gradlew :measure:connectedAndroidTest
 ```
 
 The _Measure gradle plugin_ also contains both unit tests and functional tests. The functional tests
-are run using the [testkit by autonomous apps](https://github.com/autonomousapps/dependency-analysis-gradle-plugin/tree/main/testkit) 
+are run using
+the [testkit by autonomous apps](https://github.com/autonomousapps/dependency-analysis-gradle-plugin/tree/main/testkit)
 and use JUnit5 for testing as it provides an easy way to run parameterized tests.
 
 TO run the unit tests, use the following command:
+
 ```shell
 ./gradlew :measure-gradle-plugin:test
 ```
 
 To run the functional tests, use the following command:
+
 ```shell
 ./gradlew :measure-gradle-plugin:functionalTest
 ```
diff --git a/android/gradle/libs.versions.toml b/android/gradle/libs.versions.toml
@@ -1,6 +1,5 @@
 [versions]
 agp = "8.6.0"
-benchmarkJunit4 = "1.2.4"
 bundletool = "1.17.0"
 # android-tools is used for calculating app size and aab size by the gradle plugin.
 # The version should remain compatible with lower versions of android gradle plugin. New versions
@@ -25,7 +24,7 @@ androidx-annotation = "1.7.1"
 androidx-test-rules = "1.5.0"
 androidx-lifecycle = "2.6.2"
 androidx-uiautomator = "2.3.0"
-androidx-benchmark-macro-junit4 = "1.2.3"
+androidx-benchmark-macro-junit4 = "1.3.3"
 androidx-navigation-fragment = "2.7.6"
 androidx-navigation-compose = "2.6.0"
 androidx-activity = "1.9.2"
@@ -56,7 +55,7 @@ mavenPublish = "0.29.0"
 androidx-activity-compose = { module = "androidx.activity:activity-compose", version.ref = "androidx-activity-compose" }
 androidx-annotation = { module = "androidx.annotation:annotation", version.ref = "androidx-annotation" }
 androidx-appcompat = { module = "androidx.appcompat:appcompat", version.ref = "androidx-appcompat" }
-androidx-benchmark-junit4 = { module = "androidx.benchmark:benchmark-junit4", version.ref = "benchmarkJunit4" }
+androidx-benchmark-junit4 = { module = "androidx.benchmark:benchmark-junit4", version.ref = "androidx-benchmark-macro-junit4" }
 androidx-benchmark-macro-junit4 = { module = "androidx.benchmark:benchmark-macro-junit4", version.ref = "androidx-benchmark-macro-junit4" }
 androidx-constraintlayout = { module = "androidx.constraintlayout:constraintlayout", version.ref = "androidx-constraintlayout" }
 androidx-core-ktx = { module = "androidx.core:core-ktx", version.ref = "androidx-core-ktx" }

diff --git a/android/measure/src/main/java/sh/measure/android/Measure.kt b/android/measure/src/main/java/sh/measure/android/Measure.kt
@@ -63,7 +63,12 @@ object Measure {
      */
     fun start() {
         if (isInitialized.get()) {
-            measure.start()
+            InternalTrace.trace(
+                label = { "msr-start" },
+                block = {
+                    measure.start()
+                },
+            )
         }
     }
 
@@ -74,7 +79,12 @@ object Measure {
      */
     fun stop() {
         if (isInitialized.get()) {
-            measure.stop()
+            InternalTrace.trace(
+                label = { "msr-stop" },
+                block = {
+                    measure.stop()
+                },
+            )
         }
     }
 

diff --git a/android/measure/src/main/java/sh/measure/android/events/EventProcessor.kt b/android/measure/src/main/java/sh/measure/android/events/EventProcessor.kt
@@ -175,12 +175,12 @@ internal class EventProcessorImpl(
         sessionId: String?,
         userTriggered: Boolean = false,
     ) {
-        InternalTrace.trace(
-            label = { "msr-track-event" },
-            block = {
-                val threadName = Thread.currentThread().name
-                try {
-                    ioExecutor.submit {
+        val threadName = Thread.currentThread().name
+        try {
+            ioExecutor.submit {
+                InternalTrace.trace(
+                    label = { "msr-track-event" },
+                    block = {
                         val event = createEvent(
                             data = data,
                             timestamp = timestamp,
@@ -208,16 +208,16 @@ internal class EventProcessorImpl(
                         } else {
                             logger.log(LogLevel.Debug, "Event dropped: $type")
                         }
-                    }
-                } catch (e: RejectedExecutionException) {
-                    logger.log(
-                        LogLevel.Error,
-                        "Failed to submit event processing task to executor",
-                        e,
-                    )
-                }
-            },
-        )
+                    },
+                )
+            }
+        } catch (e: RejectedExecutionException) {
+            logger.log(
+                LogLevel.Error,
+                "Failed to submit event processing task to executor",
+                e,
+            )
+        }
     }
 
     private fun <T> onEventTracked(event: Event<T>) {

diff --git a/docs/android/README.md b/docs/android/README.md
@@ -4,10 +4,13 @@
 * [Quick reference](#quick-reference)
 * [Getting started](#getting-started)
 * [Custom events](#custom-events)
-  * [Handled exceptions](#handled-exceptions)
-  * [Navigation](#navigation)
+    * [Handled exceptions](#handled-exceptions)
+    * [Navigation](#navigation)
 * [Features](#features)
-* [Benchmarks](#benchmarks)
+* [Performance Impact](#performance-impact)
+    * [Benchmarks](#benchmarks)
+    * [Profiling](#profiling)
+    * [Implementation](#implementation)
 
 # Minimum requirements
 
@@ -23,7 +26,6 @@ A quick reference to the entire public API for Measure Android SDK.
 
 <img src="https://github.com/user-attachments/assets/ff835b3b-2953-4920-b5fa-060761862cac" width="60%" alt="Cheatsheet">
 
-
 # Getting started
 
 Once you have access to the dashboard, create a new app and follow the steps below:
@@ -216,6 +218,7 @@ after the SDK is initialized:
 ```kotlin
 throw RuntimeException("This is a test crash")
 ```
+
 Reopen the app and launch the dashboard, you should see the crash report in the dashboard.
 
 > [!CAUTION]
@@ -283,22 +286,41 @@ or when there's been no activity for a 20-minute period. A single session can co
 foreground events; brief interruptions will not cause a new session to be created. This approach is helpful when reviewing 
 session replays, as it shows the app switching between background and foreground states within the same session.
 
-# Benchmarks
 
-Measure SDK has a set of benchmarks to measure the performance impact of the SDK on the app.
-These benchmarks are collected using macro-benchmark on a Pixel 4a device running Android 13 (API 33).
-Each benchmark is run 35 times. See the [android/benchmarks](../../android/benchmarks/README.md) for
-more details, and the raw results are available in the 
-[android/benchmarkData](../../android/benchmarks/README.md) folder.
+# Performance Impact
+
+## Benchmarks
+
+We benchmark the SDK's performance impact using a Pixel 4a running Android 13 (API 33). Each test runs 35 times using 
+macro-benchmark. For detailed methodology, see [android/benchmarks](../../android/benchmarks/README.md).
 
 > [!IMPORTANT]
 > Benchmark results are specific to the device and the app. It is recommended to run the benchmarks
 > for your app to get results specific to your app. These numbers are published to provide
 > a reference point and are used internally to detect any performance regressions.
 
-For v0.7.0, the following benchmarks are available.
+Benchmarks results for v0.7.0:
 
 * Adds 26.258ms-34.416ms to the app startup time (Time to Initial Display) for a simple app.
 * Takes 0.30ms to find the target view for every click/scroll gesture in a deep view hierarchy.
 * Takes 0.45ms to find the target composable for every click/scroll gesture in a deep composable
   hierarchy.
+
+## Profiling
+
+To measure the SDK's impact on your app, we've added traces to key areas of the code. These traces help you track 
+performance using [Macro Benchmark](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview) 
+or by using [Perfetto](https://perfetto.dev/docs/quickstart/android-tracing) directly.
+
+* `msr-init` — time spent on main thread while initializing.
+* `msr-start` — time spent on main thread when `Measure.start` is called.
+* `msr-stop` — — time spent on main thread when `Measure.stop` is called.
+* `msr-track-event` — time spent in storing an event to local storage. Almost all of this time is spent _off_ the main
+  thread. 
+* `msr-click-getTarget` — time spent on main thread to find out the clicked view.
+* `msr-scroll-getTarget` — time spent on main thread to find out the scrolled view.
+
+## Implementation
+
+For details on data storage, syncing behavior, and threading, see
+our [Internal Documentation](../../android/docs/internal-documentation.md).