From 23fc38e60670659eaf28379fca4f2116f7a7adf5 Mon Sep 17 00:00:00 2001 From: Abhay Sood Date: Thu, 28 Nov 2024 07:36:25 +0530 Subject: [PATCH] docs(android): update benchmarks & related information in README --- android/docs/internal-documentation.md | 83 +++++++------------ android/gradle/libs.versions.toml | 5 +- .../main/java/sh/measure/android/Measure.kt | 14 +++- .../measure/android/events/EventProcessor.kt | 32 +++---- docs/android/README.md | 44 +++++++--- 5 files changed, 95 insertions(+), 83 deletions(-) diff --git a/android/docs/internal-documentation.md b/android/docs/internal-documentation.md index e051ea5d3..457563712 100644 --- a/android/docs/internal-documentation.md +++ b/android/docs/internal-documentation.md @@ -5,33 +5,27 @@ * [Periodic batching and export](#periodic-batching-and-export) * [Exceptions and ANRs export](#exceptions-and-anrs-export) * [Thread management](#thread-management) -* [Configuration](#configuration) * [Testing](#testing) # Storage Measure primarily uses **SQLite database** to store events. However, it also uses the file system to -store parts of large events, like exceptions, attachments, etc. Deciding which events are directly -stored in the database and which are stored in the file system is based on the size of the event. -Even though SQLite can store large blobs, due to the cursor window size limit on Android, it can -lead to -a [TransactionTooLargeException](https://developer.android.com/reference/android/os/TransactionTooLargeException) -if a query exceeds the limit, and it makes working with large blobs cumbersome. +store certain large events, like exceptions, attachments, etc to +avoid [TransactionTooLargeException](https://developer.android.com/reference/android/os/TransactionTooLargeException). -Sqlite database is configured with the following settings: +SQLite database is configured with the following settings: * [journal_mode](https://sqlite.org/pragma.html#pragma_journal_mode): WAL * [foreign_keys](https://sqlite.org/pragma.html#pragma_foreign_keys): ON -Events are written to the database & file storage (if needed) as soon as they are received by -the `Event Processor`. This can be improved in future by adding a queue which batches the inserts. +Events are written to the database & file storage (if needed) as soon as they are received. This can be +improved in future by adding a queue which batches the inserts. However, as WAL mode enabled, this optimization +has been ignored for now. # Batching & export -Measure exports events to the server in batches. All events for sessions that contain a crash are -exported. All non-crashed sessions are exported by default, a sampling rate can be applied to -non-crashed sessions to reduce the number of sessions exported. See [README](../README.md) for -more details about configuring sampling rate. +Events are sent to the server in batches at regular intervals (30s) while the app is in foreground and when the app +goes to background. * [Periodic batching and export](#periodic-batching-and-export) * [Exceptions and ANRs export](#exceptions-and-anrs-export) @@ -39,9 +33,11 @@ more details about configuring sampling rate. ## Periodic batching and export Measure creates and sends one batch at a time to the server at a regular interval of 30 seconds. -Batching is done to reduce the number of network calls and to reduce the battery consumption while +Batching is done to reduce the number of network calls and battery consumption while also ensuring that the events are sent to the server without too much delay. +Only **one** batch of events is sent to the server at a time to reduce memory usage and optimize batch size. + The following algorithm is used to periodically batch and send events to the server: ```mermaid @@ -51,15 +47,20 @@ flowchart TD D0 -->|No| D1 D0 -->|Yes| Skip P[[Pulse]] -->|every 30 secs| D0 - CL[[Cold Launch]] --> D0 + AF[[App Foreground]] --> D0 AB[[App Background]] --> D0 D1{existing batches available?} - D1 -->|Yes\neach batch\nsequentially| Export + D1 -->|Yes| Export D2{last batch created recently?} D1 -->|No| D2 D2 -->|No| CNB[Create new batch] - CNB --> Export((Export)) + CNB --> Export((Export batch)) D2 -->|Yes| Skip((Skip)) + + classDef skip fill:#003300,stroke:#333,stroke-width:2px,color:white + classDef export fill:#003300,stroke:#333,stroke-width:2px,color:white + class Skip skip + class Export export ``` Following considerations were in mind when designing the algorithm: @@ -88,10 +89,8 @@ In worse case scenarios the following must be ensured: ## Exceptions and ANRs export All events except for exceptions and ANRs are sent to the server in batches, periodically, as shown -above. Exceptions -and ANRs however, are sent to the server immediately as soon as they are received. This is done to -ensure that clients -can be notified of issues as soon as possible. +above. Exceptions and ANRs however, are attempted to be sent as soon as they occur. This is done to +ensure that clients can be notified of issues as soon as possible. # Thread management @@ -100,65 +99,47 @@ tasks. This makes it easy to manage the lifecycle of executors and also to provi tune the number of threads used for various tasks. The following executors are used: + 1. IO Executor: Used for all long running operations like writing to the database, reading from the database, writing to the file system, etc. 2. Export Executor: Used for exporting events to the server over the network. -3. Default Executor: Used for short running tasks that need to be run in background like processing -events, etc. +3. Default Executor: Used for short running tasks that need to be run in background like processing + events, etc. -All executors are configured to be single-threaded and internally use a scheduled executor service +All executors are configured to be single-threaded and internally use a scheduled executor service with unbounded queue, which can be tuned in the future. - -# Configuration - -The SDK is configured using the `MeasureConfig` object. The client can pass in a custom config at -the time of initialization. These configurations can help in enabling/disabling features, preventing -sensitive information from being sent or modifying the behavior of the SDK. - -Any configuration change made to `MeasureConfig` is a public API change and must also result in -updating the documentation. - -See [README](../../docs/android/configuration-options.md) for more details about the -available configurations. - -## Applying configs - -Configs which modify events, like removing fields or decision to drop events are all centralized in -the `EventTransformer` which makes it easy to keep these modifications in one place instead of -scattering them throughout the codebase. - -However, some configs modify the behavior of collection itself, like `screenshotMaskColor`which -changes -the color of the mask applied to the screenshot. These configs are applied at the time of collection -itself. - # Testing -The SDK is tested using both unit tests and integration tests. Certain unit tests which require +The SDK is tested using both unit tests and integration tests. Certain unit tests which require Android framework classes are run using Robolectric. The integration tests are run using Espresso and UI Automator. To run unit tests, use the following command: + ```shell ./gradlew :measure:test ``` To run integration tests (requires a device), use the following command: + ```shell ./gradlew :measure:connectedAndroidTest ``` The _Measure gradle plugin_ also contains both unit tests and functional tests. The functional tests -are run using the [testkit by autonomous apps](https://github.com/autonomousapps/dependency-analysis-gradle-plugin/tree/main/testkit) +are run using +the [testkit by autonomous apps](https://github.com/autonomousapps/dependency-analysis-gradle-plugin/tree/main/testkit) and use JUnit5 for testing as it provides an easy way to run parameterized tests. TO run the unit tests, use the following command: + ```shell ./gradlew :measure-gradle-plugin:test ``` To run the functional tests, use the following command: + ```shell ./gradlew :measure-gradle-plugin:functionalTest ``` diff --git a/android/gradle/libs.versions.toml b/android/gradle/libs.versions.toml index ae4acaccb..33c2e8091 100644 --- a/android/gradle/libs.versions.toml +++ b/android/gradle/libs.versions.toml @@ -1,6 +1,5 @@ [versions] agp = "8.6.0" -benchmarkJunit4 = "1.2.4" bundletool = "1.17.0" # android-tools is used for calculating app size and aab size by the gradle plugin. # The version should remain compatible with lower versions of android gradle plugin. New versions @@ -25,7 +24,7 @@ androidx-annotation = "1.7.1" androidx-test-rules = "1.5.0" androidx-lifecycle = "2.6.2" androidx-uiautomator = "2.3.0" -androidx-benchmark-macro-junit4 = "1.2.3" +androidx-benchmark-macro-junit4 = "1.3.3" androidx-navigation-fragment = "2.7.6" androidx-navigation-compose = "2.6.0" androidx-activity = "1.9.2" @@ -56,7 +55,7 @@ mavenPublish = "0.29.0" androidx-activity-compose = { module = "androidx.activity:activity-compose", version.ref = "androidx-activity-compose" } androidx-annotation = { module = "androidx.annotation:annotation", version.ref = "androidx-annotation" } androidx-appcompat = { module = "androidx.appcompat:appcompat", version.ref = "androidx-appcompat" } -androidx-benchmark-junit4 = { module = "androidx.benchmark:benchmark-junit4", version.ref = "benchmarkJunit4" } +androidx-benchmark-junit4 = { module = "androidx.benchmark:benchmark-junit4", version.ref = "androidx-benchmark-macro-junit4" } androidx-benchmark-macro-junit4 = { module = "androidx.benchmark:benchmark-macro-junit4", version.ref = "androidx-benchmark-macro-junit4" } androidx-constraintlayout = { module = "androidx.constraintlayout:constraintlayout", version.ref = "androidx-constraintlayout" } androidx-core-ktx = { module = "androidx.core:core-ktx", version.ref = "androidx-core-ktx" } diff --git a/android/measure/src/main/java/sh/measure/android/Measure.kt b/android/measure/src/main/java/sh/measure/android/Measure.kt index 23dc3823f..abba052c9 100644 --- a/android/measure/src/main/java/sh/measure/android/Measure.kt +++ b/android/measure/src/main/java/sh/measure/android/Measure.kt @@ -63,7 +63,12 @@ object Measure { */ fun start() { if (isInitialized.get()) { - measure.start() + InternalTrace.trace( + label = { "msr-start" }, + block = { + measure.start() + }, + ) } } @@ -74,7 +79,12 @@ object Measure { */ fun stop() { if (isInitialized.get()) { - measure.stop() + InternalTrace.trace( + label = { "msr-stop" }, + block = { + measure.stop() + }, + ) } } diff --git a/android/measure/src/main/java/sh/measure/android/events/EventProcessor.kt b/android/measure/src/main/java/sh/measure/android/events/EventProcessor.kt index 7fe369a12..81580be18 100644 --- a/android/measure/src/main/java/sh/measure/android/events/EventProcessor.kt +++ b/android/measure/src/main/java/sh/measure/android/events/EventProcessor.kt @@ -175,12 +175,12 @@ internal class EventProcessorImpl( sessionId: String?, userTriggered: Boolean = false, ) { - InternalTrace.trace( - label = { "msr-track-event" }, - block = { - val threadName = Thread.currentThread().name - try { - ioExecutor.submit { + val threadName = Thread.currentThread().name + try { + ioExecutor.submit { + InternalTrace.trace( + label = { "msr-track-event" }, + block = { val event = createEvent( data = data, timestamp = timestamp, @@ -208,16 +208,16 @@ internal class EventProcessorImpl( } else { logger.log(LogLevel.Debug, "Event dropped: $type") } - } - } catch (e: RejectedExecutionException) { - logger.log( - LogLevel.Error, - "Failed to submit event processing task to executor", - e, - ) - } - }, - ) + }, + ) + } + } catch (e: RejectedExecutionException) { + logger.log( + LogLevel.Error, + "Failed to submit event processing task to executor", + e, + ) + } } private fun onEventTracked(event: Event) { diff --git a/docs/android/README.md b/docs/android/README.md index 37a9773a4..aed42ec6a 100644 --- a/docs/android/README.md +++ b/docs/android/README.md @@ -4,10 +4,13 @@ * [Quick reference](#quick-reference) * [Getting started](#getting-started) * [Custom events](#custom-events) - * [Handled exceptions](#handled-exceptions) - * [Navigation](#navigation) + * [Handled exceptions](#handled-exceptions) + * [Navigation](#navigation) * [Features](#features) -* [Benchmarks](#benchmarks) +* [Performance Impact](#performance-impact) + * [Benchmarks](#benchmarks) + * [Profiling](#profiling) + * [Implementation](#implementation) # Minimum requirements @@ -23,7 +26,6 @@ A quick reference to the entire public API for Measure Android SDK. Cheatsheet - # Getting started Once you have access to the dashboard, create a new app and follow the steps below: @@ -216,6 +218,7 @@ after the SDK is initialized: ```kotlin throw RuntimeException("This is a test crash") ``` + Reopen the app and launch the dashboard, you should see the crash report in the dashboard. > [!CAUTION] @@ -283,22 +286,41 @@ or when there's been no activity for a 20-minute period. A single session can co foreground events; brief interruptions will not cause a new session to be created. This approach is helpful when reviewing session replays, as it shows the app switching between background and foreground states within the same session. -# Benchmarks -Measure SDK has a set of benchmarks to measure the performance impact of the SDK on the app. -These benchmarks are collected using macro-benchmark on a Pixel 4a device running Android 13 (API 33). -Each benchmark is run 35 times. See the [android/benchmarks](../../android/benchmarks/README.md) for -more details, and the raw results are available in the -[android/benchmarkData](../../android/benchmarks/README.md) folder. +# Performance Impact + +## Benchmarks + +We benchmark the SDK's performance impact using a Pixel 4a running Android 13 (API 33). Each test runs 35 times using +macro-benchmark. For detailed methodology, see [android/benchmarks](../../android/benchmarks/README.md). > [!IMPORTANT] > Benchmark results are specific to the device and the app. It is recommended to run the benchmarks > for your app to get results specific to your app. These numbers are published to provide > a reference point and are used internally to detect any performance regressions. -For v0.7.0, the following benchmarks are available. +Benchmarks results for v0.7.0: * Adds 26.258ms-34.416ms to the app startup time (Time to Initial Display) for a simple app. * Takes 0.30ms to find the target view for every click/scroll gesture in a deep view hierarchy. * Takes 0.45ms to find the target composable for every click/scroll gesture in a deep composable hierarchy. + +## Profiling + +To measure the SDK's impact on your app, we've added traces to key areas of the code. These traces help you track +performance using [Macro Benchmark](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview) +or by using [Perfetto](https://perfetto.dev/docs/quickstart/android-tracing) directly. + +* `msr-init` — time spent on main thread while initializing. +* `msr-start` — time spent on main thread when `Measure.start` is called. +* `msr-stop` — — time spent on main thread when `Measure.stop` is called. +* `msr-track-event` — time spent in storing an event to local storage. Almost all of this time is spent _off_ the main + thread. +* `msr-click-getTarget` — time spent on main thread to find out the clicked view. +* `msr-scroll-getTarget` — time spent on main thread to find out the scrolled view. + +## Implementation + +For details on data storage, syncing behavior, and threading, see +our [Internal Documentation](../../android/docs/internal-documentation.md).