-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #57 from willarmiros/performance
Added Performance tests
- Loading branch information
Showing
26 changed files
with
1,153 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| ## AWS X-Ray Agent Benchmarks | ||
|
|
||
| This package contains benchmarking tests to compare the performance of common scenarios in a distributed application. | ||
| The scenarios can be run with instrumentation by the AWS X-Ray Auto-Instrumentation Agent, the AWS X-Ray SDK for Java, | ||
| and no instrumentation at all as a baseline. A local server is used to mitigate unpredictable network latencies. | ||
|
|
||
| ## Running the Benchmarks | ||
|
|
||
| The officially recorded benchmark results are run on an m5.xlarge Amazon EC2 instance with 4 vCPUs and 16 GB | ||
| of memory running Amazon Linux 2. It is recommended you run them in a similar environment for comparable results. | ||
|
|
||
| The benchmarks can be run in 3 modes: `agent`, `SDK`, or `normal`. When run in `normal` mode, the benchmark tests are ran without | ||
| any instrumentation as a baseline. When run in `SDK` mode, the benchmark tests are ran with instrumentation by the AWS X-Ray SDK. | ||
| When run in `agent` mode, the benchmark tests are ran with the X-Ray Agent included on the benchmarking JVM (these can take | ||
| quite a while to complete because the agent must re-initialize before each benchmark trial). | ||
|
|
||
| To run in Agent mode, run from the root of this repo: | ||
|
|
||
| ```shell script | ||
| ./gradlew clean jmh -Pagent | ||
| ``` | ||
|
|
||
| To run in SDK mode, run from the root of this repo: | ||
|
|
||
| ```shell script | ||
| ./gradlew clean jmh -Psdk | ||
| ``` | ||
|
|
||
| To run in normal mode, run from the root of this repo: | ||
|
|
||
| ```shell script | ||
| ./gradlew clean jmh | ||
| ``` | ||
|
|
||
| The results will be output into the `build/reports/jmh` directory after the tests are completed. | ||
|
|
||
| ## Benchmark Results | ||
|
|
||
| The below table summarizes the **approximate** latency added using both manual X-Ray SDK and automatic | ||
| X-Ray agent-based instrumentation for different activities in microseconds. These numbers should not be | ||
| considered precisely accurate in all applications, they are just to give an idea of added latency. | ||
| Overall, you can see that in a typical application the SDK or the agent would not add more than a single-digit | ||
| amount of milliseconds to a request. | ||
|
|
||
| | | SDK Instrumentation | Auto Instrumentation | | ||
| |--------------------------|---------------------|----------------------| | ||
| | Service incoming request | +100 us | +110 us | | ||
| | SQL Query | +70 us | +60 us | | ||
| | AWS SDK V1 Request | +30 us | +60 us | | ||
| | AWS SDK V2 Request | +30 us | +90 us | | ||
| | Apache HTTP Request | +20 us | +40 us | | ||
|
|
||
| You can also take a look the [results directory](https://github.com/aws/aws-xray-java-agent/tree/master/aws-xray-agent-benchmark/results) | ||
| for detailed benchmarking data from previous versions of the agent. Since we use an arbitrary delay to simulate the effect | ||
| of a network, the *absolute* values of each individual benchmark are not particularly meaningful. The results are more | ||
| useful when analyzed in context, that is comparing the times of the same scenario in all 3 modes, and noticing | ||
| the difference between those times. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| plugins { | ||
| id("me.champeau.gradle.jmh") version "0.5.0" | ||
| } | ||
|
|
||
| val JMH_VERSION = "1.23" | ||
|
|
||
| sourceSets { | ||
| named("jmh") { | ||
| java { | ||
| srcDir("src/jmh/java") | ||
|
|
||
| } | ||
| } | ||
| } | ||
|
|
||
| dependencies { | ||
| jmh("com.amazonaws:aws-xray-recorder-sdk-core") | ||
| jmh("com.amazonaws:aws-xray-recorder-sdk-sql") | ||
| jmh("com.amazonaws:aws-xray-recorder-sdk-aws-sdk") { | ||
| isTransitive = false // Don't bring in all clients | ||
| } | ||
| jmh("com.amazonaws:aws-java-sdk-dynamodb:1.11.837") | ||
| jmh("com.amazonaws:aws-xray-recorder-sdk-aws-sdk-v2") | ||
| jmh("software.amazon.awssdk:dynamodb:2.13.76") | ||
| jmh("com.amazonaws:aws-xray-recorder-sdk-apache-http") | ||
|
|
||
| jmh("org.apache.httpcomponents:httpclient:4.5.12") | ||
| jmh("javax.servlet:javax.servlet-api:4.0.1") | ||
| jmh("org.eclipse.jetty:jetty-server:9.4.1.v20170120") | ||
| jmh("org.eclipse.jetty:jetty-servlet:9.4.1.v20170120") | ||
|
|
||
| jmh("org.mockito:mockito-core:2.28.2") | ||
| jmh("org.openjdk.jmh:jmh-generator-annprocess:${JMH_VERSION}") | ||
| jmhAnnotationProcessor("org.openjdk.jmh:jmh-generator-annprocess:${JMH_VERSION}") | ||
|
|
||
| jmh(platform("com.amazonaws:aws-xray-recorder-sdk-bom:${version}")) | ||
| } | ||
|
|
||
| // JMH plugin docs: https://github.com/melix/jmh-gradle-plugin | ||
| jmh { | ||
| benchmarkMode = listOf("thrpt", "sample") | ||
| fork = 1 | ||
| timeUnit = "ms" | ||
|
|
||
| warmupIterations = 10 | ||
| warmup = "1s" | ||
|
|
||
| iterations = 5 | ||
| timeOnIteration = "1s" | ||
|
|
||
| if (project.hasProperty("agent")) { | ||
| val discoPath = "${rootProject.projectDir}/aws-xray-agent-plugin/build/libs/disco" | ||
| jvmArgs = listOf("-javaagent:$discoPath/disco-java-agent.jar=pluginPath=$discoPath/disco-plugins", | ||
| "-Dcom.amazonaws.xray.strategy.tracingName=Benchmark") | ||
|
|
||
| resultsFile = project.file("$buildDir/reports/jmh/auto-instrumentation.txt") | ||
| } else if (project.hasProperty("sdk")) { | ||
| jvmArgs = listOf("-Dcom.amazonaws.xray.sdk=true") // Propagate the system property into testing environment to use appropriate clients | ||
| resultsFile = project.file("$buildDir/reports/jmh/sdk-instrumentation.txt") | ||
| } else { | ||
| resultsFile = project.file("$buildDir/reports/jmh/no-instrumentation.txt") | ||
| } | ||
|
|
||
| duplicateClassesStrategy = DuplicatesStrategy.EXCLUDE | ||
| isZip64 = true | ||
| } | ||
|
|
||
| // Need to have the disco directory assembled to run benchmarks with X-Ray agent | ||
| tasks.jmh { | ||
| if (project.hasProperty("agent")) { | ||
| dependsOn(":aws-xray-agent-plugin:build") | ||
| } | ||
| } |
51 changes: 51 additions & 0 deletions
51
aws-xray-agent-benchmark/results/2.7.x/jmh/auto-instrumentation.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| Benchmark Mode Cnt Score Error Units | ||
| AwsSdkBenchmark.awsV1Request thrpt 5 0.416 ± 0.007 ops/ms | ||
| AwsSdkBenchmark.awsV2Request thrpt 5 0.402 ± 0.015 ops/ms | ||
| HttpDownstreamBenchmark.makeHttpRequest thrpt 5 0.419 ± 0.012 ops/ms | ||
| ServletBenchmark.serviceRequest thrpt 5 0.460 ± 0.001 ops/ms | ||
| SqlBenchmark.sqlQuery thrpt 5 0.469 ± 0.003 ops/ms | ||
| AwsSdkBenchmark.awsV1Request sample 2069 2.417 ± 0.004 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.00 sample 2.339 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.50 sample 2.408 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.90 sample 2.454 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.95 sample 2.494 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.99 sample 2.572 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.999 sample 2.736 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.9999 sample 4.211 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p1.00 sample 4.211 ms/op | ||
| AwsSdkBenchmark.awsV2Request sample 2011 2.486 ± 0.005 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.00 sample 2.400 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.50 sample 2.474 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.90 sample 2.552 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.95 sample 2.580 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.99 sample 2.682 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.999 sample 2.863 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.9999 sample 4.588 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p1.00 sample 4.588 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest sample 2098 2.383 ± 0.020 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.00 sample 2.277 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.50 sample 2.367 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.90 sample 2.413 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.95 sample 2.433 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.99 sample 2.490 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.999 sample 9.244 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.9999 sample 10.207 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p1.00 sample 10.207 ms/op | ||
| ServletBenchmark.serviceRequest sample 2277 2.196 ± 0.064 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.00 sample 2.163 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.50 sample 2.171 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.90 sample 2.183 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.95 sample 2.208 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.99 sample 2.295 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.999 sample 2.358 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.9999 sample 46.596 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p1.00 sample 46.596 ms/op | ||
| SqlBenchmark.sqlQuery sample 2350 2.129 ± 0.001 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.00 sample 2.101 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.50 sample 2.126 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.90 sample 2.134 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.95 sample 2.142 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.99 sample 2.220 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.999 sample 2.270 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.9999 sample 2.310 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p1.00 sample 2.310 ms/op |
51 changes: 51 additions & 0 deletions
51
aws-xray-agent-benchmark/results/2.7.x/jmh/no-instrumentation.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| Benchmark Mode Cnt Score Error Units | ||
| AwsSdkBenchmark.awsV1Request thrpt 5 0.426 ± 0.008 ops/ms | ||
| AwsSdkBenchmark.awsV2Request thrpt 5 0.419 ± 0.008 ops/ms | ||
| HttpDownstreamBenchmark.makeHttpRequest thrpt 5 0.429 ± 0.009 ops/ms | ||
| ServletBenchmark.serviceRequest thrpt 5 0.482 ± 0.011 ops/ms | ||
| SqlBenchmark.sqlQuery thrpt 5 0.484 ± 0.001 ops/ms | ||
| AwsSdkBenchmark.awsV1Request sample 2132 2.345 ± 0.007 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.00 sample 2.265 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.50 sample 2.339 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.90 sample 2.380 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.95 sample 2.414 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.99 sample 2.488 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.999 sample 3.999 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p0.9999 sample 5.964 ms/op | ||
| AwsSdkBenchmark.awsV1Request:awsV1Request·p1.00 sample 5.964 ms/op | ||
| AwsSdkBenchmark.awsV2Request sample 2099 2.383 ± 0.008 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.00 sample 2.294 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.50 sample 2.376 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.90 sample 2.421 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.95 sample 2.445 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.99 sample 2.540 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.999 sample 2.944 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p0.9999 sample 7.135 ms/op | ||
| AwsSdkBenchmark.awsV2Request:awsV2Request·p1.00 sample 7.135 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest sample 2140 2.335 ± 0.024 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.00 sample 2.253 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.50 sample 2.306 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.90 sample 2.339 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.95 sample 2.359 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.99 sample 2.608 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.999 sample 9.664 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p0.9999 sample 10.715 ms/op | ||
| HttpDownstreamBenchmark.makeHttpRequest:makeHttpRequest·p1.00 sample 10.715 ms/op | ||
| ServletBenchmark.serviceRequest sample 2419 2.068 ± 0.001 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.00 sample 2.023 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.50 sample 2.066 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.90 sample 2.073 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.95 sample 2.083 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.99 sample 2.101 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.999 sample 2.123 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p0.9999 sample 2.310 ms/op | ||
| ServletBenchmark.serviceRequest:serviceRequest·p1.00 sample 2.310 ms/op | ||
| SqlBenchmark.sqlQuery sample 2424 2.063 ± 0.001 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.00 sample 2.025 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.50 sample 2.062 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.90 sample 2.066 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.95 sample 2.073 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.99 sample 2.087 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.999 sample 2.128 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p0.9999 sample 2.142 ms/op | ||
| SqlBenchmark.sqlQuery:sqlQuery·p1.00 sample 2.142 ms/op |
Oops, something went wrong.