Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add perf metrics for 2.40.0 #48915

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add perf metrics for 2.40.0 #48915

wants to merge 1 commit into from

Conversation

khluu
Copy link
Contributor

@khluu khluu commented Nov 24, 2024 

REGRESSION 11.06%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 12.047828247398513 to 10.715440745783182 in microbenchmark.json
REGRESSION 9.50%: 1_1_async_actor_calls_async (THROUGHPUT) regresses from 5128.909004080372 to 4641.857377193749 in microbenchmark.json
REGRESSION 9.33%: placement_group_create/removal (THROUGHPUT) regresses from 845.3611380520398 to 766.4710772352788 in microbenchmark.json
REGRESSION 9.14%: single_client_tasks_async (THROUGHPUT) regresses from 7850.804688217522 to 7133.343594327644 in microbenchmark.json
REGRESSION 8.65%: 1_1_async_actor_calls_with_args_async (THROUGHPUT) regresses from 3278.2411157746983 to 2994.804100896146 in microbenchmark.json
REGRESSION 7.64%: client__put_calls (THROUGHPUT) regresses from 862.5855312761047 to 796.6672485266845 in microbenchmark.json
REGRESSION 7.48%: 1_n_async_actor_calls_async (THROUGHPUT) regresses from 7853.325955437287 to 7265.632549025338 in microbenchmark.json
REGRESSION 6.69%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 7.780451969433976 to 7.259983498667911 in microbenchmark.json
REGRESSION 5.86%: client__get_calls (THROUGHPUT) regresses from 1066.7684719081053 to 1004.2954376391058 in microbenchmark.json
REGRESSION 5.41%: client__1_1_actor_calls_async (THROUGHPUT) regresses from 980.7790835269525 to 927.7378599556337 in microbenchmark.json
REGRESSION 4.85%: single_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 5122.443353616408 to 4873.791178055619 in microbenchmark.json
REGRESSION 4.69%: single_client_put_gigabytes (THROUGHPUT) regresses from 17.176054266128194 to 16.370163798832593 in microbenchmark.json
REGRESSION 4.58%: 1_1_async_actor_calls_sync (THROUGHPUT) regresses from 1541.090559096205 to 1470.5838167142256 in microbenchmark.json
REGRESSION 4.41%: 1_1_actor_calls_concurrent (THROUGHPUT) regresses from 5596.612088571596 to 5349.871656557709 in microbenchmark.json
REGRESSION 3.78%: multi_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 16647.860352850137 to 16018.142349512491 in microbenchmark.json
REGRESSION 3.41%: 1_n_actor_calls_async (THROUGHPUT) regresses from 8405.676715400055 to 8118.8880400901135 in microbenchmark.json
REGRESSION 3.22%: n_n_actor_calls_async (THROUGHPUT) regresses from 26933.018765424662 to 26065.409129685446 in microbenchmark.json
REGRESSION 3.07%: client__tasks_and_put_batch (THROUGHPUT) regresses from 14405.107490454913 to 13963.575440510822 in microbenchmark.json
REGRESSION 2.95%: actors_per_second (THROUGHPUT) regresses from 591.3539212974848 to 573.9193627849573 in benchmarks/many_actors.json
REGRESSION 2.62%: single_client_tasks_sync (THROUGHPUT) regresses from 1001.5048485799107 to 975.2895608656044 in microbenchmark.json
REGRESSION 2.56%: 1_1_actor_calls_async (THROUGHPUT) regresses from 8898.74161889686 to 8670.630812242769 in microbenchmark.json
REGRESSION 1.68%: n_n_actor_calls_with_arg_async (THROUGHPUT) regresses from 2719.6626076209245 to 2674.014821048363 in microbenchmark.json
REGRESSION 1.63%: n_n_async_actor_calls_async (THROUGHPUT) regresses from 22994.669871663224 to 22620.58710137489 in microbenchmark.json
REGRESSION 0.90%: client__put_gigabytes (THROUGHPUT) regresses from 0.15371223612909402 to 0.1523218281420353 in microbenchmark.json
REGRESSION 0.63%: client__1_1_actor_calls_concurrent (THROUGHPUT) regresses from 974.1707706196564 to 968.0008238131173 in microbenchmark.json
REGRESSION 268.59%: dashboard_p95_latency_ms (LATENCY) regresses from 8.286 to 30.541 in benchmarks/many_pgs.json
REGRESSION 58.31%: avg_pg_create_time_ms (LATENCY) regresses from 0.9711981891879983 to 1.5375056531531037 in stress_tests/stress_test_placement_group.json
REGRESSION 41.29%: avg_pg_remove_time_ms (LATENCY) regresses from 0.9386635615607924 to 1.3262595690691725 in stress_tests/stress_test_placement_group.json
REGRESSION 3.90%: dashboard_p95_latency_ms (LATENCY) regresses from 2548.0 to 2647.381 in benchmarks/many_actors.json
REGRESSION 1.95%: 10000_get_time (LATENCY) regresses from 23.672960529000008 to 24.133820766 in scalability/single_node.json
REGRESSION 0.13%: 10000_args_time (LATENCY) regresses from 17.524066807000004 to 17.54721164899999 in scalability/single_node.json
REGRESSION 0.01%: 1000000_queued_time (LATENCY) regresses from 189.086426824 to 189.11093958500004 in scalability/single_node.json

@dayshah dayshah requested a review from jjyao November 24, 2024 22:19
dentiny
dentiny reviewed Nov 25, 2024
View reviewed changes
release/perf_metrics/benchmarks/many_tasks.json
"num_tasks": 10000,
"perf_metrics": [
{
"perf_metric_name": "tasks_per_second",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 469.18167402268733
"perf_metric_value": 563.3604483841477
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jjyao / @rynewang performance seems better for test_many_tasks

dentiny
dentiny reviewed Nov 25, 2024
View reviewed changes
release/perf_metrics/stress_tests/stress_test_many_tasks.json
"stage_2_time": 194.8006236553192,
"stage_3_creation_time": 1.3092761039733887,
"stage_3_time": 2052.3795640468597,
"stage_4_spread": 0.20646123231684818,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jjyao / @rynewang performance seems better for test_many_tasks

@jjyao jjyao added the go add ONLY when ready to merge, run all tests label Nov 25, 2024
@jjyao
Copy link
Collaborator

jjyao commented Nov 25, 2024 •
edited
Loading

REGRESSION 11.06%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 12.047828247398513 to 10.715440745783182 in microbenchmark.json

Noise.

REGRESSION 9.50%: 1_1_async_actor_calls_async (THROUGHPUT) regresses from 5128.909004080372 to 4641.857377193749 in microbenchmark.json

Noise.

REGRESSION 9.33%: placement_group_create/removal (THROUGHPUT) regresses from 845.3611380520398 to 766.4710772352788 in microbenchmark.json

Noise.

REGRESSION 9.14%: single_client_tasks_async (THROUGHPUT) regresses from 7850.804688217522 to 7133.343594327644 in microbenchmark.json

Noise.

REGRESSION 58.31%: avg_pg_create_time_ms (LATENCY) regresses from 0.9711981891879983 to 1.5375056531531037 in stress_tests/stress_test_placement_group.json
REGRESSION 41.29%: avg_pg_remove_time_ms (LATENCY) regresses from 0.9386635615607924 to 1.3262595690691725 in stress_tests/stress_test_placement_group.json

Due to the instance type update. #48851

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dentiny dentiny dentiny left review comments

@jjyao jjyao jjyao approved these changes

Assignees

@jjyao jjyao

Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@khluu @jjyao @dentiny @dayshah