Skip to content

Latest commit

 

History

History
152 lines (142 loc) · 14.4 KB

20240202.md

File metadata and controls

152 lines (142 loc) · 14.4 KB
Raw

Report 2024-02-02

Versions

  • Blaze version: 2.0.8
  • Vanilla spark version: spark-3.3.3

Environment

Hadoop 2.6.0 cluster mode running on 4 nodes, See Kwai server conf.

Configuration

  • Blaze
spark.executor.memory 5g
spark.executor.memoryOverhead 3072
spark.blaze.memoryFraction 0.7
spark.blaze.enable.caseconvert.functions true
spark.blaze.enable.smjInequalityJoin false
spark.blaze.enable.bhjFallbacksToSmj false
  • Vanilla Spark
spark.executor.memory 6g
spark.executor.memoryOverhead 2048
  • Common configurations
spark.speculation false
spark.sql.adaptive.coalescePartitions.initialPartitionNum 1000
spark.sql.adaptive.coalescePartitions.minPartitionNum 20
spark.sql.adaptive.coalescePartitions.minPartitionSize 65536
spark.sql.files.maxPartitionBytes 268435456
spark.sql.autoBroadcastJoinThreshold 20971520

Results

Query time comparison (seconds): blaze-query-time-comparison-20240202.png

Executor time comparison (Memory Bytes * Seconds): blaze-cluster-resources-cost-comparison-20240202.png

Memcost Spark Memcost Blaze Blaze / non-Blaze Query time Spark Query time Blaze Blaze / non-Blaze
q01 1064427008 449834240 0.422606939 q01 10.6 6.6 0.622641509
q02 2368745984 2721906176 1.149091627 q02 13 49.4 3.8
q03 2393231360 1122979200 0.469231358 q03 12 7.4 0.616666667
q04 70389727232 24298207232 0.34519536 q04 292.8 105.3 0.359631148
q05 14231480320 4769662976 0.33514876 q05 58.5 22.4 0.382905983
q06 14682045440 2202324480 0.150001203 q06 72 11.9 0.165277778
q07 5505946624 3115785216 0.565894555 q07 27.3 17.6 0.644688645
q08 2245357056 1041340864 0.463775176 q08 12.3 7.5 0.609756098
q09 12896677888 6773364224 0.525202248 q09 53.6 28.8 0.537313433
q10 3314154496 975653056 0.294389733 q10 22.8 15.5 0.679824561
q11 11654519808 12878838784 1.105051001 q11 53.4 58.8 1.101123596
q12 686692864 502959360 0.732437144 q12 5 4.8 0.96
q13 7461695488 3379008256 0.452847247 q13 37.2 17.2 0.462365591
q14a 51038400512 18640779264 0.365230475 q14a 323 182 0.563467492
q14b 37148839936 16267428864 0.437898704 q14b 259.7 177.7 0.684251059
q15 1749106432 1486633472 0.849938829 q15 9.5 8.9 0.936842105
q16 27584167936 13769887744 0.499195327 q16 123.3 65.1 0.527980535
q17 58074173440 14808196096 0.254987634 q17 257.3 67.6 0.262728333
q18 3536184832 2220624640 0.627971881 q18 21.8 14.2 0.651376147
q19 3174092032 1489778944 0.469355938 q19 24.1 9.4 0.390041494
q20 1256781568 897696896 0.714282353 q20 8 5.6 0.7
q21 8047820 2254438 0.280130271 q21 5.4 2.1 0.388888889
q22 12014388 5016781 0.417564424 q22 7.4 3.2 0.432432432
q23a 81502380032 31732547584 0.389345042 q23a 357.9 149.3 0.41715563
q23b 80083304448 30878326784 0.385577581 q23b 351.1 136.8 0.389632583
q24a 49402916864 16241479680 0.328755481 q24a 215.1 79.8 0.370990237
q24b 49900470272 16226333696 0.325173963 q24b 215.9 79.4 0.367762853
q25 78905024512 18997395456 0.24076281 q25 352.4 84.6 0.240068104
q26 2931277056 1529305728 0.521719953 q26 16.5 10.9 0.660606061
q27 5451323904 2849344768 0.522688583 q27 28.1 16.4 0.583629893
q28 21703208960 11386787840 0.524659181 q28 88.8 49 0.551801802
q29 54227959808 16019457024 0.295409547 q29 242 74.6 0.308264463
q30 302891104 135692304 0.447990391 q30 8.6 5.4 0.627906977
q31 9615978496 4488949248 0.466821889 q31 41.2 23.3 0.565533981
q32 2419902464 958353984 0.396030005 q32 14.6 6.6 0.452054795
q33 4866960896 2139079296 0.43951027 q33 22.6 13.7 0.60619469
q34 2799148544 1131107328 0.404089783 q34 15.4 11.5 0.746753247
q35 5119056896 1135299840 0.221779102 q35 42.5 16.1 0.378823529
q36 4177485312 2434962944 0.582877679 q36 21.4 13.8 0.644859813
q37 7496896512 932590400 0.124396862 q37 49.1 21 0.427698574
q38 4062855168 2151781376 0.529622959 q38 30.4 17 0.559210526
q39a 17670146 6114509 0.346036133 q39a 5.5 2.5 0.454545455
q39b 15856440 6545408 0.412791774 q39b 5.3 2.5 0.471698113
q40 15339921408 5915641344 0.385637005 q40 75.2 31.3 0.416223404
q41 429260 339148 0.790075945 q41 0.7 0.8 1.142857143
q42 2318457088 973158720 0.419744116 q42 11.8 5.7 0.483050847
q43 2643484416 1623958784 0.614325083 q43 32.7 8.4 0.256880734
q44 4751142912 2148898048 0.452290762 q44 23.4 45.1 1.927350427
q45 859250816 774846720 0.90177013 q45 5.7 5.4 0.947368421
q46 5104092160 2574011392 0.504303471 q46 25.9 16.4 0.633204633
q47 4227521024 3525534464 0.833948417 q47 22.9 21.8 0.951965066
q48 5522035200 2340343040 0.423818928 q48 27.8 12.3 0.442446043
q49 18826491904 7428071936 0.394554226 q49 85.9 34.6 0.402793946
q50 40762097664 7524097536 0.184585631 q50 177.7 35.4 0.199212155
q51 6946065920 2681179904 0.385999778 q51 42.2 16.5 0.390995261
q52 2423949056 989473792 0.408207338 q52 13.3 6.3 0.473684211
q53 2978816512 1240832768 0.416552266 q53 14.8 8.2 0.554054054
q54 18391373824 4565063680 0.248217655 q54 111.6 22.2 0.198924731
q55 3162129152 972509952 0.307549093 q55 105.5 6.9 0.065402844
q56 5093037568 2148585216 0.421867145 q56 23.8 13.4 0.56302521
q57 1928995200 1695657856 0.879036846 q57 11.3 12.6 1.115044248
q58 4623944704 3115723520 0.673823698 q58 22.1 17.3 0.78280543
q59 3398420992 3844507648 1.131262918 q59 16.7 20.4 1.221556886
q60 5130304512 2142691072 0.41765378 q60 23.9 13.9 0.581589958
q61 7130669568 3692626944 0.517851361 q61 31.7 17.4 0.548895899
q62 861110208 734616064 0.853103421 q62 5.2 5.7 1.096153846
q63 2913974784 1220212608 0.418745081 q63 14.3 8.3 0.58041958
q64 90821681152 45009911808 0.495585539 q64 393 217.5 0.553435115
q65 6929706496 4274123520 0.616782763 q65 31 21 0.677419355
q66 4168856064 2715433216 0.65136171 q66 43 14.1 0.327906977
q67 16813858816 11755253760 0.699140744 q67 80.9 55.6 0.687268232
q68 6089261056 3126788096 0.513492207 q68 30.5 17.9 0.586885246
q69 3271612928 955816128 0.292154405 q69 20.3 10.7 0.527093596
q70 5136711680 2733509120 0.532151557 q70 24.8 14.6 0.588709677
q71 5308280832 2298735360 0.433047051 q71 24.9 17.5 0.702811245
q72 34922856448 25395292160 0.72718256 q72 174.4 131 0.751146789
q73 2774287872 1024954944 0.369447942 q73 14.1 9.1 0.645390071
q74 8379421696 6975468544 0.832452262 q74 38.8 33.8 0.871134021
q75 21823078400 10669447168 0.488906605 q75 98.5 57.9 0.587817259
q76 5128589824 1943261056 0.378907482 q76 36.8 11.7 0.317934783
q77 6024270336 2879578880 0.477996292 q77 33.3 15 0.45045045
q78 90786725888 36193472512 0.398664807 q78 377 154.4 0.409549072
q79 5194788864 2371766016 0.456566393 q79 25.4 14.4 0.566929134
q80 86372556800 32238481408 0.373249127 q80 351.9 133.5 0.379369139
q81 537159104 260399136 0.484770963 q81 9.4 6.2 0.659574468
q82 14299611136 1773495680 0.12402405 q82 83.6 22.9 0.273923445
q83 370437344 190986624 0.515570655 q83 13.5 8.7 0.644444444
q84 719424896 87703536 0.121907841 q84 12.9 2.7 0.209302326
q85 2387102720 839722560 0.351774791 q85 28.9 11.6 0.401384083
q86 810701824 355365728 0.438343319 q86 35.8 4.7 0.131284916
q87 4160053504 2180360192 0.524118305 q87 30.6 17 0.555555556
q88 14849074176 5351185920 0.360371688 q88 61.4 24.8 0.403908795
q89 3119070464 1416152448 0.454030284 q89 15.7 10.2 0.649681529
q90 947142592 274404096 0.28971783 q90 7.5 2.7 0.36
q91 140899136 61533388 0.436719413 q91 5.3 2.9 0.547169811
q92 1332381184 475557024 0.356922651 q92 9.2 4.1 0.445652174
q93 45288034304 12807971840 0.282811388 q93 200.3 61.8 0.308537194
q94 16076772352 6874429440 0.427600098 q94 82.9 36.3 0.43787696
q95 23590234112 17770180608 0.753285471 q95 120.8 103.8 0.859271523
q96 1928228480 637357376 0.330540381 q96 9.8 4.1 0.418367347
q97 10208055296 3761395456 0.368473264 q97 56.9 26.3 0.462214411
q98 2648099584 1812735104 0.684541894 q98 15.4 11.8 0.766233766
q99 1629526400 1538929536 0.944402948 q99 8.7 9 1.034482759
sum 1.50781E+12 6.1011E+11 7367.7 3390.8