Skip to content

Benchmarks 2024 02 22 TVM LLVM

Jump to bottom
GitHub Action edited this page Feb 22, 2024 · 3 revisions

Setup

Simulator

Toolchains

  • LLVM/Clang:
    • TODO: Version
    • Linker: lld (TODO)
    • RISC-V GCC for Headers, libc,...

Models

Package Versions

  • MLonMCU : main

  • TVM : Nightly Pre-Build

  • Spike : 0bc176b3fca43560b9e8586cdbc41cfde073e17a

  • Spike PK : 7e9b671c0415dfd7b562ac934feb9380075d4aa2

Miscellaneous

  • Used -Os flag for compilation.
  • Benchmarks generated using MLonMCU deployment tool with minimal efforts.
  • Memory metrics are reported in Bytes

Results (Framework: tvm, Backend: tvmaot, Toolchain: llvm)

Audio Wake Words (aww)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
33508602
( 0.3x )		 109102
( 1.044 )		 59508
( 1.0 )		 0 NCHW TVM Fallback RV32GC -
27511073
( 0.3x )		 102506
( 0.981 )		 59508
( 1.0 )		 0 NHWC TVM Fallback RV32GC -
13706216
( 0.7x )		 102504
( 0.98 )		 51336
( 0.863 )		 0 NCHW TVM Autotuned RV32GC -
27505623
( 0.3x )		 102618
( 0.982 )		 59508
( 1.0 )		 0 NHWC TVM Autotuned RV32GC -
3384606
( 2.8x )		 105404
( 1.008 )		 59508
( 1.0 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
3384606
( 2.8x )		 105404
( 1.008 )		 59508
( 1.0 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
3384606
( 2.8x )		 105404
( 1.008 )		 59508
( 1.0 )		 4096 NCHW TVM Fallback RV32GCV Loop+SLP
9566669
( 1.0x )		 103606
( 0.991 )		 59508
( 1.0 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
6682669
( 1.4x )		 103606
( 0.991 )		 59508
( 1.0 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
6682669
( 1.4x )		 103606
( 0.991 )		 59508
( 1.0 )		 4096 NHWC TVM Fallback RV32GCV Loop+SLP
5607715
( 1.7x )		 106776
( 1.021 )		 51336
( 0.863 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
3881984
( 2.5x )		 106776
( 1.021 )		 51336
( 0.863 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
3888762
( 2.5x )		 106776
( 1.021 )		 51336
( 0.863 )		 4096 NCHW TVM Autotuned RV32GCV Loop+SLP
9565134
( Base )		 104544
( Base )		 59508
( Base )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
6683403
( 1.4x )		 104544
( 1.0 )		 59508
( 1.0 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
6690181
( 1.4x )		 104544
( 1.0 )		 59508
( 1.0 )		 4096 NHWC TVM Autotuned RV32GCV Loop+SLP
15615253
( 0.6x )		 90744
( 0.868 )		 19212
( 0.323 )		 0 NHWC muRISCV-NN Scalar RV32GC -
6838472
( 1.4x )		 93956
( 0.899 )		 19212
( 0.323 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
5367945
( 1.8x )		 93956
( 0.899 )		 19212
( 0.323 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
5374723
( 1.8x )		 93956
( 0.899 )		 19212
( 0.323 )		 4096 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
5393590
( 1.8x )		 90936
( 0.87 )		 23676
( 0.398 )		 128 NHWC muRISCV-NN Vector RV32GCV -
3650062
( 2.6x )		 90936
( 0.87 )		 23676
( 0.398 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
3656840
( 2.6x )		 90936
( 0.87 )		 23676
( 0.398 )		 4096 NHWC muRISCV-NN Vector RV32GCV -

Image Classification (resnet)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
144789098
( 0.2x )		 218264
( 1.035 )		 108420
( 1.0 )		 0 NCHW TVM Fallback RV32GC -
112394400
( 0.3x )		 209174
( 0.992 )		 108420
( 1.0 )		 0 NHWC TVM Fallback RV32GC -
53701816
( 0.7x )		 212580
( 1.008 )		 92236
( 0.851 )		 0 NCHW TVM Autotuned RV32GC -
112389726
( 0.3x )		 209264
( 0.992 )		 108420
( 1.0 )		 0 NHWC TVM Autotuned RV32GC -
12825990
( 2.8x )		 213676
( 1.013 )		 108420
( 1.0 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
12825992
( 2.8x )		 213684
( 1.013 )		 108420
( 1.0 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
12825990
( 2.8x )		 213676
( 1.013 )		 108420
( 1.0 )		 4096 NCHW TVM Fallback RV32GCV Loop+SLP
36071695
( 1.0x )		 210194
( 0.997 )		 108420
( 1.0 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
24311825
( 1.5x )		 210202
( 0.997 )		 108420
( 1.0 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
24311825
( 1.5x )		 210202
( 0.997 )		 108420
( 1.0 )		 4096 NHWC TVM Fallback RV32GCV Loop+SLP
19686221
( 1.8x )		 226234
( 1.073 )		 92236
( 0.851 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
13965592
( 2.6x )		 226234
( 1.073 )		 92236
( 0.851 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
14023205
( 2.6x )		 226234
( 1.073 )		 92236
( 0.851 )		 4096 NCHW TVM Autotuned RV32GCV Loop+SLP
36069954
( Base )		 210930
( Base )		 108420
( Base )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
24309153
( 1.5x )		 210938
( 1.0 )		 108420
( 1.0 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
24315929
( 1.5x )		 210930
( 1.0 )		 108420
( 1.0 )		 4096 NHWC TVM Autotuned RV32GCV Loop+SLP
58407308
( 0.6x )		 138192
( 0.655 )		 55516
( 0.512 )		 0 NHWC muRISCV-NN Scalar RV32GC -
28255208
( 1.3x )		 141914
( 0.673 )		 55516
( 0.512 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
13704813
( 2.6x )		 141914
( 0.673 )		 55516
( 0.512 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
12134631
( 3.0x )		 141914
( 0.673 )		 55516
( 0.512 )		 4096 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
15905989
( 2.3x )		 139024
( 0.659 )		 55516
( 0.512 )		 128 NHWC muRISCV-NN Vector RV32GCV -
6531691
( 5.5x )		 139024
( 0.659 )		 55516
( 0.512 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
5360777
( 6.7x )		 139024
( 0.659 )		 55516
( 0.512 )		 4096 NHWC muRISCV-NN Vector RV32GCV -

Anomaly Detection (toycar)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
3404880
( 0.4x )		 581362
( 0.968 )		 5572
( 0.809 )		 0 NCHW TVM Fallback RV32GC -
3404880
( 0.4x )		 581362
( 0.968 )		 5572
( 0.809 )		 0 NHWC TVM Fallback RV32GC -
2245737
( 0.6x )		 609080
( 1.014 )		 6884
( 1.0 )		 0 NCHW TVM Autotuned RV32GC -
2245737
( 0.6x )		 609080
( 1.014 )		 6884
( 1.0 )		 0 NHWC TVM Autotuned RV32GC -
984695
( 1.3x )		 581106
( 0.968 )		 5572
( 0.809 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
984695
( 1.3x )		 581106
( 0.968 )		 5572
( 0.809 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
984693
( 1.3x )		 581098
( 0.968 )		 5572
( 0.809 )		 4096 NCHW TVM Fallback RV32GCV Loop+SLP
984695
( 1.3x )		 581106
( 0.968 )		 5572
( 0.809 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
984693
( 1.3x )		 581098
( 0.968 )		 5572
( 0.809 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
984693
( 1.3x )		 581098
( 0.968 )		 5572
( 0.809 )		 4096 NHWC TVM Fallback RV32GCV Loop+SLP
1280619
( 1.0x )		 600432
( 1.0 )		 6884
( 1.0 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
1148032
( 1.1x )		 600432
( 1.0 )		 6884
( 1.0 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
1151904
( 1.1x )		 600432
( 1.0 )		 6884
( 1.0 )		 4096 NCHW TVM Autotuned RV32GCV Loop+SLP
1280619
( Base )		 600432
( Base )		 6884
( Base )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
1148032
( 1.1x )		 600432
( 1.0 )		 6884
( 1.0 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
1151904
( 1.1x )		 600432
( 1.0 )		 6884
( 1.0 )		 4096 NHWC TVM Autotuned RV32GCV Loop+SLP
1724219
( 0.7x )		 315976
( 0.526 )		 4772
( 0.693 )		 0 NHWC muRISCV-NN Scalar RV32GC -
620488
( 2.1x )		 316614
( 0.527 )		 4772
( 0.693 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
421128
( 3.0x )		 316614
( 0.527 )		 4772
( 0.693 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
405102
( 3.2x )		 316608
( 0.527 )		 4772
( 0.693 )		 4096 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
618525
( 2.1x )		 316516
( 0.527 )		 4772
( 0.693 )		 128 NHWC muRISCV-NN Vector RV32GCV -
414660
( 3.1x )		 316518
( 0.527 )		 4772
( 0.693 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
409173
( 3.1x )		 316518
( 0.527 )		 4772
( 0.693 )		 4096 NHWC muRISCV-NN Vector RV32GCV -

Visual Wake Words (vww)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
96665972
( 0.3x )		 545176
( 1.041 )		 181032
( 1.0 )		 0 NCHW TVM Fallback RV32GC -
79940190
( 0.4x )		 521128
( 0.995 )		 181032
( 1.0 )		 0 NHWC TVM Fallback RV32GC -
42404605
( 0.7x )		 525206
( 1.003 )		 181032
( 1.0 )		 0 NCHW TVM Autotuned RV32GC -
79940189
( 0.4x )		 521126
( 0.995 )		 181032
( 1.0 )		 0 NHWC TVM Autotuned RV32GC -
11010121
( 2.8x )		 532512
( 1.017 )		 181032
( 1.0 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
11010119
( 2.8x )		 532508
( 1.017 )		 181032
( 1.0 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
11010120
( 2.8x )		 532512
( 1.017 )		 181032
( 1.0 )		 4096 NCHW TVM Fallback RV32GCV Loop+SLP
30451802
( 1.0x )		 523638
( 1.0 )		 181032
( 1.0 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
22700929
( 1.3x )		 523638
( 1.0 )		 181032
( 1.0 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
22307809
( 1.4x )		 523638
( 1.0 )		 181032
( 1.0 )		 4096 NHWC TVM Fallback RV32GCV Loop+SLP
24965513
( 1.2x )		 550722
( 1.051 )		 181032
( 1.0 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
19204882
( 1.6x )		 550722
( 1.051 )		 181032
( 1.0 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
19039942
( 1.6x )		 550718
( 1.051 )		 181032
( 1.0 )		 4096 NCHW TVM Autotuned RV32GCV Loop+SLP
30450319
( Base )		 523750
( Base )		 181032
( Base )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
22698969
( 1.3x )		 523748
( 1.0 )		 181032
( 1.0 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
22305797
( 1.4x )		 523746
( 1.0 )		 181032
( 1.0 )		 4096 NHWC TVM Autotuned RV32GCV Loop+SLP
46765906
( 0.7x )		 323762
( 0.618 )		 85664
( 0.473 )		 0 NHWC muRISCV-NN Scalar RV32GC -
19684204
( 1.5x )		 327948
( 0.626 )		 85664
( 0.473 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
14888472
( 2.0x )		 327942
( 0.626 )		 85664
( 0.473 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
14671294
( 2.1x )		 327944
( 0.626 )		 85664
( 0.473 )		 4096 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
14675146
( 2.1x )		 325034
( 0.621 )		 85664
( 0.473 )		 128 NHWC muRISCV-NN Vector RV32GCV -
9540606
( 3.2x )		 325034
( 0.621 )		 85664
( 0.473 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
9498994
( 3.2x )		 325036
( 0.621 )		 85664
( 0.473 )		 4096 NHWC muRISCV-NN Vector RV32GCV -

Original data

Click here to download the raw files for this benchmark.

2024-11-26
2024-11-21
2024-11-19
2024-11-18
2024-07-12
2024-06-29
2024-03-02
2024-02-26
2024-02-23
2024-02-22
2024-02-20
2024-02-11
2023-12-22
2024-11-26
2024-11-21
2024-11-19
2024-11-18
2024-07-12
2024-06-29
2024-03-02
2024-02-26
2024-02-23
2024-02-22
2024-02-20
2024-02-11
2023-12-22
Clone this wiki locally