Skip to content

Benchmarks 2024 02 11 TVM GCC

Jump to bottom
Philipp van Kempen edited this page Feb 11, 2024 · 6 revisions

Setup

Simulator

Toolchains

Models

Package Versions

  • MLonMCU : main

  • TVM : Nightly Pre-Build

  • Spike : 0bc176b3fca43560b9e8586cdbc41cfde073e17a

  • Spike PK : 7e9b671c0415dfd7b562ac934feb9380075d4aa2

Miscellaneous

  • Used -Os flag for compilation.
  • Benchmarks generated using MLonMCU deployment tool with minimal efforts.
  • Memory metrics are reported in Bytes

Results (Framework: tvm, Backend: tvmaot, Toolchain: gcc)

Audio Wake Words (aww)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
31536718
( 0.5x )		 107754
( 1.269 )		 59536
( 3.097 )		 0 NCHW TVM Fallback RV32GC -
29578962
( 0.6x )		 101832
( 1.199 )		 59536
( 3.097 )		 0 NHWC TVM Fallback RV32GC -
15873108
( 1.1x )		 101286
( 1.192 )		 51364
( 2.672 )		 0 NCHW TVM Autotuned RV32GC -
29574349
( 0.6x )		 101974
( 1.201 )		 59536
( 3.097 )		 0 NHWC TVM Autotuned RV32GC -
31529514
( 0.5x )		 107730
( 1.268 )		 59536
( 3.097 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
31529514
( 0.5x )		 107730
( 1.268 )		 59536
( 3.097 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
29578958
( 0.6x )		 101832
( 1.199 )		 59536
( 3.097 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
29578958
( 0.6x )		 101832
( 1.199 )		 59536
( 3.097 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
15873104
( 1.1x )		 101286
( 1.192 )		 51364
( 2.672 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
15873104
( 1.1x )		 101286
( 1.192 )		 51364
( 2.672 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
29574345
( 0.6x )		 101974
( 1.201 )		 59536
( 3.097 )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
29574345
( 0.6x )		 101974
( 1.201 )		 59536
( 3.097 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
28034895
( 0.6x )		 119608
( 1.408 )		 59484
( 3.094 )		 0 NCHW TVM Fallback RV32GCP -
26289773
( 0.6x )		 113910
( 1.341 )		 59484
( 3.094 )		 0 NHWC TVM Fallback RV32GCP -
12897999
( 1.3x )		 113210
( 1.333 )		 51312
( 2.669 )		 0 NCHW TVM Autotuned RV32GCP -
26285154
( 0.6x )		 114032
( 1.343 )		 59484
( 3.094 )		 0 NHWC TVM Autotuned RV32GCP -
16859470
( Base )		 84938
( Base )		 19224
( Base )		 0 NHWC muRISCV-NN Scalar RV32GC -
16859470
( 1.0x )		 84938
( 1.0 )		 19224
( 1.0 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
16859470
( 1.0x )		 84938
( 1.0 )		 19224
( 1.0 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
8513393
( 2.0x )		 85408
( 1.006 )		 23688
( 1.232 )		 128 NHWC muRISCV-NN Vector RV32GCV -
4600465
( 3.7x )		 85408
( 1.006 )		 23688
( 1.232 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
13443818
( 1.3x )		 96894
( 1.141 )		 19172
( 0.997 )		 0 NHWC muRISCV-NN Scalar RV32GCP -
15707734
( 1.1x )		 99208
( 1.168 )		 20324
( 1.057 )		 0 NHWC muRISCV-NN Packed RV32GCP -

Image Classification (resnet)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
132014570
( 0.6x )		 217002
( 1.617 )		 108448
( 1.953 )		 0 NCHW TVM Fallback RV32GC -
115757507
( 0.7x )		 207908
( 1.549 )		 108448
( 1.953 )		 0 NHWC TVM Fallback RV32GC -
57193829
( 1.4x )		 211090
( 1.573 )		 92264
( 1.662 )		 0 NCHW TVM Autotuned RV32GC -
115753589
( 0.7x )		 208012
( 1.55 )		 108448
( 1.953 )		 0 NHWC TVM Autotuned RV32GC -
132147940
( 0.6x )		 216992
( 1.617 )		 108448
( 1.953 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
132147940
( 0.6x )		 216990
( 1.617 )		 108448
( 1.953 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
115757501
( 0.7x )		 207910
( 1.549 )		 108448
( 1.953 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
115757501
( 0.7x )		 207910
( 1.549 )		 108448
( 1.953 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
57193823
( 1.4x )		 211096
( 1.573 )		 92264
( 1.662 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
57193823
( 1.4x )		 211096
( 1.573 )		 92264
( 1.662 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
115753583
( 0.7x )		 208010
( 1.55 )		 108448
( 1.953 )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
115753583
( 0.7x )		 208012
( 1.55 )		 108448
( 1.953 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
134393924
( 0.6x )		 229126
( 1.708 )		 108396
( 1.952 )		 0 NCHW TVM Fallback RV32GCP -
102349260
( 0.8x )		 219946
( 1.639 )		 108396
( 1.952 )		 0 NHWC TVM Fallback RV32GCP -
44503696
( 1.8x )		 222790
( 1.66 )		 92212
( 1.661 )		 0 NCHW TVM Autotuned RV32GCP -
102345331
( 0.8x )		 220022
( 1.64 )		 108396
( 1.952 )		 0 NHWC TVM Autotuned RV32GCP -
81014730
( Base )		 134184
( Base )		 55528
( Base )		 0 NHWC muRISCV-NN Scalar RV32GC -
81014730
( 1.0x )		 134184
( 1.0 )		 55528
( 1.0 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
81014730
( 1.0x )		 134184
( 1.0 )		 55528
( 1.0 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
29990082
( 2.7x )		 135008
( 1.006 )		 55528
( 1.0 )		 128 NHWC muRISCV-NN Vector RV32GCV -
8377242
( 9.7x )		 135008
( 1.006 )		 55528
( 1.0 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
62992383
( 1.3x )		 146142
( 1.089 )		 55476
( 0.999 )		 0 NHWC muRISCV-NN Scalar RV32GCP -
68441624
( 1.2x )		 149086
( 1.111 )		 55476
( 0.999 )		 0 NHWC muRISCV-NN Packed RV32GCP -

Anomaly Detection (toycar)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
3230152
( 0.6x )		 580142
( 1.84 )		 5580
( 1.167 )		 0 NCHW TVM Fallback RV32GC -
3230152
( 0.6x )		 580142
( 1.84 )		 5580
( 1.167 )		 0 NHWC TVM Fallback RV32GC -
2147464
( 0.9x )		 609314
( 1.933 )		 6892
( 1.442 )		 0 NCHW TVM Autotuned RV32GC -
2147464
( 0.9x )		 609314
( 1.933 )		 6892
( 1.442 )		 0 NHWC TVM Autotuned RV32GC -
3230152
( 0.6x )		 580130
( 1.84 )		 5580
( 1.167 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
3230152
( 0.6x )		 580130
( 1.84 )		 5580
( 1.167 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
3230152
( 0.6x )		 580130
( 1.84 )		 5580
( 1.167 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
3230152
( 0.6x )		 580130
( 1.84 )		 5580
( 1.167 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
2147464
( 0.9x )		 609314
( 1.933 )		 6892
( 1.442 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
2147464
( 0.9x )		 609314
( 1.933 )		 6892
( 1.442 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
2147464
( 0.9x )		 609314
( 1.933 )		 6892
( 1.442 )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
2147464
( 0.9x )		 609314
( 1.933 )		 6892
( 1.442 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
2867860
( 0.7x )		 592550
( 1.88 )		 5528
( 1.156 )		 0 NCHW TVM Fallback RV32GCP -
2867860
( 0.7x )		 592550
( 1.88 )		 5528
( 1.156 )		 0 NHWC TVM Fallback RV32GCP -
1769706
( 1.1x )		 618794
( 1.963 )		 6840
( 1.431 )		 0 NCHW TVM Autotuned RV32GCP -
1769706
( 1.1x )		 618794
( 1.963 )		 6840
( 1.431 )		 0 NHWC TVM Autotuned RV32GCP -
2023540
( Base )		 315254
( Base )		 4780
( Base )		 0 NHWC muRISCV-NN Scalar RV32GC -
2023540
( 1.0x )		 315254
( 1.0 )		 4780
( 1.0 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
2023540
( 1.0x )		 315254
( 1.0 )		 4780
( 1.0 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
662389
( 3.1x )		 315044
( 0.999 )		 4780
( 1.0 )		 128 NHWC muRISCV-NN Vector RV32GCV -
482515
( 4.2x )		 315038
( 0.999 )		 4780
( 1.0 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
1927607
( 1.0x )		 327486
( 1.039 )		 4728
( 0.989 )		 0 NHWC muRISCV-NN Scalar RV32GCP -
996956
( 2.0x )		 327838
( 1.04 )		 4728
( 0.989 )		 0 NHWC muRISCV-NN Packed RV32GCP -

Visual Wake Words (vww)

Cycles (Speedup) Total ROM (rel.) Total RAM (rel.) VLEN Layout Kernels Mode Arch Auto-Vectorization
91779188
( 0.5x )		 540914
( 1.697 )		 181056
( 2.113 )		 0 NCHW TVM Fallback RV32GC -
86076684
( 0.6x )		 520540
( 1.633 )		 181056
( 2.113 )		 0 NHWC TVM Fallback RV32GC -
46542367
( 1.1x )		 521920
( 1.637 )		 181056
( 2.113 )		 0 NCHW TVM Autotuned RV32GC -
86076687
( 0.6x )		 520544
( 1.633 )		 181056
( 2.113 )		 0 NHWC TVM Autotuned RV32GC -
91775924
( 0.5x )		 540820
( 1.697 )		 181056
( 2.113 )		 128 NCHW TVM Fallback RV32GCV Loop+SLP
91775924
( 0.5x )		 540820
( 1.697 )		 181056
( 2.113 )		 1024 NCHW TVM Fallback RV32GCV Loop+SLP
86076684
( 0.6x )		 520542
( 1.633 )		 181056
( 2.113 )		 128 NHWC TVM Fallback RV32GCV Loop+SLP
86076684
( 0.6x )		 520542
( 1.633 )		 181056
( 2.113 )		 1024 NHWC TVM Fallback RV32GCV Loop+SLP
46557547
( 1.1x )		 521910
( 1.637 )		 181056
( 2.113 )		 128 NCHW TVM Autotuned RV32GCV Loop+SLP
46557547
( 1.1x )		 521910
( 1.637 )		 181056
( 2.113 )		 1024 NCHW TVM Autotuned RV32GCV Loop+SLP
86076687
( 0.6x )		 520544
( 1.633 )		 181056
( 2.113 )		 128 NHWC TVM Autotuned RV32GCV Loop+SLP
86076687
( 0.6x )		 520544
( 1.633 )		 181056
( 2.113 )		 1024 NHWC TVM Autotuned RV32GCV Loop+SLP
81680879
( 0.6x )		 551532
( 1.73 )		 181004
( 2.113 )		 0 NCHW TVM Fallback RV32GCP -
76410709
( 0.7x )		 531796
( 1.668 )		 181004
( 2.113 )		 0 NHWC TVM Fallback RV32GCP -
37728221
( 1.3x )		 533010
( 1.672 )		 181004
( 2.113 )		 0 NCHW TVM Autotuned RV32GCP -
76410712
( 0.7x )		 531800
( 1.668 )		 181004
( 2.113 )		 0 NHWC TVM Autotuned RV32GCP -
49801846
( Base )		 318762
( Base )		 85672
( Base )		 0 NHWC muRISCV-NN Scalar RV32GC -
49801846
( 1.0x )		 318762
( 1.0 )		 85672
( 1.0 )		 128 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
49801846
( 1.0x )		 318762
( 1.0 )		 85672
( 1.0 )		 1024 NHWC muRISCV-NN Scalar RV32GCV Loop+SLP
22057141
( 2.3x )		 319530
( 1.002 )		 85672
( 1.0 )		 128 NHWC muRISCV-NN Vector RV32GCV -
10627692
( 4.7x )		 319530
( 1.002 )		 85672
( 1.0 )		 1024 NHWC muRISCV-NN Vector RV32GCV -
40880380
( 1.2x )		 330522
( 1.037 )		 85620
( 0.999 )		 0 NHWC muRISCV-NN Scalar RV32GCP -
49304040
( 1.0x )		 333102
( 1.045 )		 85620
( 0.999 )		 0 NHWC muRISCV-NN Packed RV32GCP -

Original data

Click here to download the raw files for this benchmark.

2024-11-26
2024-11-21
2024-11-19
2024-11-18
2024-07-12
2024-06-29
2024-03-02
2024-02-26
2024-02-23
2024-02-22
2024-02-20
2024-02-11
2023-12-22
2024-11-26
2024-11-21
2024-11-19
2024-11-18
2024-07-12
2024-06-29
2024-03-02
2024-02-26
2024-02-23
2024-02-22
2024-02-20
2024-02-11
2023-12-22
Clone this wiki locally