Benchmarks 2023 12 22 TVM GCC

Setup

Simulator

• Spike (riscv-isa-sim ) (ISS, CPI=1)

Toolchains

• Scalar: Self compiled on master branch of riscv-gnu-toolchain
• Packed: Self compiled using patches found in https://github.com/riscv-collab/riscv-gcc/pull/258 and https://github.com/riscvarchive/riscv-binutils-gdb/pull/257
• Vector: Self compiled using this working-branch: riscv-gcc-rvv-next

Models

• MLPerfTiny Benchmark

Package Versions

• MLonMCU : main
• TFLM : main
• TVM : Nightly Pre-Build
• Spike : 0bc176b3fca43560b9e8586cdbc41cfde073e17a
• Spike PK : 7e9b671c0415dfd7b562ac934feb9380075d4aa2

Miscellaneous

• Used -Os flag for compilation.
• Benchmarks generated using MLonMCU deployment tool with minimal efforts.
• Memory metrics are reported in Bytes

Results (Framework: tvm, Toolchain: gcc, Backend: tvmaot)

Audio Wake Words (aww)

Cycles	Cycles(Relative)	Total ROM	Total ROM(Relative)	Total RAM	Total RAM(Relative)	ROM read-only	ROM code	VLEN	Layout	Kernels	Extensions
31536718	1.871	107668	1.269	59536	3.097	55404	52248	0	NCHW	Default	-
29578962	1.754	101746	1.199	59536	3.097	55404	46326	0	NHWC	Default	-
15873108	0.941	101202	1.193	51364	2.672	55404	45782	0	NCHW	Autotuned	-
29574349	1.754	101888	1.201	59536	3.097	55404	46468	0	NHWC	Autotuned	-
16859464	1.0	84850	1.0	19224	1.0	35836	48998	0	-	muRISCV-NN	-
9575012	0.568	97186	1.145	23640	1.23	36800	60370	64	-	muRISCV-NN	VEXT
4086691	0.242	97186	1.145	23640	1.23	36800	60370	1024	-	muRISCV-NN	VEXT
15707730	0.932	99206	1.169	20324	1.057	36552	62578	0	-	muRISCV-NN	PEXT

Image Classification (resnet)

Cycles	Cycles(Relative)	Total ROM	Total ROM(Relative)	Total RAM	Total RAM(Relative)	ROM read-only	ROM code	VLEN	Layout	Kernels	Extensions
132014570	1.63	216916	1.618	108448	1.953	162348	54552	0	NCHW	Default	-
115757507	1.429	207824	1.55	108448	1.953	162348	45460	0	NHWC	Default	-
57193829	0.706	211006	1.574	92264	1.662	162348	48642	0	NCHW	Autotuned	-
115753589	1.429	207928	1.551	108448	1.953	162348	45564	0	NHWC	Autotuned	-
81014730	1.0	134096	1.0	55528	1.0	86524	47556	0	-	muRISCV-NN	-
34120461	0.421	146414	1.092	55480	0.999	87488	58910	64	-	muRISCV-NN	VEXT
5891882	0.073	146414	1.092	55480	0.999	87488	58910	1024	-	muRISCV-NN	VEXT
68441624	0.845	149082	1.112	55476	0.999	87240	61766	0	-	muRISCV-NN	PEXT

Anomaly Detection (toycar)

Cycles	Cycles(Relative)	Total ROM	Total ROM(Relative)	Total RAM	Total RAM(Relative)	ROM read-only	ROM code	VLEN	Layout	Kernels	Extensions
3230152	1.596	580056	1.84	5580	1.167	537180	42860	0	NCHW	Default	-
3230152	1.596	580056	1.84	5580	1.167	537180	42860	0	NHWC	Default	-
2147464	1.061	609298	1.933	6892	1.442	537180	72102	0	NCHW	Autotuned	-
2147464	1.061	609298	1.933	6892	1.442	537180	72102	0	NHWC	Autotuned	-
2023540	1.0	315168	1.0	4780	1.0	273116	42036	0	-	muRISCV-NN	-
642338	0.317	327050	1.038	4732	0.99	274080	52954	64	-	muRISCV-NN	VEXT
451634	0.223	327054	1.038	4732	0.99	274080	52958	1024	-	muRISCV-NN	VEXT
996956	0.493	327838	1.04	4728	0.989	273832	53930	0	-	muRISCV-NN	PEXT

Visual Wake Words (vww)

Cycles	Cycles(Relative)	Total ROM	Total ROM(Relative)	Total RAM	Total RAM(Relative)	ROM read-only	ROM code	VLEN	Layout	Kernels	Extensions
91779188	1.843	540898	1.697	181056	2.113	462124	78758	0	NCHW	Default	-
86076684	1.728	520456	1.633	181056	2.113	462124	58316	0	NHWC	Default	-
46547501	0.935	521902	1.638	181056	2.113	462124	59762	0	NCHW	Autotuned	-
86076687	1.728	520458	1.633	181056	2.113	462124	58318	0	NHWC	Autotuned	-
49801833	1.0	318674	1.0	85672	1.0	265516	53142	0	-	muRISCV-NN	-
24644360	0.495	331054	1.039	85624	0.999	266480	64558	64	-	muRISCV-NN	VEXT
8138573	0.163	331040	1.039	85624	0.999	266480	64544	1024	-	muRISCV-NN	VEXT
49304027	0.99	333072	1.045	85620	0.999	266232	66764	0	-	muRISCV-NN	PEXT

Original data

Click here to download the raw files for this benchmark.