SPEC2017

Info

SPEC2017 is a benchmark, that consists of an integer and floating-point test sets. To run this test you need an image with a native toolchain. Also, you need to have SPEC2017 sources on your file system.

You need to have SPEC tools built before you run the test suite. SPEC tools may be built with the Building tools instruction below or use prebuilt tools.

Then you can move to How to run SPEC2017 step.

More info can be found in the official documentation: SPEC2017: Quick start

Building tools

1) Extract tools

$ cd $SPEC/install_archives
$tar -xvf tools-src/tar
$ cp -r tools/src $SPEC/tools

[Done in custom repo] Update TimeDate module to 2.31 for usage in 2020

https://metacpan.org/release/TimeDate

Download link: https://cpan.metacpan.org/authors/id/A/AT/ATOOMIC/TimeDate-2.31.tar.gz

2) Set env variables

FORCE_UNSAFE_CONFIGURE=1
BUILDTOOLS_KEEP_GOING=1
MAKEFLAGS=-j4

3) Run buildtools

$ ./buildtools

4)Source shrc

$ . $SPEC/shrc

[Optional]To save tools as prebuilt run:

$ packagetools <name-of-tools>

5) Run install.sh

$ $SPEC/install.sh -d <where_to_install>

How to run SPEC2017

1) If you have tools for ARC in $SPEC/tools/bin, then you can run install.sh script:

./install.sh -d <destination dir>

2) Go to the directory, where you have installed SPEC with a script. and source shrc file

. ./shrc

3) Installed SPEC tools are located in $SPEC/bin/. But specmake has some problems. You can use the usual make instead. Delete specmake and create symlink to system's make

cd $SPEC/bin
rm specmake
ln -s /usr/bin/make specmake

4) SPEC checks sha512sum of tools. Count it for new specmake and replace old sum in $SPEC/TOOLS.sha512 with a new one

sha512sum $SPEC/bin/specmake

5) You should have a config to run SPEC with. One can be found in repo by a link above. Or you can take any default config and edit it. Configs are located in $SPEC/config

6) Run SPEC with appropriate config and tests list You can specify intrate, fprate or list of tests. Here is a command with a list of tests that can be executed successfully now.

runcpu --config=arc.cfg 500.perlbench_r 502.gcc_r 520.omnetpp_r 523.xalancbmk_r 525.x264_r 531.deepsjeng_r 541.leela_r 548.exchange2_r 557.xz_r

runcpu --config=arc.cfg 503.bwaves_r 508.namd_r 510.parest_r 511.povray_r 519.lbm_r 526.blender_r 527.cam4_r 544.nab_r  549.fotonik3d_r 554.roms_r

How to run a test from its executable file

Built executables files are located in SPEC/CPU/<test_name>/run/run_base_refrate_mytest-m32.000x/.
A file, that contains commands is named speccmds.cmd. Commands are located at the end of the file.
Common pattern: -o <output_file> -e <error_file> <command_for_running_test>.
There might be some of these commands.

Example

1. Go to the directory of a test:

$ cd $SPEC/CPU/541.leela_r/run/run_base_refrate_mytest-m32.0001/

2. Check commands in speccmd.cmd:

$ tail -5 speccmds.cmd
-E USER root
-r
-N C
-C /home/root/build/benchspec/CPU/541.leela_r/run/run_base_refrate_mytest-m32.0001
-o ref.out -e ref.err ../run_base_refrate_mytest-m32.0001/leela_r_base.mytest-m32 ref.sgf > ref.out 2>> ref.err

3. Run test:

$ time ../run_base_refrate_mytest-m32.0001/leela_r_base.mytest-m32 ref.sgf > ref.out 2>> ref.err

Troubleshooting

Fixed in repo

1. A problem in glob.c in make

add this fix in $SPEC/tools/src/make-4.2.1/glob/glob.c: https://stackoverflow.com/questions/51675200/install-older-version-of-gnu-make-in-ubuntu-18-04

2. Delete these lines from wildcard test in $SPEC/tools/src/make-4.2.1/tests/scripts/functions/wildcard

# TEST #5: wildcard used to verify file existence

touch('xxx.yyy');

run_make_test(q!exists: ; @echo file=$(wildcard xxx.yyy)!,
              '', "file=xxx.yyy\n");

unlink('xxx.yyy');

run_make_test(q!exists: ; @echo file=$(wildcard xxx.yyy)!,
              '', "file=\n");