Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mmtests update #469

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open

mmtests update #469

wants to merge 24 commits into from

Conversation

manul7
Copy link

@manul7 manul7 commented Oct 11, 2023 •
edited
Loading

I've updated shell script for launching mmtests as well as YAML config. New version gets multiple fixes and new option to control mmtest execution.

Tests

  1. Clean SUT, no MMTests overlay installed

Actions

  1. Launch ./mmtests.sh -p /mmtests -c configs/config-db-sqlite-insert-small -r 3 -i 1 -s

Expected result

...
INFO: Installation skipped
+ pushd /mmtests
./mmtests.sh: line 283: pushd: /mmtests: No such file or directory
+ exit 1
  1. Clean SUT, no MMTests overlay installed

Actions

  1. Launch ./mmtests.sh -p /mmtests -c configs/config-db-sqlite-insert-small -r 3 -i 1

Expected results

  1. Sys deps installed
  2. Perl deps installed
  3. MMTests source code checked out into /mmtests
  4. Target benchmark built
  5. Benchmark launched 1 time
  6. Results collected into /mmtests/output
...+ mv /tmp/tmp.aVWr6ZuULI /mmtests/output/BENCHMARKsqlite_CONFIGdb-sqlite-insert-small.json
  1. mmtests installed, no MMTests overlay installed

Actions

  1. Launch ./mmtests.sh -p /mmtests -c configs/config-db-sqlite-insert-small -r 3 -i 3 -s

Expected results

  1. Deps installation skipped
  2. Source code checkout skipped
  3. Benchmark launched 3 times
  4. Collects results into /mmtests/output
+ mv /tmp/tmp.xHGS3Xh0sx /mmtests/output/BENCHMARKsqlite_CONFIGdb-sqlite-insert-small.json
  1. 3 iterations in /mmtests/work/log/config-db-sqlite-insert-small/
root@4d19d5392b66:/opt/linux/mmtests# ls /mmtests/work/log/config-db-sqlite-insert-small/
iter-0  iter-1  iter-2

@manul7 manul7 changed the title mmtests update WIP: mmtests update Oct 12, 2023
@manul7 manul7 force-pushed the master branch 6 times, most recently from 7095805 to fe2b2af Compare October 12, 2023 15:00
@manul7 manul7 changed the title WIP: mmtests update mmtests update Oct 12, 2023
@roxell
Copy link
Collaborator

roxell commented Oct 12, 2023

patch 77113cf ("mmtests.yml: parameters update") has multple changes that can be splitted up into multiple commits maybe or?

@manul7
Copy link
Author

manul7 commented Oct 13, 2023

patch 77113cf ("mmtests.yml: parameters update") has multple changes that can be splitted up into multiple commits maybe or?

Sure, np. Done

@manul7 manul7 force-pushed the master branch 2 times, most recently from 5ed0be3 to cbbcbfd Compare October 13, 2023 13:41
roxell
roxell reviewed Oct 17, 2023
View reviewed changes
automated/linux/mmtests/mmtests.yaml Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.yaml Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
automated/linux/mmtests/mmtests.sh Show resolved Hide resolved
@manul7 manul7 force-pushed the master branch 2 times, most recently from a24ed3d to a8d9917 Compare October 17, 2023 11:52
@manul7 manul7 force-pushed the master branch 9 times, most recently from 0b2bf52 to ea25796 Compare October 21, 2023 20:16
@manul7 manul7 changed the title mmtests update WIP: mmtests update Oct 22, 2023
@manul7 manul7 force-pushed the master branch 2 times, most recently from cf4e098 to 38bcda8 Compare October 26, 2023 15:21
@manul7 manul7 changed the title WIP: mmtests update mmtests update Oct 26, 2023
@roxell roxell marked this pull request as ready for review August 2, 2024 05:36
@manul7 manul7 force-pushed the master branch 2 times, most recently from fc543d9 to 0d587c3 Compare November 7, 2024 11:45
Konstantin Belov added 23 commits November 26, 2024 12:51
automated: linux: mmtests: removed results processing
172367a 
Removed all parts related to results extraction and processing.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored parameters
d240fc9 
Moved parameters set up logic after script arguments processing.
This allows to calculate required parameters according arguments.
Added default values to parameters.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored perl deps installation
eedd89a 
Moved MMTest Perl dependencies installation into separate function.
This allows to control these deps independently from system packages

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored prepare_system
7df6d8b 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored run_test
e633c30 
Refactored function sets MMTEST_ITERATIONS, disables packages
installation before benchmarking procedure, increases priority.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored installation phase
ec96a8a 
Now operations, which required only while preparation are grouped
under installation branch and can be skipped if system has all
dependencies and binaries.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: refactored help
e8457e4 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: add results collection
07e1b7b 
Now, by default MMTests results remain untouched after test run.
To collect results and system information, use -k option.

Added helper script collect.py, which collects system information, environment
variables, extracts results from raw MMTests results. Extracted and collected
information stored in output directory in JSON format.

If needed full archive with raw MMTests results can be created with -f option.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: test definition update
2835dc0 
Test definition was updated to accommodate recent changes in
mmtests.sh.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: results retrival
4805d5a 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: new Perl dependency
02e5d7e 
Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
8ed9955 
Added SHA256 collection for kernel binary
Refactored kernel info func

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
8c2be10 
Added cloud instance type collection

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
cc7d7cf 
Added benchmark's archive SHA256 collection

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: collector update
bbf377a 
Packages list simplified

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: main script update
fd22461 
Removed explicit 'set +x' command to reduce output.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: added monitors control arg
9e06599 
Added new argument -m to control monitors function
while benchmarking.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: added results dir arg
7053a1e 
Added new argument -w to control output dir location.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: checks & logs
e8f0050 
Added additional check for config presence
Added clearer log messages

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: tasksel
490eaec 
Added tasksel call for benchmark launch to
bind benchmark to single cpu. This allows to reduce
variance of the results.

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: control file update
598cde1 
Added monitors control in test definition file

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: reworked parse_boottime
feed9dd 
New implementation is more robust and covers s, ms, us

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: results collector
0b62b86 
Switched results collector to standard logging
Improved missing not critical files handling

Signed-off-by: Konstantin Belov <[email protected]>
automated: linux: mmtests: added override feature
4c415c5 
Sometimes benchmarks used in MMTests require configuration tweaks.
This can be done with manual config file edits or can be done with
override feature, which takes a string in format
"KEY1=VAL1;KEY2=VAL2;..." and replaces values for required keys
automatically. Usually such tweaks are required to control the number
of iterations and execution time on benchmark level, not on MMTests
level (with MMTEST_ITERATIONS). For example, for aim9 benchmark
user can specify "AIM9_TESTTIME=5;AIM9_TESTLIST=page_test" to
reduce test time and use only one test instead of all.

Signed-off-by: Konstantin Belov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roxell roxell roxell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@manul7 @roxell