Putting benchmark comparison in utils #474

paigerube14 · 2022-09-12T18:15:54Z

Description

Adding the ability to run the same benchmark_comparison functions for kube-burner, network-perf and router-perf all from the utils folder
With this I moved the touchstone-configs into the utils area so that they are reusable by the different workload types

Fixes

#395
#428

comet-perf-ci · 2022-09-12T18:17:06Z

Can one of the admins verify this patch?

paigerube14 · 2022-09-12T19:07:55Z

This PR doesn't cover all workload types. I can completely take out the references to run-compare.sh file but I'm not familiar with the other workload types and might be good to do in separate PR, thoughts?

rsevilla87 · 2022-09-14T08:27:03Z

utils/touchstone-configs/uperf-touchstone-norm.json

@@ -0,0 +1,28 @@
+{


Why are you adding this config?, norm_ops comparison is already covered by uperf-touchstone.json

I didn't mean to add that there, I wanted to keep the uperf-touchstone.json file as is and add in other files that added in other data that used to be reported to help resolve: #428
I removed it now

rsevilla87 · 2022-09-14T08:33:15Z

utils/compare.sh

+  fi
+}
+
+run_benchmark_comparison() {


The original logic from networking and router tests was different. This is adding the kube-burner's comparison which I think won't work

I added a back up way to return the data in utils/csv_modifier.py that I think should cover this. I will run the old way and with my updated code and verify that the data is the same in the sheet.

@rsevilla87 (or whoever else is going to review this PR) can you request access to these sheets and I'm only able to give access to everyone with the link which I'm not sure we want that.
I used the same UUID and generated both the below sheets
Old sheet: https://docs.google.com/spreadsheets/d/1Xv1LEJkwk9xnKifRTy7GbmBwNl6Wpkrb8lGRfBBVeq0/edit#gid=1904663657

New sheet (added more data points at bottom): https://docs.google.com/spreadsheets/d/1TAG5Fw4F7uppNgH6UsltwwI8G034CZ5zB2TFqgSF9Bg/edit#gid=2039516611
I used export COMPARISON_CONFIG="uperf-touchstone-stream.json uperf-touchstone-rr.json uperf-touchstone-norm.json" to generate this sheet

Any advice on how to order the data in the sheet. That's the one functionality I seem to lose

Added environment variable to not sort by the end column value that was reordering the network-perf data

ack, thanks
can you add the updated results sheet here if you have one

@mohit-sheth newest sheet https://docs.google.com/spreadsheets/d/172Tj5MXbjsSAMEBP4MeEIcmyIDadWnQ9LIGYpNfnWlg/edit?usp=sharing

nathan-weinberg · 2022-10-03T18:44:56Z

Commenting for updates as I will have a followup PR after this, happy to review/help in any other way I can as well to get this in

paigerube14 · 2022-10-04T14:06:15Z

@mohit-sheth @rsevilla87 @jtaleric can you PTAL when you get a chance

paigerube14 · 2022-10-04T16:39:01Z

Comment from @mohit-sheth: now added bytes to be transformed into Mb

used below to generate newest sheet

export COMPARISON_CONFIG="uperf-touchstone-stream.json uperf-touchstone-rr.json uperf-touchstone-norm-ltcy.json uperf-touchstone-norm-ops.json"

mohit-sheth

lgtm, we can split the num_pairs in future PRs

nathan-weinberg · 2022-10-04T18:41:39Z

@mohit-sheth can we merge?

paigerube14 · 2022-10-04T19:16:04Z

This also helps take care of data issues like the below

10-04 14:19:59.936  Traceback (most recent call last):
10-04 14:19:59.936    File "/home/jenkins/ws/workspace/aige-e2e-multibranch_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 70, in <module>
10-04 14:19:59.936      main()
10-04 14:19:59.936    File "/home/jenkins/ws/workspace/aige-e2e-multibranch_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 61, in main
10-04 14:19:59.936      df[df.columns[select_columns:num_columns]] = df[
10-04 14:19:59.936    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/frame.py", line 9558, in apply
10-04 14:19:59.936      return op.apply().__finalize__(self, method="apply")
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/apply.py", line 741, in apply
10-04 14:19:59.937      return self.apply_standard()
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/apply.py", line 868, in apply_standard
10-04 14:19:59.937      results, res_index = self.apply_series_generator()
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/apply.py", line 884, in apply_series_generator
10-04 14:19:59.937      results[i] = self.f(v)
10-04 14:19:59.937    File "/home/jenkins/ws/workspace/aige-e2e-multibranch_kube-burner/workloads/kube-burner/../../utils/csv_modifier.py", line 41, in divide_col
10-04 14:19:59.937      return round(x / float((div_by)), 3)
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/ops/common.py", line 72, in new_method
10-04 14:19:59.937      return method(self, other)
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/arraylike.py", line 127, in __truediv__
10-04 14:19:59.937      return self._arith_method(other, operator.truediv)
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/series.py", line 6262, in _arith_method
10-04 14:19:59.937      return base.IndexOpsMixin._arith_method(self, other, op)
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/base.py", line 1325, in _arith_method
10-04 14:19:59.937      result = ops.arithmetic_op(lvalues, rvalues, op)
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/ops/array_ops.py", line 226, in arithmetic_op
10-04 14:19:59.937      res_values = _na_arithmetic_op(left, right, op)  # type: ignore[arg-type]
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/ops/array_ops.py", line 172, in _na_arithmetic_op
10-04 14:19:59.937      result = _masked_arith_op(left, right, op)
10-04 14:19:59.937    File "/tmp/tmp.GKFIDKNeJ2/lib/python3.9/site-packages/pandas/core/ops/array_ops.py", line 129, in _masked_arith_op
10-04 14:19:59.937      result[mask] = op(xrav[mask], y)
10-04 14:19:59.937  TypeError: unsupported operand type(s) for /: 'str' and 'float'

afcollins · 2022-10-05T17:05:09Z

workloads/network-perf/common.sh


-openshift_login
+openshift_login 


extra space at end of line can be removed.

should be removed now

paigerube14 · 2022-10-06T12:06:03Z

@afcollins @rsevilla87 any comments left on this pr, can we get this merged? Thanks

rsevilla87

lgtm

paigerube14 requested review from mohit-sheth and rsevilla87 and removed request for mohit-sheth September 12, 2022 18:15

paigerube14 force-pushed the compare_combine branch from 70226db to 2aa15ba Compare September 12, 2022 18:18

jtaleric added the ok to test Kick off our CI framework label Sep 12, 2022

paigerube14 force-pushed the compare_combine branch from 367ec1b to 41e3338 Compare September 12, 2022 18:26

paigerube14 force-pushed the compare_combine branch from 4576e82 to a97f5bd Compare September 12, 2022 19:45

rsevilla87 reviewed Sep 14, 2022

View reviewed changes

paigerube14 force-pushed the compare_combine branch 2 times, most recently from 9ba44f4 to 8d715e6 Compare September 14, 2022 18:42

paigerube14 force-pushed the compare_combine branch from 5cd2752 to f36d8ab Compare October 3, 2022 15:30

paigerube14 force-pushed the compare_combine branch from f36d8ab to 99cf0e7 Compare October 4, 2022 14:04

mohit-sheth self-requested a review October 4, 2022 16:45

mohit-sheth approved these changes Oct 4, 2022

View reviewed changes

paigerube14 mentioned this pull request Oct 4, 2022

Separate out pairs data per worksheet in network-perf #482

Open

afcollins reviewed Oct 5, 2022

View reviewed changes

paigerube14 force-pushed the compare_combine branch from 89dd3ff to 25b1126 Compare October 5, 2022 17:15

putting benchmark comparison in utils

8223a4f

paigerube14 force-pushed the compare_combine branch from 72c8232 to 8223a4f Compare October 5, 2022 17:19

nathan-weinberg mentioned this pull request Oct 6, 2022

Added spreadsheet generation via workload jobs openshift-qe/ocp-qe-perfscale-ci#283

Merged

rsevilla87 approved these changes Oct 6, 2022

View reviewed changes

rsevilla87 merged commit 6e6dc1c into cloud-bulldozer:master Oct 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Putting benchmark comparison in utils #474

Putting benchmark comparison in utils #474

paigerube14 commented Sep 12, 2022

comet-perf-ci commented Sep 12, 2022

paigerube14 commented Sep 12, 2022

rsevilla87 Sep 14, 2022

paigerube14 Sep 14, 2022 •

edited

Loading

rsevilla87 Sep 14, 2022

paigerube14 Sep 14, 2022

paigerube14 Sep 14, 2022 •

edited

Loading

paigerube14 Sep 14, 2022

mohit-sheth Sep 14, 2022

paigerube14 Sep 15, 2022

nathan-weinberg commented Oct 3, 2022

paigerube14 commented Oct 4, 2022

paigerube14 commented Oct 4, 2022

mohit-sheth left a comment

nathan-weinberg commented Oct 4, 2022

paigerube14 commented Oct 4, 2022

afcollins Oct 5, 2022

paigerube14 Oct 5, 2022

paigerube14 commented Oct 6, 2022

rsevilla87 left a comment


		openshift_login
		openshift_login

Putting benchmark comparison in utils #474

Putting benchmark comparison in utils #474

Conversation

paigerube14 commented Sep 12, 2022

Description

Fixes

comet-perf-ci commented Sep 12, 2022

paigerube14 commented Sep 12, 2022

Choose a reason for hiding this comment

paigerube14 Sep 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paigerube14 Sep 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nathan-weinberg commented Oct 3, 2022

paigerube14 commented Oct 4, 2022

paigerube14 commented Oct 4, 2022

mohit-sheth left a comment

Choose a reason for hiding this comment

nathan-weinberg commented Oct 4, 2022

paigerube14 commented Oct 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paigerube14 commented Oct 6, 2022

rsevilla87 left a comment

Choose a reason for hiding this comment

paigerube14 Sep 14, 2022 •

edited

Loading

paigerube14 Sep 14, 2022 •

edited

Loading