Baremetal e2e scripts #248

mukrishn · 2021-09-15T18:54:37Z

Description

Modifications to run multiple benchmark resources for baremetal cluster, so included conditional checks for operators.
New workload for baremetal upgrade, that would bring up multiple application pods, services and active http traffic between them while upgrade.

Fixes

Developed by @jdowni000, I just fixed few things and corrected conflicts as he is on PTO. PR #225 can be marked as obsolete.

* Updated workloads to run on baremetal * first commit: skipping cleanup on baremetal * first commit: skipping cleanup on baremetal * fixed spacing * Checking if cloud is on bareMetal. If it is, we're no longer deleting the benchmark-operator namespace in between runs Co-authored-by: jdowni000 <[email protected]>

* first commit: adding calculations for allocatable CPU per MCP * Changing approach to creating new n number mcps depending on node count * removing old machineConfig_pool func * removing bash script testing code left in common.sh * fixed a couple naming convention issues * small fixes from testing in actual cluster * fixed applying mcp.yaml to use envsubst * more issues resolved from testing * more fixes in while loops while testing * completed deployment fix * changing function name, adding check for mcps and ns if exists * adding json creation and use of mb * adding logic to allow MCP_SIZE and MCP_NODE_COUNT to be set as a variable * changing env var MCP_SIZE to TOTAL_MCPS * fixed spacing in all of new function * adding mb_pod.yml and sending mb operations to pod instead of cli * removing response.csv file as it may be too large depending on how many mcps are genereated * added var to set to choose whether MCPs are created or not * just fix...its Friday...its fixed * fixed if statemnet with elif * creating logic to check env var inputs * adding if condition for sleep depending on size of node_count * adding resources to mb-pod * adding logic to check for sample app pods to be ready before mb-pod deployment * Update common.sh mb-pod and sample app

mohit-sheth · 2021-09-15T19:21:21Z

rerun all

mohit-sheth · 2021-09-15T19:21:29Z

/rerun all

comet-perf-ci · 2021-09-15T20:29:36Z

Results for e2e-benchmarking CI Tests

Workload	Test	Result	Runtime
router-perf-v2	ingress-performance.sh	FAIL	00:04:17
etcd-perf	run_etcd_tests_fromgit.sh	PASS	00:02:42
scale-perf	run_scale_fromgit.sh	FAIL	00:17:23
network-perf	smoke_test.sh	PASS	00:07:23
kube-burner	run_clusterdensity_test_fromgit.sh	PASS	00:07:32
kube-burner	run_maxnamespaces_test_fromgit.sh	PASS	00:03:54
kube-burner	run_maxservices_test_fromgit.sh	PASS	00:03:00
kube-burner	run_nodedensity_test_fromgit.sh	PASS	00:03:28
kube-burner	run_nodedensity-heavy_test_fromgit.sh	PASS	00:04:12
storage-perf	run_storage_tests_fromgit.sh	PASS	00:02:35
upgrade-perf	run_upgrade_fromgit.sh	PASS	00:00:56

smalleni · 2021-09-15T21:14:42Z

workloads/network-perf/common.sh

+
+  #If using baremetal we use different query to find worker nodes
+  if [[ "${isBareMetal}" == "true" ]]; then
+    log "Colocating uperf pods for baremetal"


We are not really colocating are we?

right, we are not. we randomly pick 2 worker nodes. I will delete this log.

whitleykeith · 2021-09-15T21:29:24Z

workloads/kube-burner/common.sh

-  kubectl apply -f benchmark-operator/resources/kube-burner-role.yml
-  log "Waiting for benchmark-operator to be running"
-  oc wait --for=condition=available "deployment/benchmark-controller-manager" -n benchmark-operator --timeout=300s
+  if [[ "${isBareMetal}" == "false" ]]; then


tbh i think we should just leave the operator around each time and change this to apply to all cluster types. I don't see a reason why we need to delete/recreate the operator

Once the operator is deployed, we shouldn't delete it.
I'd modify the current code to always try to deploy the operator. With the current make deploy implementation, if the operator is already running it won't be redeployed.

Removed delete command

workloads/network-perf/common.sh

workloads/kube-burner/common.sh

mukrishn · 2021-09-24T14:38:21Z

@mohit-sheth @rsevilla87 @whitleykeith any more suggestions/comments ?

whitleykeith

lgtm

mohit-sheth · 2021-10-12T19:23:10Z

/rerun all

comet-perf-ci · 2021-10-12T20:37:56Z

Results for e2e-benchmarking CI Tests

Workload	Test	Result	Runtime
router-perf-v2	ingress-performance.sh	FAIL	00:06:05
etcd-perf	run_etcd_tests_fromgit.sh	PASS	00:03:26
scale-perf	run_scale_fromgit.sh	PASS	00:16:01
network-perf	smoke_test.sh	PASS	00:05:12
kube-burner	run_clusterdensity_test_fromgit.sh	PASS	00:06:05
kube-burner	run_maxnamespaces_test_fromgit.sh	PASS	00:04:00
kube-burner	run_maxservices_test_fromgit.sh	PASS	00:02:58
kube-burner	run_poddensity-heavy_test_fromgit.sh	PASS	00:04:01
kube-burner	run_nodedensity_test_fromgit.sh	PASS	00:04:35
kube-burner	run_nodedensity-heavy_test_fromgit.sh	PASS	00:04:52
upgrade-perf	run_upgrade_fromgit.sh	PASS	00:00:56

mohit-sheth

LGTM

mukrishn · 2021-10-26T17:32:32Z

@rsevilla87 fixed conflicts and tested, ptal.

workloads/logging/common.sh

workloads/kube-burner/common.sh

Signed-off-by: Raul Sevilla <[email protected]>

rsevilla87 · 2021-10-28T10:04:53Z

/rerun all

comet-perf-ci · 2021-10-28T10:49:11Z

Results for e2e-benchmarking CI Tests

Workload	Test	Result	Runtime
router-perf-v2	ingress-performance.sh	FAIL	00:01:07
etcd-perf	run_etcd_tests_fromgit.sh	PASS	00:02:44
scale-perf	run_scale_fromgit.sh	PASS	00:12:54
network-perf	smoke_test.sh	PASS	00:05:53
kube-burner	run_clusterdensity_test_fromgit.sh	PASS	00:03:12
kube-burner	run_maxnamespaces_test_fromgit.sh	PASS	00:03:02
kube-burner	run_maxservices_test_fromgit.sh	PASS	00:02:14
kube-burner	run_poddensity-heavy_test_fromgit.sh	PASS	00:03:34
kube-burner	run_nodedensity_test_fromgit.sh	PASS	00:03:04
kube-burner	run_nodedensity-heavy_test_fromgit.sh	PASS	00:03:47
upgrade-perf	run_upgrade_fromgit.sh	PASS	00:00:55

rsevilla87 · 2021-10-28T11:15:59Z

/rerun all router-perf-v2:ingress-performance.sh

comet-perf-ci · 2021-10-28T11:27:12Z

Results for e2e-benchmarking CI Tests

Workload	Test	Result	Runtime
router-perf-v2	ingress-performance.sh	PASS	00:09:42

Changes for baremetal Co-authored-by: jdowni000 <[email protected]> Co-authored-by: Marko Karg <[email protected]> Co-authored-by: jdowni000 <[email protected]> Co-authored-by: Raul Sevilla <[email protected]>

jdowni000 and others added 16 commits August 10, 2021 16:21

changes for baremetal

b760183

exiting if benchmark state becomes Failed

fa36045

Merge branch 'master' of https://github.com/jdowni000/e2e-benchmarking

e093af1

removing duplicated check for baremetal from PR

443d56a

adding log() for kube-burner

aa2f5a9

to pick 1 client pod (cloud-bulldozer#6)

b208b54

fixed conflicts

398719f

Merge branch 'master' into master

320994b

Merge branch 'master' into master

eb5d646

moved script to upgrade dir

cfaef9c

rebased and fixed conflict

77697e1

modified mc selector

9993e9b

excluding few more workers

09c3be9

static mb configuration

dc32aa5

mohit-sheth requested review from whitleykeith, mohit-sheth and rsevilla87 September 15, 2021 18:58

mohit-sheth added the ok to test Kick off our CI framework label Sep 15, 2021

smalleni reviewed Sep 15, 2021

View reviewed changes

whitleykeith suggested changes Sep 15, 2021

View reviewed changes

rsevilla87 reviewed Sep 16, 2021

View reviewed changes

workloads/kube-burner/common.sh Outdated Show resolved Hide resolved

rsevilla87 mentioned this pull request Sep 16, 2021

Baremetal #225

Closed

mukrishn added 2 commits September 16, 2021 09:17

corrected log format

8b7bc18

added logs and missing variable

4037270

mukrishn force-pushed the final_patch branch from 2580f8b to 137526c Compare September 16, 2021 15:48

removed operator delete command

fdb68a6

mukrishn force-pushed the final_patch branch from 137526c to fdb68a6 Compare September 16, 2021 15:52

mukrishn requested review from rsevilla87 and whitleykeith September 24, 2021 14:35

mukrishn added 2 commits September 24, 2021 11:19

Updated pod selector

268c5cb

removed cleanup functions

282339f

whitleykeith approved these changes Sep 27, 2021

View reviewed changes

Merge branch 'master' into final_patch

a103f2c

mohit-sheth approved these changes Oct 12, 2021

View reviewed changes

mohit-sheth requested a review from smalleni October 12, 2021 21:09

smalleni approved these changes Oct 13, 2021

View reviewed changes

mkarg75 mentioned this pull request Oct 15, 2021

Additional workloads: cyclictest, oslat and testpmd add #264

Merged

Merge branch 'master' into final_patch

f6d0276

python3.8 for ripsaw-cli

3aed0e9

mukrishn force-pushed the final_patch branch from cec0ba9 to 3aed0e9 Compare October 26, 2021 20:34

rsevilla87 reviewed Oct 28, 2021

View reviewed changes

workloads/logging/common.sh Outdated Show resolved Hide resolved

rsevilla87 reviewed Oct 28, 2021

View reviewed changes

workloads/kube-burner/common.sh Outdated Show resolved Hide resolved

rsevilla87 added 2 commits October 28, 2021 10:16

Small logging fixes

433d76d

Signed-off-by: Raul Sevilla <[email protected]>

Improve benchmark finished message

04d9156

Signed-off-by: Raul Sevilla <[email protected]>

rsevilla87 merged commit dd11ead into cloud-bulldozer:master Oct 28, 2021

rsevilla87 self-requested a review October 28, 2021 11:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Baremetal e2e scripts #248

Baremetal e2e scripts #248

mukrishn commented Sep 15, 2021

mohit-sheth commented Sep 15, 2021

mohit-sheth commented Sep 15, 2021

comet-perf-ci commented Sep 15, 2021

smalleni Sep 15, 2021

mukrishn Sep 16, 2021

whitleykeith Sep 15, 2021

mukrishn Sep 16, 2021 •

edited

Loading

rsevilla87 Sep 16, 2021 •

edited

Loading

mukrishn Sep 16, 2021

mukrishn commented Sep 24, 2021

whitleykeith left a comment

mohit-sheth commented Oct 12, 2021

comet-perf-ci commented Oct 12, 2021

mohit-sheth left a comment

mukrishn commented Oct 26, 2021

rsevilla87 commented Oct 28, 2021

comet-perf-ci commented Oct 28, 2021

rsevilla87 commented Oct 28, 2021

comet-perf-ci commented Oct 28, 2021

Baremetal e2e scripts #248

Baremetal e2e scripts #248

Conversation

mukrishn commented Sep 15, 2021

Description

Fixes

mohit-sheth commented Sep 15, 2021

mohit-sheth commented Sep 15, 2021

comet-perf-ci commented Sep 15, 2021

smalleni Sep 15, 2021

Choose a reason for hiding this comment

mukrishn Sep 16, 2021

Choose a reason for hiding this comment

whitleykeith Sep 15, 2021

Choose a reason for hiding this comment

mukrishn Sep 16, 2021 • edited Loading

Choose a reason for hiding this comment

rsevilla87 Sep 16, 2021 • edited Loading

Choose a reason for hiding this comment

mukrishn Sep 16, 2021

Choose a reason for hiding this comment

mukrishn commented Sep 24, 2021

whitleykeith left a comment

Choose a reason for hiding this comment

mohit-sheth commented Oct 12, 2021

comet-perf-ci commented Oct 12, 2021

mohit-sheth left a comment

Choose a reason for hiding this comment

mukrishn commented Oct 26, 2021

rsevilla87 commented Oct 28, 2021

comet-perf-ci commented Oct 28, 2021

rsevilla87 commented Oct 28, 2021

comet-perf-ci commented Oct 28, 2021

mukrishn Sep 16, 2021 •

edited

Loading

rsevilla87 Sep 16, 2021 •

edited

Loading