Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ResourceLimits for test pods #253

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add ResourceLimits for test pods #253

wants to merge 1 commit into from
+975 −4

Conversation

lpiwowar
Copy link
Collaborator

@lpiwowar lpiwowar commented Nov 25, 2024 •
edited
Loading

The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 25, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 25, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lpiwowar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lpiwowar
lpiwowar commented Nov 25, 2024
View reviewed changes
Copy link
Collaborator Author

@lpiwowar lpiwowar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eshulman2, @eduolivares, @ashu-011: I would be really grateful ❤️ for your input here. This PR is about setting sane default resource limits [1] for the test pods. The limits are going to be different for each framework therefore I need an estimation from each of you, if possible:).

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

api/v1beta1/horizontest_types.go
// The desired amount of resources that should be assigned to each test pod
// spawned using the HorizonTest CR. https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements
// +kubebuilder:default:={limits: {cpu: "2000m", memory: "4Gi"}, requests: {cpu: "1000m", memory: "2Gi"}}
Resources corev1.ResourceRequirements `json:"resources,omitempty"`
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashu-011, can you please 🙏 suggest the correct Resource Limits for the horizon test pods? It will be possible to override the default value in the HorizonTest CR.

api/v1beta1/tobiko_types.go Show resolved Hide resolved
api/v1beta1/ansibletest_types.go Show resolved Hide resolved
@lpiwowar lpiwowar force-pushed the feature/resource-limit branch 2 times, most recently from 8827409 to 5147c15 Compare November 26, 2024 16:50
lpiwowar added a commit to openstack-k8s-operators/ci-framework that referenced this pull request Dec 18, 2024
@lpiwowar
[Test-operator] Set resource limits
cdcfd6c 
This PR exposes the resources parameter in for all test-operator
related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter
can be used to specify amount of resources the test pods spawned by the
test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as
the propagation of the fix for Tempest memory leak bug [1] did not
reach the upstream openstack-tempest-all image yet. Once the fix is
in place the default value of cifmw_test_operator_tempest_resources
can be changed to {}.

[1] openstack-k8s-operators/test-operator#253
@lpiwowar lpiwowar mentioned this pull request Dec 18, 2024
Draft
@lpiwowar lpiwowar force-pushed the feature/resource-limit branch from 5147c15 to 8e66c54 Compare December 18, 2024 14:51
@softwarefactory-project-zuul softwarefactory-project-zuul
Copy link

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#2617 is needed.

@lpiwowar
Copy link
Collaborator Author

recheck

@lpiwowar lpiwowar mentioned this pull request Dec 18, 2024
Closed
lpiwowar added a commit to openstack-k8s-operators/ci-framework that referenced this pull request Dec 19, 2024
@lpiwowar
[Test-operator] Set resource limits
356c19b 
This PR exposes the resources parameter in for all test-operator
related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter
can be used to specify amount of resources the test pods spawned by the
test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as
the propagation of the fix for Tempest memory leak bug [1] did not
reach the upstream openstack-tempest-all image yet. Once the fix is
in place the default value of cifmw_test_operator_tempest_resources
can be changed to {}.

[1] openstack-k8s-operators/test-operator#253
@lpiwowar
Copy link
Collaborator Author

/test all

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Dec 19, 2024

@lpiwowar: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/precommit-check 8e66c54 link true /test precommit-check

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@lpiwowar lpiwowar force-pushed the feature/resource-limit branch from 8e66c54 to 9a14c77 Compare December 19, 2024 10:54
@lpiwowar
Add ResourceLimits for test pods
b157f44 
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
@lpiwowar lpiwowar force-pushed the feature/resource-limit branch from 9a14c77 to b157f44 Compare December 19, 2024 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
approved do-not-merge/work-in-progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@lpiwowar