Enable caching using memcached #164

ASBishop · 2023-10-27T14:20:53Z

Jira: OSP-30046

ASBishop · 2023-10-30T21:05:12Z

/retest

ASBishop · 2023-10-30T21:34:34Z

/hold

Looks like I need to update the functional tests.

ASBishop · 2023-11-01T20:45:24Z

/retest

"Error from server (NotFound): deployments.apps "openstack-operator-controller-manager" not found"

ASBishop · 2023-11-02T03:15:57Z

/retest

failed to wait for the created cluster claim to become ready

ASBishop · 2023-11-02T12:50:32Z

/retest

5 tempest failures all due to 401 Unauthorized errors when deleting a share in the teardown phase. The tests themselves all passed.

gouthampacha · 2023-11-02T17:24:09Z

/retest-required

ASBishop · 2023-11-02T19:23:53Z

/retest

Current analysis is a spurious timeout in one test triggers a cascade of false timeouts in other tests. There are ongoing upstream efforts to address these issues in tempest, and the current advice is "cross your fingers and try again."

gouthampacha · 2023-11-02T21:43:57Z

There's an issue here I think that's causing these failures:

a number of "deferred" cleanups fail; The way this works is that the tests request a project credential, and create a bunch of resources for multiple test cases to share... and these common resources are cleaned up at the end of the test class. The cleanup is done with the project credentials that were saved when the resource was created... Under normal circumstances, this is fine.. but, I think that tokens are expiring by the time the tests come around to clean up..

one of the failures indicates this from the tempest log:

2023-11-02 20:12:01.034 174 INFO tempest.lib.common.rest_client [-] Request (ShareSnapshotMetadataNegativeTest:tearDownClass): 401 DELETE https://manila-public-openstack.apps.oko-15.oooci.ccitredhat.com/v2/bffc542dde8b4d2a934dc2f6344708e6/shares/4b715c31-0237-4393-b522-fde3f16aa8c5 0.150s
2023-11-02 20:12:01.034 174 DEBUG tempest.lib.common.rest_client [-] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'x-openstack-manila-api-version': '2.78', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Thu, 02 Nov 2023 20:12:00 GMT', 'server': 'Apache', 'www-authenticate': 'Keystone uri="https://keystone-public-openstack.apps.oko-15.oooci.ccitredhat.com"', 'content-length': '114', 'content-type': 'application/json', 'set-cookie': '19c75a96fcae8ddcdc1109ace5ea16b4=6b523f58cd0e10dfdb8231e69cf2a94d; path=/; HttpOnly; Secure; SameSite=None', 'connection': 'close', 'status': '401', 'content-location': 'https://manila-public-openstack.apps.oko-15.oooci.ccitredhat.com/v2/bffc542dde8b4d2a934dc2f6344708e6/shares/4b715c31-0237-4393-b522-fde3f16aa8c5'}
        Body: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}' _log_request_full /usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py:464
2023-11-02 20:12:01.035 174 ERROR tempest.test [-] teardown of resources failed: Unauthorized
Details: {'code': 401, 'title': 'Unauthorized', 'message': 'The request you have made requires authentication.'}: tempest.lib.exceptions.Unauthorized: Unauthorized
Details: {'code': 401, 'title': 'Unauthorized', 'message': 'The request you have made requires authentication.'}
2023-11-02 20:12:01.035 174 ERROR tempest.test Traceback (most recent call last):
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/tempest/test.py", line 231, in tearDownClass
2023-11-02 20:12:01.035 174 ERROR tempest.test     teardown()
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/manila_tempest_tests/tests/api/base.py", line 1201, in resource_cleanup
2023-11-02 20:12:01.035 174 ERROR tempest.test     super(BaseSharesMixedTest, cls).resource_cleanup()
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/manila_tempest_tests/tests/api/base.py", line 183, in resource_cleanup
2023-11-02 20:12:01.035 174 ERROR tempest.test     cls.clear_resources(cls.class_resources)
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/manila_tempest_tests/tests/api/base.py", line 863, in clear_resources
2023-11-02 20:12:01.035 174 ERROR tempest.test     client.delete_share(res_id)
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/manila_tempest_tests/services/share/v2/json/shares_client.py", line 355, in delete_share
2023-11-02 20:12:01.035 174 ERROR tempest.test     resp, body = self.delete(uri, version=version)
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/manila_tempest_tests/services/share/v2/json/shares_client.py", line 85, in delete
2023-11-02 20:12:01.035 174 ERROR tempest.test     resp, body = super(SharesV2Client, self).delete(url, headers=headers,
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 339, in delete
2023-11-02 20:12:01.035 174 ERROR tempest.test     return self.request('DELETE', url, extra_headers, headers, body)
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 742, in request
2023-11-02 20:12:01.035 174 ERROR tempest.test     self._error_checker(resp, resp_body)
2023-11-02 20:12:01.035 174 ERROR tempest.test   File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 842, in _error_checker
2023-11-02 20:12:01.035 174 ERROR tempest.test     raise exceptions.Unauthorized(resp_body, resp=resp)
2023-11-02 20:12:01.035 174 ERROR tempest.test tempest.lib.exceptions.Unauthorized: Unauthorized
2023-11-02 20:12:01.035 174 ERROR tempest.test Details: {'code': 401, 'title': 'Unauthorized', 'message': 'The request you have made requires authentication.'}
2023-11-02 20:12:01.035 174 ERROR tempest.test 
2023-11-02 20:12:01.304 174 INFO tempest.lib.common.rest_client [req-d5d6e751-4ade-4432-a58f-28c0bda67e96 req-d5d6e751-4ade-4432-a58f-28c0bda67e96 ] Request (ShareSnapshotMetadataNegativeTest:tearDownClass): 204 DELETE https://keystone-public-openstack.apps.oko-15.oooci.ccitredhat.com/v3/users/e089ee11f54c49d69c80d0d946249008 0.269s
2023-11-02 20:12:01.305 174 DEBUG tempest.lib.common.rest_client [req-d5d6e751-4ade-4432-a58f-28c0bda67e96 req-d5d6e751-4ade-4432-a58f-28c0bda67e96 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None

Digging into this; the test requested share creation at 2023-11-02 20:06:34.930

2023-11-02 20:06:34.930 15 INFO manila.api.openstack.wsgi [None req-9a6e7e2d-625d-4028-bbc2-e81487e20c61 0e8a8e1b67504046a9f0258119b05f4e bffc542dde8b4d2a934dc2f6344708e6 - - default default] POST https://manila-public-openstack.apps.oko-15.oooci.ccitredhat.com/v2/bffc542dde8b4d2a934dc2f6344708e6/shares

and cleanup was attempted at 2023-11-02 20:12:01.035 - that's a difference of 05:26.105 (m:s) ...
The default TTL for the cache is 300s (token_cache_time)... so manila API would need to re-validate the token; and that seems to be failing. Should we try changing this -i.e., allow operators to configure this value - it'll for one, stabilize the CI i think; we can test it out.

ASBishop · 2023-11-06T19:32:04Z

templates/manila/config/00-config.conf

 auth_type = password
 project_domain_name = Default
 user_domain_name = Default
 project_name = service
 username = {{ .ServiceUser }}
 password = {{ .ServicePassword }}
 interface = internal
+token_cache_time = 600


WIP: This is just to test whether tempest tests pass when the token timeout is increased.

gouthampacha

/retest-required

gouthampacha · 2023-11-06T22:00:00Z

Three auth failures on the last run that resulted in two cleanup failures in the last run.. re-running the job to see if these repeat.

2023-11-06 20:21:10.288 17 WARNING keystonemiddleware.auth_token [None req-12246e36-4743-4a36-91f2-ecbca90c274e 008a357059784e6ca91079bac852312d 561e0ebd3a2f47c2bf02ef8318158c39 - - default default] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed

gouthampacha · 2023-11-06T23:45:55Z

/retest-required

image pull error:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openstack-k8s-operators_manila-operator/164/pull-ci-openstack-k8s-operators-manila-operator-main-manila-operator-build-deploy-tempest/1721646297523425280/artifacts/manila-operator-build-deploy-tempest/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-87cb35e74bf88873ba52c0886f7e63e40c72d7c935590f767c3f1e099f818f7a/namespaces/openstack-operators/pods/openstack-operator-controller-manager-59fcc68f8d-vqrxl/openstack-operator-controller-manager-59fcc68f8d-vqrxl-describe

fmount · 2023-11-13T17:00:37Z

/retest

templates/manila/config/00-config.conf

Jira: OSP-30046

gouthampacha · 2023-11-13T20:09:34Z

/retest-required

ASBishop · 2023-11-14T04:00:24Z

/unhold

fmount

/lgtm

abays

/lgtm

openshift-ci · 2023-11-14T10:35:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, ASBishop, fmount

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abays,fmount]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fmount · 2023-11-14T11:40:54Z

/test manila-operator-build-deploy-tempest

ASBishop requested a review from fmount October 27, 2023 14:20

openshift-ci bot requested review from gouthampacha and stuggi October 27, 2023 14:20

openshift-ci bot added the do-not-merge/hold label Oct 30, 2023

ASBishop force-pushed the memcached branch 3 times, most recently from c73dbd2 to c2e3678 Compare November 1, 2023 17:31

ASBishop force-pushed the memcached branch from c2e3678 to d44f5a5 Compare November 6, 2023 19:29

ASBishop commented Nov 6, 2023

View reviewed changes

gouthampacha reviewed Nov 6, 2023

View reviewed changes

ASBishop force-pushed the memcached branch from d44f5a5 to 95a72f0 Compare November 9, 2023 22:34

gouthampacha reviewed Nov 13, 2023

View reviewed changes

templates/manila/config/00-config.conf Outdated Show resolved Hide resolved

Enable caching using memcached

9831c6e

Jira: OSP-30046

ASBishop force-pushed the memcached branch from 95a72f0 to 9831c6e Compare November 13, 2023 18:12

openshift-ci bot removed the do-not-merge/hold label Nov 14, 2023

fmount approved these changes Nov 14, 2023

View reviewed changes

openshift-ci bot assigned fmount Nov 14, 2023

openshift-ci bot added lgtm approved labels Nov 14, 2023

abays approved these changes Nov 14, 2023

View reviewed changes

openshift-ci bot assigned abays Nov 14, 2023

openshift-merge-bot bot merged commit 250f67f into openstack-k8s-operators:main Nov 14, 2023
1 check passed

ASBishop deleted the memcached branch November 14, 2023 16:55

fmount mentioned this pull request Feb 26, 2024

Enable caching using memcached openstack-k8s-operators/glance-operator#462

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable caching using memcached #164

Enable caching using memcached #164

ASBishop commented Oct 27, 2023 •

edited by openshift-ci bot

Loading

ASBishop commented Oct 30, 2023

ASBishop commented Oct 30, 2023

ASBishop commented Nov 1, 2023

ASBishop commented Nov 2, 2023

ASBishop commented Nov 2, 2023

gouthampacha commented Nov 2, 2023

ASBishop commented Nov 2, 2023

gouthampacha commented Nov 2, 2023 •

edited

Loading

ASBishop Nov 6, 2023

gouthampacha left a comment

gouthampacha commented Nov 6, 2023

gouthampacha commented Nov 6, 2023

fmount commented Nov 13, 2023

gouthampacha commented Nov 13, 2023

ASBishop commented Nov 14, 2023

fmount left a comment

abays left a comment

openshift-ci bot commented Nov 14, 2023

fmount commented Nov 14, 2023

Enable caching using memcached #164

Enable caching using memcached #164

Conversation

ASBishop commented Oct 27, 2023 • edited by openshift-ci bot Loading

ASBishop commented Oct 30, 2023

ASBishop commented Oct 30, 2023

ASBishop commented Nov 1, 2023

ASBishop commented Nov 2, 2023

ASBishop commented Nov 2, 2023

gouthampacha commented Nov 2, 2023

ASBishop commented Nov 2, 2023

gouthampacha commented Nov 2, 2023 • edited Loading

ASBishop Nov 6, 2023

Choose a reason for hiding this comment

gouthampacha left a comment

Choose a reason for hiding this comment

gouthampacha commented Nov 6, 2023

gouthampacha commented Nov 6, 2023

fmount commented Nov 13, 2023

gouthampacha commented Nov 13, 2023

ASBishop commented Nov 14, 2023

fmount left a comment

Choose a reason for hiding this comment

abays left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 14, 2023

fmount commented Nov 14, 2023

ASBishop commented Oct 27, 2023 •

edited by openshift-ci bot

Loading

gouthampacha commented Nov 2, 2023 •

edited

Loading