Deploy an arbitrary number of glanceAPI

[fmount]

This patch introduces a new API interface that allows to specify an arbitrary number of glanceAPI. By giving this ability from an interface point of view, the glance-operator is now responsible to reconcile the deployed instances in terms of lifecycle, as well as making sure we always have a selected API registered to the keystone image service.
As part of this change we are now able to:

• grow or shrink the API deployment
• update the keystoneEndpoint to select which instance should be registered in the keystone catalog
• propagate extraVolumes to the single instance as done for both Cinder and Manila

Demo: https://asciinema.org/a/625117

[fmount]

/retest-required

[fmount]

I think there's a valid issue in tempest and it might be related to the endpoint creation:

2023-12-05 23:01:10.201 130 WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5c2f9fa160>: Failed to establish a new connection: [Errno -2] Name or service not known')': /�[00m
2023-12-05 23:01:10.205 130 WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5c2f9fa5e0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /�[00m
2023-12-05 23:01:10.209 130 WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5c2f9fa700>: Failed to establish a new connection: [Errno -2] Name or service not known')': /�[00m
2023-12-05 23:01:10.217 130 ERROR config_tempest.constants [-] Request on service 'image' with url 'http://glance-default-external-public.openstack.svc:9292' failed: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='glance-default-external-public.openstack.svc', port=9292): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5c2f9fa880>: Failed to establish a new connection: [Errno -2] Name or service not known'))�[00m
2023-12-05 23:01:10.220 130 CRITICAL tempest [-] Unhandled error: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='glance-default-external-public.openstack.svc', port=9292): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5c2f9fa880>: Failed to establish a new connection: [Errno -2] Name or service not known'))
2023-12-05 23:01:10.220 130 ERROR tempest Traceback (most recent call last):
2023-12-05 23:01:10.220 130 ERROR tempest   File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
2023-12-05 23:01:10.220 130 ERROR tempest     conn = connection.create_connection(
2023-12-05 23:01:10.220 130 ERROR tempest   File "/usr/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
2023-12-05 23:01:10.220 130 ERROR tempest     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
2023-12-05 23:01:10.220 130 ERROR tempest   File "/usr/lib64/python3.9/socket.py", line 954, in getaddrinfo
2023-12-05 23:01:10.220 130 ERROR tempest     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2023-12-05 23:01:10.220 130 ERROR tempest socket.gaierror: [Errno -2] Name or service not known
2023-12-05 23:01:10.220 130 ERROR tempest 
2023-12-05 23:01:10.220 130 ERROR tempest During handling of the above exception, another exception occurred:

[fmount]

/retest-required

[fmount]

/test glance-operator-build-deploy-tempest

[fmount]

/retest

[fmount]

Despite I'm still trying to get the CI green on PR#587 and predict any potential failure
with patch [2], I think this change it's ready to go.
I tested with and without network isolation, with multiple instances deployed through
the openstack-operator (see [1]). As an additional test, I deployed multiple ceph
clusters (in the Pod form [4]) and use extraVolumes to attach each of them to
a different glanceAPI.

As the demo [1] shows:

- api0 => upload an image => ceph0
- patch the `openstackcontrolplane` to switch endpoint to api1
- api1 => upload an image => ceph1

The three steps above are basically the common part of the tests I made. I'm going
to create a doc patch as a follow up to better describe the design choices and how
the whole flow works, for both the glanceAPI lifecycle and the endpoints management.
As a follow up, we might add more conditions for the edge type API, so that we
can skip any public endpoint creation (even if not exposed through a Route).
By doing this we can prevent them to have the ability to be registered in keystone.

@abays if you are ok with this patch, once is approved, we should rebase/merge [2]
to avoid issues and inconsistencies with the API.
Both kuttl and tempest test prove that the covered use cases work as expected,
(see also the Prow job logs in [2]).
I'll keep the do-not-merge/hold label to block merging it until you've checked [2].
(that of course should be rebased on this one, while right now kuttl uses my custom
bundle [4] based on this commit).

[1] https://asciinema.org/a/626515
[2] openstack-k8s-operators/openstack-operator#587
[3] openstack-k8s-operators/install_yamls#654
[4] https://quay.io/repository/fpantano/glance-operator-bundle:3598c3d1fad3

[abays]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, fmount

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abays,fmount]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fmount]

Given that this change is ready and the openstack-operator patch is green [1], I'm removing the hold label and merge this change.

[1] https://review.rdoproject.org/zuul/buildset/f120eaadda9344ab8efe4c0db053e15d

[fmount]

/unhold