Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

certificate verify failed issue when using Get Namespaced Pod Exec #123

Open
yannpichon-su opened this issue Dec 12, 2022 · 6 comments · Fixed by #124
Open

certificate verify failed issue when using Get Namespaced Pod Exec #123

yannpichon-su opened this issue Dec 12, 2022 · 6 comments · Fixed by #124

Comments

@yannpichon-su
Copy link

When using the Get Namespaced Pod Exec keywork on a k8s cluster using a custom CA, the following error occurs :

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)

Other keywords (Read Namespaced Pod Status, List Namespaced Pod By Pattern ...) are working as expected.

As a quick fix, I'm adding the following line in the _add_api method of the library :

def _add_api(self, reference, class_name):
    self.__dict__[reference] = class_name(self.api_client)
    if not self.cert_validation:
        self.__dict__[reference].api_client.rest_client.pool_manager.connection_pool_kw['cert_reqs'] = ssl.CERT_NONE
        self.__dict__[reference].api_client.configuration.verify_ssl = False

Am I missing something regarding the library configuration ?

Versions :

KubeLibrary: 0.8.0
Python: 3.9.13
Kubernetes: 1.24

KubeLibrary :

Library    KubeLibrary    kube_config=${KUBECONFIG_FILE}    cert_validation=False

KubeLibrary.Get Namespaced Pod Exec    
...    name=my-pod
...    namespace=${namespace}
...    argv_cmd=${command}
@m-wcislo m-wcislo mentioned this issue Dec 16, 2022
Merged
@m-wcislo
Copy link
Collaborator

Yeah your workaround is way to go, it does the current setting underneath so it is preferred, Please take a look at v0.8.1 at let me know if it works for you.

@fdaguin
Copy link

fdaguin commented Dec 19, 2022

Thanks for the release @m-wcislo! However, removing the line https://github.com/devopsspiral/KubeLibrary/pull/124/files#diff-65edc7c7cbe7902bb5d18f0b4555bbf308e8cac89745ce248bec177fd9a9b334L279 introduced a regression.

For example, using the keyword List Namespaced Daemon Set leads to the following error:

WARN	Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.767
WARN	Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.850
WARN	Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.929
FAIL	MaxRetryError: HTTPSConnectionPool(host='***redacted***', port=443): Max retries exceeded with url: /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))
09:30:49.930
DEBUG	Traceback (most recent call last):
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***redacted***/.venv/lib/python3.9/site-packages/KubeLibrary/KubeLibrary.py", line 1378, in list_namespaced_daemon_set
    ret = self.appsv1.list_namespaced_daemon_set(namespace, watch=False, label_selector=label_selector)
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api/apps_v1_api.py", line 3181, in list_namespaced_daemon_set
    return self.list_namespaced_daemon_set_with_http_info(namespace, **kwargs)  # noqa: E501
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api/apps_v1_api.py", line 3296, in list_namespaced_daemon_set_with_http_info
    return self.api_client.call_api(
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='***redacted***', port=443): Max retries exceeded with url: /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))

Could it be reverted please?

@m-wcislo m-wcislo reopened this Dec 19, 2022
@m-wcislo
Copy link
Collaborator

ok, reverted in 0.8.2, need to take closer look.

@fdaguin
Copy link

fdaguin commented Dec 19, 2022

Thank you! From our tests, keeping both

    if not self.cert_validation:
        self.__dict__[reference].api_client.rest_client.pool_manager.connection_pool_kw['cert_reqs'] = ssl.CERT_NONE
        self.__dict__[reference].api_client.configuration.verify_ssl = False

was working as expected.

@m-wcislo
Copy link
Collaborator

Actually it is reverted in 0.8.3. yeah need to track down how the setting is propagated, latest setting should set both

@m-wcislo
Copy link
Collaborator

Ok, did some testing and cannot really reproduce it on my end, so have couple questions:

  1. Did it worked in the past for you and is just broken recently? If yes what changes were made?
  2. Your kubeconfig doesn't have :
clusters:
- cluster:
    certificate-authority-data: ?

This is the easiest way to make your connection trust the API server cert, without need of use cert_validation=False
3. Are the failing (Get Namespaced Pod Exec) tests are executed somehow differently (different kubeconfig, settings etc.)than the ones that are passing (List Namespaced Daemon Set)?
4. Would it be possible to list packages versions you are using (pip freeze) at least for things in reuirements.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix disabling cert validation devopsspiral/KubeLibrary
3 participants
@m-wcislo @fdaguin @yannpichon-su