Connection to microk8s on the wrong ip

[shipperizer]

I'm having issues running a simple test while using the ops_test fixture

async def test_dummy(ops_test: OpsTest) -> None:
    assert True

it tries to connect to 127.0.0.1:16443 while myu local k8s config points to a completely different ip

shipperizer in ~/shipperizer/iam-bundle on main ● λ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: *********************************************************************************
    server: https://192.168.1.249:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: ******************************************************

below stack trace and execution

shipperizer in ~/shipperizer/iam-bundle on main ● λ TESTING_KUBECONFIG=/home/shipperizer/.kube/config tox -e integration -- --keep-models -x -k test_dummy
integration create: /home/shipperizer/shipperizer/iam-bundle/.tox/integration
integration installdeps: -r/home/shipperizer/shipperizer/iam-bundle/integration-requirements.txt
integration installed: anyio==3.7.1,asttokens==2.2.1,backcall==0.2.0,bcrypt==4.0.1,cachetools==5.3.1,certifi==2023.7.22,cffi==1.15.1,charset-normalizer==3.2.0,cryptography==41.0.3,decorator==5.1.1,executing==1.2.0,google-auth==2.22.0,greenlet==2.0.2,h11==0.14.0,httpcore==0.17.3,httpx==0.24.1,idna==3.4,iniconfig==2.0.0,ipdb==0.13.13,ipython==8.14.0,jedi==0.19.0,Jinja2==3.1.2,juju==3.2.0.1,jujubundlelib==0.5.7,kubernetes==27.2.0,lightkube==0.14.0,lightkube-models==1.27.1.4,macaroonbakery==1.3.1,MarkupSafe==2.1.3,matplotlib-inline==0.1.6,mypy-extensions==1.0.0,oauthlib==3.2.2,ops==2.5.1,packaging==23.1,paramiko==2.12.0,parso==0.8.3,pexpect==4.8.0,pickleshare==0.7.5,playwright==1.37.0,pluggy==1.2.0,prompt-toolkit==3.0.39,protobuf==3.20.3,ptyprocess==0.7.0,pure-eval==0.2.2,pyasn1==0.5.0,pyasn1-modules==0.3.0,pycparser==2.21,pyee==9.0.4,Pygments==2.16.1,pymacaroons==0.13.0,PyNaCl==1.5.0,pyRFC3339==1.1,pytest==7.4.0,pytest-asyncio==0.21.1,pytest-base-url==2.0.0,pytest-operator==0.28.0,pytest-playwright==0.4.2,python-dateutil==2.8.2,python-slugify==8.0.1,pytz==2023.3,PyYAML==6.0.1,requests==2.31.0,requests-oauthlib==1.3.1,rsa==4.9,six==1.16.0,sniffio==1.3.0,stack-data==0.6.2,text-unidecode==1.3,theblues==0.5.2,toposort==1.10,traitlets==5.9.0,typing-inspect==0.9.0,typing_extensions==4.7.1,urllib3==1.26.16,wcwidth==0.2.6,websocket-client==1.6.1,websockets==11.0.3
integration run-test-pre: PYTHONHASHSEED='3322065513'
integration run-test: commands[0] | playwright install
integration run-test: commands[1] | pytest -v --tb native /home/shipperizer/shipperizer/iam-bundle/tests/integration --log-cli-level=INFO -s --keep-models -x -k test_dummy
=========================================================== test session starts ============================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /home/shipperizer/shipperizer/iam-bundle/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/shipperizer/shipperizer/iam-bundle
configfile: pyproject.toml
plugins: playwright-0.4.2, anyio-3.7.1, operator-0.28.0, base-url-2.0.0, asyncio-0.21.1
asyncio: mode=Mode.AUTO
collected 11 items / 10 deselected / 1 selected                                                                                            

tests/integration/test_bundle.py::test_dummy 
-------------------------------------------------------------- live log setup --------------------------------------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:823 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))': /api/v1/namespaces/controller-microk8s/services/controller-service
WARNING  urllib3.connectionpool:connectionpool.py:823 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))': /api/v1/namespaces/controller-microk8s/services/controller-service
WARNING  urllib3.connectionpool:connectionpool.py:823 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))': /api/v1/namespaces/controller-microk8s/services/controller-service
ERROR

================================================================== ERRORS ==================================================================
_______________________________________________________ ERROR at setup of test_dummy _______________________________________________________
Traceback (most recent call last):
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 567, in read
    data = self._fp_read(amt) if not fp_closed else b""
           ^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 533, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
                                                      ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 479, in read
    s = self.fp.read()
        ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/connectionpool.py", line 734, in urlopen
    response = self.ResponseCls.from_httplib(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 653, in from_httplib
    resp = ResponseCls(
           ^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 266, in __init__
    self._body = self.read(decode_content=decode_content)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 566, in read
    with self._error_catcher():
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
                                ^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_callers.py", line 155, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_result.py", line 108, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/runner.py", line 157, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/runner.py", line 497, in setup
    raise exc
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/runner.py", line 494, in setup
    col.setup()
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/python.py", line 1791, in setup
    self._request._fillfixtures()
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 566, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 585, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 607, in _get_active_fixturedef
    self._compute_fixture_value(fixturedef)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 693, in _compute_fixture_value
    fixturedef.execute(request=subrequest)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 1069, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_callers.py", line 155, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_result.py", line 108, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 1123, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/_pytest/fixtures.py", line 902, in call_fixture_func
    fixture_result = fixturefunc(**kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 304, in _asyncgen_fixture_wrapper
    result = event_loop.run_until_complete(setup())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 286, in setup
    res = await gen_obj.__anext__()
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pytest_operator/plugin.py", line 231, in ops_test
    await ops_test._setup_model()
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/pytest_operator/plugin.py", line 708, in _setup_model
    await self._controller.connect(self.controller_name)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/juju/controller.py", line 104, in connect
    await self._connector.connect_controller(controller_name, **kwargs)
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/juju/client/connector.py", line 111, in connect_controller
    await self.connect(
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/juju/client/connector.py", line 75, in connect
    self._connection = await Connection.connect(**kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/juju/client/connection.py", line 338, in connect
    self.proxy.connect()
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/juju/client/proxy/kubernetes/proxy.py", line 42, in connect
    service = corev1.read_namespaced_service(self.service, self.namespace)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 25141, in read_namespaced_service
    return self.read_namespaced_service_with_http_info(name, namespace, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 25228, in read_namespaced_service_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/shipperizer/shipperizer/iam-bundle/.tox/integration/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=16443): Max retries exceeded with url: /api/v1/namespaces/controller-microk8s/services/controller-service (Caused by ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')))
------------------------------------------------------------ Captured log setup ------------------------------------------------------------
WARNING  urllib3.connectionpool:connectionpool.py:823 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))': /api/v1/namespaces/controller-microk8s/services/controller-service
WARNING  urllib3.connectionpool:connectionpool.py:823 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))': /api/v1/namespaces/controller-microk8s/services/controller-service
WARNING  urllib3.connectionpool:connectionpool.py:823 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))': /api/v1/namespaces/controller-microk8s/services/controller-service
========================================================= short test summary info ==========================================================
ERROR tests/integration/test_bundle.py::test_dummy - urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=16443): Max retries exceeded with url: /api/v1/namespaces/c...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================== 10 deselected, 1 error in 0.02s ======================================================
ERROR: InvocationError for command /home/shipperizer/shipperizer/iam-bundle/.tox/integration/bin/pytest -v --tb native tests/integration --log-cli-level=INFO -s --keep-models -x -k test_dummy (exited with code 1)
_________________________________________________________________ summary __________________________________________________________________
ERROR:   integration: commands failed

my juju environment seems to be working just fine

shipperizer in ~/shipperizer/iam-bundle on main ● λ juju controllers
Use --refresh option with this command to see the latest information.

Controller  Model   User   Access     Cloud/Region        Models  Nodes  HA  Version
microk8s*   bundle  admin  superuser  microk8s/localhost       3      1   -  3.1.5  
shipperizer in ~/shipperizer/iam-bundle on main ● λ juju clouds     
Only clouds with registered credentials are shown.
There are more clouds, use --all to see them.

Clouds available on the controller:
Cloud     Regions  Default    Type
microk8s  1        localhost  k8s  

Clouds available on the client:
Cloud      Regions  Default    Type  Credentials  Source    Description
localhost  1        localhost  lxd   1            built-in  LXD Container Hypervisor
microk8s   1        localhost  k8s   1            built-in  A Kubernetes Cluster

[shipperizer]

issue seems to be related to the proxy-config property in the juju controller configuration, see thread https://chat.charmhub.io/charmhub/pl/w3y9mwzyxirpirha6cb6rp4jcr for more info

[PietroPasotti]

same issue here. hard to understand when/why it started happening

[cderici]

How does your controllers.yaml look like? pylibjuju should pick it up if it's in there, otherwise we'll need to improve the connect_controller to accept some kwargs for the proxy, currently it looks like the controllers.yaml is the only way to get that https://github.com/juju/python-libjuju/blob/b56963442a6deffe46ce1fb5cb41970344263649/juju/client/connector.py#L143.

[PietroPasotti]

the definition for my k8s cloud looks like:

 microk8s-localhost:
        uuid: 351b04f0-19b5-4ddf-86d5-5a2497dda9ce
        api-endpoints: ['10.152.183.40:17070']
        dns-cache: {localhost: [127.0.0.1]}
        ca-cert: SNIP
        cloud: microk8s
        region: localhost
        type: kubernetes
        agent-version: 3.5.0
        controller-machine-count: 1
        active-controller-machine-count: 0
        machine-count: 1
        proxy-config:
            type: kubernetes-port-forward
            config:
                api-host: 127.0.0.1:16443
                ca-cert: SNIP
                namespace: controller-microk8s-localhost
                remote-port: "17070"
                service: controller-service
                service-account-token: SNIP

I tried manually editing and replace 127.0.0.1 with the IP that I get from microk8s config:

but that made no difference whatsoever.