Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNS resolution fail on Kubernetes #119

Open
mDistefanoFacileIt opened this issue Sep 6, 2024 · 3 comments
Open

DNS resolution fail on Kubernetes #119

mDistefanoFacileIt opened this issue Sep 6, 2024 · 3 comments

Comments

@mDistefanoFacileIt
Copy link

mDistefanoFacileIt commented Sep 6, 2024

Hello folks, with my team we are trying to use AMPHP to execute a series of concurrency tasks.
Specifically, each task deals with:

  • reading values from a redis cache
  • if the values are not in the cache, executes one or more requests to APIs provided by our partners
  • records the values in the cache

To do this, we installed amphp/amp (^3.0), amphp/http-client (^5.1), amphp/redis (^2.0), league/uri (7.4.1) as dependency. Our project is based on Symfony and run PHP 8.1 We build our Amp\Futures and execute them with Amp\Future\awaitAll().

Locally we use docker-compose to execute our docker images and everything works as we expect.
In the test and production environment we use K8S with the same docker images used locally. Running the tests, however, we get an error when running requests, specifically when amp/dns tries to resolve the DNS, this is regardless of the configured uri. The error we get is as follows:

Connection to 'host.api.com:443' failed

Amp\Socket\ConnectException: Connection to tcp://host.api.com:443 failed after 3 attempts; previous attempts: DNS resolution for host.api.com failed: Name resolution failed for 'host.api.com.svc.cluster.local'; server returned error code: 3 (NXDomain), DNS resolution for host.api.com failed: Name resolution failed for 'host.api.com.svc.cluster.local'; server returned error code: 3 (NXDomain) in vendor/amphp/socket/src/RetrySocketConnector.php:50

These are systems outside of our network. The requests we want to execute are constructed quite simply:

        $request = new Request(
            'https://host.api.com/example',
            'POST',
            {'foo' : 'bar'},
        );

Reading the .svc.cluster.local it appears to be trying to resolve DNS internally within our K8S cluster. We also tried to configure the URI as https://host.api.com. (see the dot at the end) to avoid DNS resolution as reported in https://mrkaran.dev/posts/ndots-kubernetes/ . But then we encounter some certificate validation problems:

Amp\Socket\TlsException: TLS negotiation failed: stream_socket_enable_crypto(): Peer certificate CN=`host.api.com' did not match expected CN=`host.api.com.' 

We then performed a further test, creating a script that would simply run Amp\Dns\resolve() on the uri we were interested in. By simply running this function in K8S env, the DNS are resolved correctly. We think it's somenthing between \Amp\Http\Client\Connection\DefaultConnectionFactory and \Amp\Dns\Rfc1035StubDnsResolver::resolve to cause the error but cannot undestand what it is.
Do you have any suggestions as to whether we are doing it wrong or there is some problem with the resolution of the DNS under K8S?

We have already read #99 without success

@mDistefanoFacileIt
Copy link
Author

We figured out how to fix this.

Yuo can use ndots to avoid DNS resolution configuring the request as (see the dot at the end of https://host.api.com):

        $request = new Request(
            'https://host.api.com./example',
            'POST',
            {'foo' : 'bar'},
        );

Then you can configure the TLS handshake through HTTClientBuilder as:

        $tlsContext = (new ClientTlsContext())->withPeerName('host.api.com');

        $connectContext = (new ConnectContext())->withTlsContext($tlsContext);
        $factory = new DefaultConnectionFactory(null, $connectContext);
        $pool = ConnectionLimitingPool::byAuthority(3, $factory);

        return (new HttpClientBuilder())
            ->usingPool($pool)
            ->build();

@trowski
Copy link
Member

trowski commented Sep 9, 2024

Please have a look at your ndots configuration. From the looks of the error message you posted, it appears that the resolver is attempting to look up names prepended with other search names, and running out of attempts.

https://www.reddit.com/r/kubernetes/comments/duj86x/comment/f76k3hc/

Perhaps we're not handling ndots > 1 correctly and changes should be made here.

@kelunik kelunik reopened this Sep 9, 2024
@mDistefanoFacileIt
Copy link
Author

Hello @trowski in our resolve.conf we have:

options ndots:5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants