Proxy Service: Keep throwing the same error again and again on Proxy Pods.

[waleed-cariad]

We are running teleport cluster (proxy and auth service) on a Kubernetes Cluster using the helm charts provided by teleport.
Teleport Version:15.3.4

Our agents are running on VMs (we have a setup where agents are deployed in the same VNET where we want this agent to watch and discover kubernetes clusters deployed in that VM) and these agents open reverse tunnel to our teleport cluster via proxy's public address.

Configuration on teleport proxy service
proxy_protocol is set to "off" (because it's running behind Azure L4 Load Balancer which doesn't support proxy_protocol)
public_addr: <our_public_domain>

Configuration on teleport auth service
proxy_protocol is set to "off"
proxy_listener_mode is set to multiplex

Although, everything apparently working fine and we are able to connect to our kubernetes clusters using teleport but the proxy pods are keep throwing this error and our log stack is messed up with this. We have currently no idea why these proxy pods are complaining. Can you let us know what could be the possible reason for this error?

2024-07-03T06:06:16Z WARN [ALPN:PROX] Failed to handle client connection. error:[
ERROR REPORT:
Original Error: *errors.errorString tls: client offered only unsupported versions: []
Stack Trace:
        github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:395 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).handleConn
        github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:329 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).Serve.func1
        runtime/asm_amd64.s:1650 runtime.goexit
User Message: tls: client offered only unsupported versions: []] alpnproxy/proxy.go:340

also this

2024-07-03T06:07:12Z WARN [ALPN:PROX] Failed to handle client connection. error:[
ERROR REPORT:
Original Error: *errors.errorString tls: no cipher suite supported by both client and server
Stack Trace:
        github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:395 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).handleConn
        github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:329 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).Serve.func1
        runtime/asm_amd64.s:1650 runtime.goexit
User Message: tls: no cipher suite supported by both client and server] alpnproxy/proxy.go:340

This can be seen in our proxy pods logs appearing all the time where as, we can still connect to teleport using tsh and our agents are also connected and have establised reverse proxies via proxy service to teleport cluster.

[webvictim]

This message is usually caused by HTTPS connections which are not compatible with TLS 1.2/1.3 or which are not including correct Host headers/SNI and just connecting to the IP address of the LB. Sometimes this is just what happens when you put services on the public internet and random traffic hits them.

One thing to check - do you have healthchecks running from your load balancer to the workloads behind it? Are they using HTTPS? Can you control what hostname they connect to? If this message appears on a regular cadence (say every 5-30 seconds) then you may be able to tweak the settings to reduce the log spam - tell the proxy healthchecks to call https://teleport.example.com/web/login. If they're TCP healthchecks then you can try changing them to HTTPS instead.

Failing that, it's safe to ignore them assuming everything else works fine.

[waleed-cariad]

We don't have healthchecks running from LB to our workloads (proxy pods)
We also see these kind of logs recently where a public ip of a random client is also logged and we are sure it's not the ip from any of our clients/users (tsh, teleport agents) trying to connect to our proxy service.

2024-07-08T02:55:46Z WARN [MX:PROXY:] "\nERROR REPORT:\nOriginal Error: *trace.BadParameterError failed to detect connection protocol, first few bytes were: []byte{0x2a, 0x31, 0xd, 0xa, 0x24, 0x34, 0xd, 0xa}\nStack Trace:\n\tgithub.com/gravitational/teleport/lib/multiplexer/multiplexer.go:828
github.com/gravitational/teleport/lib/multiplexer.detectProto\n\tgithub.com/gravitational/teleport/lib/multiplexer/multiplexer.go:509
github.com/gravitational/teleport/lib/multiplexer.(*Mux).detect\n\tgithub.com/gravitational/teleport/lib/multiplexer/multiplexer.go:339
github.com/gravitational/teleport/lib/multiplexer.(*Mux).detectAndForward\n\truntime/asm_amd64.s:1650 runtime.goexit\
User Message: failed to detect connection protocol, first few bytes were: []byte{0x2a, 0x31, 0xd, 0xa, 0x24, 0x34, 0xd, 0xa}
(logs containing \"failed to detect connection protocol\" were seen 2 times in the past minute)" **dst_addr:10.64.8.10:3080 src_addr:87.236.176.65:40941** loglimit/loglimit.go:184

Is there a way in teleport config to block some specific ips only as I don't see anything related in teleport docs or is there any other way to handle it nicely?

[webvictim]

Teleport doesn't block IPs natively. You'd want to use a WAF or other firewall deployed in front of Teleport to handle that.