SSL termination not working end-to-end with Kong

[kubevious]

We are running Kong 2.1. Trying to terminate the SSL on the LB (certificate & domain managed by DO), and then forward to the Kong Proxy with HTTPS.

Here is the configuration on the LoadBalancer Service:

annotations:
    service.beta.kubernetes.io/do-loadbalancer-name: "my-best-lb-ever-if-we-make-it-work-eventually"
    service.beta.kubernetes.io/do-loadbalancer-protocol: "https"
    service.beta.kubernetes.io/do-loadbalancer-algorithm: "round_robin"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: "*****************"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "false"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"

This creates a forwarding rule like this.

But that port 31709 corresponds to a HTTPS Kong proxy endpoint. Obviously, Kong complains about and HTTP packet sent to HTTPS server.

Changing it manually to HTTPS does the trick, but this is definitely not an approach that can work in production. This would eventually get reverted back to HTTP and the entire service would be unaccessible.

[timoreimann]

👋

I'm getting 403s on the image links you embedded, but I think I get your request. What you're asking for is indeed not possible right now due to configuration limitations on CCM, but should be fairly easy to support through yet another configuration option. I'm happy to keep the ticket open to track the effort.

One note: the LB sends traffic into the droplets over the internal interface / VPC. I can see how HTTPS could still be desirable for extra security reasons or because applications might be easier to use on HTTPS. I wanted to mention it though to rule out potential misunderstandings on what goes over the public vs. the private network.

Thanks!

[timoreimann]

Another quick point / question: is TLS pass through an option for you? If not, could you share why TLS termination is needed in your case?

[rubenhak]

@timoreimann, updated the image urls for your reference. Please recheck the original post.

We can sacrifice HTTPS between the LB/Kong proxy, but we have to terminate the TLS on the LB in order to make use of cloud managed certificates and also to have x-forwarded-for header populated. Headers values cannot be modified unless the TLS is terminated at the LB.

So we are using Kong as api gateway. If you guys cannot allow HTTPS configuration after TLS termination as of right now, could you please help with Kong configuration to make the communication work?

Here are the overrides for Kong helm chart:

ingressController:
  enabled: true
  installCRDs: false

admin:
  enabled: true
  http:
    enabled: false

proxy:
  enabled: true
  http:
    enabled: false
  tls:
    enabled: true
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-name: "*******************"
    service.beta.kubernetes.io/do-loadbalancer-protocol: "https"
    service.beta.kubernetes.io/do-loadbalancer-algorithm: "round_robin"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: "*********************"
    service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "false"
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"

FYI, if proxy.http.enabled=true and proxy.tls.enabled=false is set, the LB gets configured without HTTPS.

[timoreimann]

@rubenhak the config looks good to me -- it should terminate TLS at the proxy and then forward the request to the Service port via HTTP. Can you check the namespace of the Service resource for any error events?

If that doesn't lead anywhere, then I'd need the rendered Service resource for full inspection. (If you're a DOKS customer, you can also submit a support request or ask on the Kubernetes Slack in the #digitalocean-k8s channel.)

[rubenhak]

@timoreimann, the result of this config is Kong Proxy service expecting a TLS connection (not a plain HTTP). That's because proxy.tls.enabled=true

Where exactly to check for errors? TBH I don't expect errors because LB gets consistently configured, but I can check if you could direct me.

Sure, will do that. Thanks.

[timoreimann]

Apologies, I understood it was possible (and acceptable) for you to disable HTTPS on the LB-to-cluster part of the route. The current config looks like it'd be a proper fit for that case.

You can see any errors by running kubectl get events -n <namespace where the Kong Service lives>.

[rubenhak]

It is acceptable to disable HTTPS between LB to cluster, but when we set proxy.tls.enabled=false the LB gets configured to accept HTTP connections only.

I dont see errors.

[timoreimann]

I'm not familiar with Kong in general and its Helm chart specifically, unfortunately. I briefly tried to understand what that proxy.tls.enabled Kong Helm template parameter does, but doesn't seem trivial to figure that out.
If you can share the rendered Service resource here, then I'm probably able to tell you why HTTP is configured only. However, I most likely can't help with why Kong renders a particular set of template parameters in a certain way (and how to tweak / change that).

Perhaps you can reach out to the Kong community for guidance in this regard. I'm happy to continue to help as far as the DO-specific parts are concerned.

[timoreimann]

Going to close this ticket as the Kong-specific part to it has not received an update in quite a while. We'll use #367 to track the HTTPS request as it seems more specific.