Make it possible to use one Bitbucket svc for http and ssh without an…

… ingress (#675)

* Make it possible to use one Bitbucket svc for http and ssh

* Fix unit test

* Update docs/docs/examples/bitbucket/BITBUCKET_SSH.md

Co-authored-by: D∇L∆N <[email protected]>

---------

Co-authored-by: Yevhen Ivantsov <[email protected]>
Co-authored-by: D∇L∆N <[email protected]>

docs/docs/examples/bitbucket/BITBUCKET_SSH.md

@@ -2,10 +2,11 @@
 
 In addition to providing a service on HTTP(S), Bitbucket also allows remote Git operations over SSH connections. By default, Kubernetes Ingress controllers only work for HTTP connections, but some ingress controllers also support TCP connections.
 
-Depending on the need of your deployment, SSH access can be provided through two mechanisms:
+Depending on the need of your deployment, SSH access can be provided through three mechanisms:
 
 1. Opening the TCP port through the ingress controller - This option should be used if the SSH service is required to be available on the same DNS name as the HTTP service.
 2. Creating a separate Kubernetes `LoadBalancer` service - This option is available if the ingress controller does not support TCP connections, or if you don’t need your deployment to have the SSH service available on the same DNS name as the HTTP service.
+3. Exposing Bitbucket service as `LoadBalancer` and setting the desired ssh port (defaults to `7999`)
 
 ## NGINX Ingress controller config for SSH connections
 We can follow the official documentation for the NGINX Ingress controller for this: [Exposing TCP and UDP services - NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/){.external}.
@@ -107,3 +108,31 @@ bitbucket:
     - name: PLUGIN_SSH_BASEURL
       value: ssh://bitbucket-ssh.example.com/
 ```
+## Expose Bitbucket Service as a LoadBalancer
+
+This method implies creation of one K8S service of a `LoadBalancer` type. A cloud provider (e.g. AWS) will create listeners for each port declared in the service. Here's an example of exposing Bitbucket service in EKS, using a Classic LoadBalancer and dynamically provisioning DNS entries with [external-dns](https://github.com/kubernetes-sigs/external-dns){.external}:
+
+```yaml
+bitbucket:
+  service:
+    port: 443
+    sshPort: 22
+    type: LoadBalancer
+    annotations:
+      external-dns.alpha.kubernetes.io/hostname: bitbucket.example.com
+      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
+      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:111111111111:certificate/8xy4ny81-0a4w-8caq-a524-1101cv3v4vwb
+  additionalEnvironmentVariables:
+    - name: PLUGIN_SSH_BASEURL
+      value: ssh://bitbucket.example.com
+ingress:
+  host: bitbucket.example.com
+```
+
+The above service annotations are specific to the Classic LoadBalancer, however, you can provide [NLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html){.external} specific [annotations](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/){.external} as well.
+
+The default `bitbucket.service.sshPort` is set to `22` so that AWS can create a listener for this port, and as a result your ssh git URL will look like `ssh://bitbucket.example.com/project/repo`. 
+
+!!!info "Ingress host"
+
+    Even though `ingress` is disabled, `ingress.host` needs to be set because it is used in a few conditions in the StatefulSet template.

src/main/charts/bitbucket/README.md

@@ -91,7 +91,7 @@ Kubernetes: `>=1.21.x-0`
 | bitbucket.podManagementStrategy | string | `"OrderedReady"` | Pod management strategy. Bitbucket Data Center requires the "OrderedReady" value but for Bitbucket Mirrors you can use the "Parallel" option. To learn more, visit https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies  |
 | bitbucket.ports.hazelcast | int | `5701` | The port on which the Hazelcast listens for client traffic  |
 | bitbucket.ports.http | int | `7990` | The port on which the Bitbucket container listens for HTTP traffic  |
-| bitbucket.ports.ssh | int | `7999` | The port on which the Bitbucket container listens for SSH traffic  |
+| bitbucket.ports.ssh | int | `7999` | The port on which the Bitbucket SSH service will listen on. Must be within 1024-65535 range  |
 | bitbucket.readinessProbe.customProbe | object | `{}` | Custom readinessProbe to override the default /status httpGet  |
 | bitbucket.readinessProbe.enabled | bool | `true` | Whether to apply the readinessProbe check to pod.  |
 | bitbucket.readinessProbe.failureThreshold | int | `60` | The number of consecutive failures of the Bitbucket container readiness probe before the pod fails readiness checks.  |
@@ -107,10 +107,11 @@ Kubernetes: `>=1.21.x-0`
 | bitbucket.service.annotations | object | `{}` | Additional annotations to apply to the Service  |
 | bitbucket.service.contextPath | string | `nil` | The context path that Bitbucket will use.  |
 | bitbucket.service.loadBalancerIP | string | `nil` | Use specific loadBalancerIP. Only applies to service type LoadBalancer.  |
-| bitbucket.service.port | int | `80` | The port on which the Bitbucket K8s Service will listen  |
+| bitbucket.service.port | int | `80` | The port on which the Bitbucket K8s HTTP Service will listen  |
 | bitbucket.service.sessionAffinity | string | `"None"` | Session affinity type. If you want to make sure that connections from a particular client are passed to the same pod each time, set sessionAffinity to ClientIP. See: https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity  |
 | bitbucket.service.sessionAffinityConfig | object | `{"clientIP":{"timeoutSeconds":null}}` | Session affinity configuration  |
 | bitbucket.service.sessionAffinityConfig.clientIP.timeoutSeconds | string | `nil` | Specifies the seconds of ClientIP type session sticky time. The value must be > 0 && <= 86400 (for 1 day) if ServiceAffinity == "ClientIP". Default value is 10800 (for 3 hours).  |
+| bitbucket.service.sshPort | int | `7999` | The port on which the Bitbucket K8s SSH Service will listen  |
 | bitbucket.service.type | string | `"ClusterIP"` | The type of K8s service to use for Bitbucket  |
 | bitbucket.setPermissions | bool | `true` | Boolean to define whether to set local home directory permissions on startup of Bitbucket container. Set to 'false' to disable this behaviour.  |
 | bitbucket.shutdown.command | string | `"/shutdown-wait.sh"` | By default pods will be stopped via a [preStop hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/), using a script supplied by the Docker image. If any other shutdown behaviour is needed it can be achieved by overriding this value. Note that the shutdown command needs to wait for the application shutdown completely before exiting; see [the default command](https://bitbucket.org/atlassian-docker/docker-atlassian-bitbucket-server/src/master/shutdown-wait.sh) for details.  |

src/main/charts/bitbucket/templates/service.yaml

@@ -25,7 +25,7 @@ spec:
       targetPort: http
       protocol: TCP
       name: http
-    - port: {{ .Values.bitbucket.ports.ssh }}
+    - port: {{ .Values.bitbucket.service.sshPort }}
       targetPort: ssh
       protocol: TCP
       name: ssh

src/main/charts/bitbucket/templates/statefulset.yaml

@@ -178,6 +178,8 @@ spec:
             {{- include "bitbucket.databaseEnvVars" . | nindent 12 }}
             {{- include "bitbucket.sysadminEnvVars" . | nindent 12 }}
             {{- include "bitbucket.elasticSearchEnvVars" . | nindent 12 }}
+            - name: PLUGIN_SSH_PORT
+              value: {{ .Values.bitbucket.ports.ssh | quote }}
             {{ if .Values.ingress.host }}
             - name: SERVER_PROXY_NAME
               value: {{ .Values.ingress.host | quote }}

src/main/charts/bitbucket/values.yaml

@@ -471,10 +471,14 @@ bitbucket:
   #
   service:
 
-    # -- The port on which the Bitbucket K8s Service will listen
+    # -- The port on which the Bitbucket K8s HTTP Service will listen
     #
     port: 80
 
+    # -- The port on which the Bitbucket K8s SSH Service will listen
+    #
+    sshPort: 7999
+
     # -- The type of K8s service to use for Bitbucket
     #
     type: ClusterIP
@@ -575,7 +579,7 @@ bitbucket:
     #
     http: 7990
 
-    # -- The port on which the Bitbucket container listens for SSH traffic
+    # -- The port on which the Bitbucket SSH service will listen on. Must be within 1024-65535 range
     #
     ssh: 7999
 

src/test/java/test/BitbucketSshServiceTest.java

@@ -8,6 +8,7 @@
 import test.model.Kind;
 import test.model.Product;
 import test.model.Service;
+import test.model.StatefulSet;
 
 import java.util.Map;
 
@@ -63,4 +64,41 @@ void values_set(Product product) throws Exception {
         resources.getStatefulSet(product.getHelmReleaseName()).getContainer().getEnv().
                 assertHasValue("PLUGIN_SSH_BASEURL", "ssh://hostname:7999/");
     }
+
+    @ParameterizedTest
+    @EnumSource(value = Product.class, mode = INCLUDE, names = "bitbucket")
+    void bitbucket_default_ssh_svc_port(Product product) throws Exception {
+        final var resources = helm.captureKubeResourcesFromHelmChart(product, Map.of());
+        var service = resources.get(Kind.Service, Service.class, product.getHelmReleaseName());
+        assertThat(service.getPort("ssh")).hasValueSatisfying(node -> assertThat(node.path("port")).hasValueEqualTo(7999));
+    }
+
+    @ParameterizedTest
+    @EnumSource(value = Product.class, mode = INCLUDE, names = "bitbucket")
+    void bitbucket_override_ssh_svc_port(Product product) throws Exception {
+        final var resources = helm.captureKubeResourcesFromHelmChart(product, Map.of(
+                product.name() + ".service.sshPort", "23"));
+        var service = resources.get(Kind.Service, Service.class, product.getHelmReleaseName());
+        assertThat(service.getPort("ssh")).hasValueSatisfying(node -> assertThat(node.path("port")).hasValueEqualTo(23));
+    }
+
+    @ParameterizedTest
+    @EnumSource(value = Product.class, mode = INCLUDE, names = "bitbucket")
+    void bitbucket_default_ssh_listen_port(Product product) throws Exception {
+        final var resources = helm.captureKubeResourcesFromHelmChart(product, Map.of());
+        StatefulSet statefulSet = resources.getStatefulSet(product.getHelmReleaseName());
+        statefulSet.getContainer("bitbucket").getEnv().assertHasValue("PLUGIN_SSH_PORT", "7999");
+        assertThat(statefulSet.getContainer("bitbucket").getPort("ssh").path("containerPort")).hasValueEqualTo(7999);
+    }
+
+    @ParameterizedTest
+    @EnumSource(value = Product.class, mode = INCLUDE, names = "bitbucket")
+    void bitbucket_override_ssh_listen_port(Product product) throws Exception {
+        final var resources = helm.captureKubeResourcesFromHelmChart(product, Map.of(
+                product.name() + ".ports.ssh", "7000"
+        ));
+        StatefulSet statefulSet = resources.getStatefulSet(product.getHelmReleaseName());
+        statefulSet.getContainer("bitbucket").getEnv().assertHasValue("PLUGIN_SSH_PORT", "7000");
+        assertThat(statefulSet.getContainer("bitbucket").getPort("ssh").path("containerPort")).hasValueEqualTo(7000);
+    }
 }

src/test/resources/expected_helm_output/bitbucket/output.yaml

@@ -228,7 +228,6 @@ spec:
   template:
     metadata:
       annotations:
-        checksum/config-jvm: de074cff37426addb13b574fb196092b2726782ce56718e3a20c4c161a16a4d1
       labels:
         app.kubernetes.io/name: bitbucket-mesh
         app.kubernetes.io/instance: unittest-bitbucket
@@ -351,7 +350,6 @@ spec:
   template:
     metadata:
       annotations:
-        checksum/config-jvm: 0f6bb14aa1b5493eb7e23b43f5ebd0a735ef0590aa8696f9ad226a9bd5d4b14a
       labels:
         app.kubernetes.io/name: bitbucket
         app.kubernetes.io/instance: unittest-bitbucket
@@ -416,6 +414,8 @@ spec:
                   fieldPath: metadata.name
             - name: JAVA_OPTS
               value: "-Dcluster.node.name=$(KUBE_POD_NAME)"
+            - name: PLUGIN_SSH_PORT
+              value: "7999"
             - name: SERVER_CONTEXT_PATH
               value: "/"
             - name: SERVER_PORT