Skip to content

Latest commit

 

History

History
700 lines (624 loc) · 25.2 KB

docker-kubernetes.adoc

File metadata and controls

700 lines (624 loc) · 25.2 KB
Raw

Java EE Application on Kubernetes Cluster

Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
— github.com/GoogleCloudPlatform/kubernetes/

Kubernetes, or “k8s” in short, allows the user to provide declarative primitives for the desired state, for example “need 5 WildFly servers and 1 MySQL server running”. Kubernetes self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers then ensure this state is met. The user just define the state and Kubernetes ensures that the state is met at all times on the cluster.

How is it related to Docker?

Docker provides the lifecycle management of containers. A Docker image defines a build time representation of the runtime containers. There are commands to start, stop, restart, link, and perform other lifecycle methods on these containers. Kubernetes uses Docker to package, instantiate, and run containerized applications.

How does Kubernetes simplify containerized application deployment?

A typical application would have a cluster of containers across multiple hosts. For example, your web tier (for example Undertow) might run as a few instances, and likely on a set of containers. Similarly, your application tier (for example, WildFly) would run on a different set of containers. The web tier would need to delegate the request to application tier. The web, application, and database tier would generally run on a separate set of containers. These containers would need to talk to each other. Using any of the solutions mentioned above would require scripting to start the containers, and monitoring/bouncing if something goes down. Kubernetes does all of that for the user after the application state has been defined.

Key Components

At a very high level, there are three key components:

  1. Pods are the smallest deployable units that can be created, scheduled, and managed. Its a logical collection of containers that belong to an application.

  2. Master is the central control point that provides a unified view of the cluster. There is a single master node that control multiple worker nodes.

  3. Node (née minion) is a worker node that run tasks as delegated by the master. Nodes can run one or more pods. It provides an application-specific “virtual host” in a containerized environment.

A picture is always worth a thousand words and so this is a high-level logical block diagram for Kubernetes:

kubernetes key components
Figure 1. Kubernetes Key Components

After the 50,000 feet view, lets fly a little lower at 30,000 feet and take a look at how Kubernetes make all of this happen. There are a few key components at Master and Node that make this happen.

  1. Replication Controller is a resource at Master that ensures that requested number of pods are running on nodes at all times.

  2. Service is an object on master that provides load balancing across a replicated group of pods. Label is an arbitrary key/value pair in a distributed watchable storage that the Replication Controller uses for service discovery.

  3. Kubelet Each node runs services to run containers and be managed from the master. In addition to Docker, Kubelet is another key service installed there. It reads container manifests as YAML files that describes a pod. Kubelet ensures that the containers defined in the pods are started and continue running.

  4. Master serves RESTful Kubernetes API that validate and configure Pod, Service, and Replication Controller.

Start Kubernetes Cluster

Kubernetes cluster can be easily started using Vagrant. There are two options to start the cluster - first using a downloaded Kubernetes distribution bundle and second by downloading the latest bundle as part of the install.

Using Previously Downloaded Kubernetes Distribution

  1. Setup a Kubernetes cluster using a previously configured Kubernetes distribution as explained in [Kubernetes_Setup]:

    cd kubernetes

export KUBERNETES_PROVIDER=vagrant
./cluster/kube-up.sh

    The KUBERNETES_PROVIDER environment variable tells all of the various cluster management scripts which variant to use.

    Note
    		 This will take a few minutes, so be patience! Vagrant will provision each machine in the cluster with all the necessary components to run Kubernetes.

    It shows the output as:

    Starting cluster using provider: vagrant
... calling verify-prereqs
... calling kube-up
Using credentials: vagrant:vagrant

. . .

Cluster validation succeeded
Done, listing cluster services:

Kubernetes master is running at https://10.245.1.2
KubeDNS is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-ui

    Note down the address for Kubernetes master, https://10.245.1.2 in this case.

Download and Start the Cluster Together

  1. Alternatively, the cluster can also be started as:

    > curl -sS https://get.k8s.io | bash
Downloading kubernetes release v0.21.1 to /Users/arungupta/tools/kubernetes.tar.gz
--2015-07-13 15:56:54--  https://storage.googleapis.com/kubernetes-release/release/v0.21.1/kubernetes.tar.gz
Resolving storage.googleapis.com... 74.125.28.128, 2607:f8b0:400e:c02::80
Connecting to storage.googleapis.com|74.125.28.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 117901998 (112M) [application/x-tar]
Saving to: 'kubernetes.tar.gz'

kubernetes.tar.gz               100%[=========================================================>] 112.44M  6.21MB/s   in 18s

2015-07-13 15:57:13 (6.13 MB/s) - 'kubernetes.tar.gz' saved [117901998/117901998]

. . .

NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   nil
scheduler            Healthy   ok                   nil
etcd-0               Healthy   {"health": "true"}   nil
Cluster validation succeeded
Done, listing cluster services:

Kubernetes master is running at https://10.245.1.2
KubeDNS is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-ui

Kubernetes binaries at /Users/arungupta/tools/kubernetes/kubernetes/cluster/
You may want to add this directory to your PATH in $HOME/.profile
Installation successful!

Verify the Cluster

  1. Verify the Kubernetes cluster as:

    kubernetes> vagrant status
Current machine states:

master                    running (virtualbox)
minion-1                  running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

    By default, the Vagrant setup will create a single Master and one node. Each VM will take 1 GB, so make sure you have at least 2GB to 4GB of free memory (plus appropriate free disk space).

    Note
    		 By default, only one node is created. This can be manipulated by setting an environment variable NUM_MINIONS variable to an integer before invoking kube-up.sh script.
    kubernetes cluster vagrant
    Figure 2. Kubernetes Cluster using Vagrant

    By default, each VM in the cluster is running Fedora, Kubelet is installed into ``systemd'', and all other Kubernetes services are running as containers on Master.

  2. Access https://10.245.1.2 (or whatever IP address is assigned to your kubernetes cluster start up log). This may present the warning as shown below:

    kubernetes master default output certificate

    Click on ‘Advanced’, on ‘Proceed to 10.245.1.2’, enter the username as ‘vagrant’ and password as ‘vagrant’ to see the output as:

    kubernetes master default output
    Figure 3. Kubernetes Output from Master

    Check the list of nodes as:

    > ./cluster/kubectl.sh get nodes
NAME         LABELS                              STATUS
10.245.1.3   kubernetes.io/hostname=10.245.1.3   Ready

  3. Check the list of pods:

    kubernetes> ./cluster/kubectl.sh get po
NAME      READY     STATUS    RESTARTS   AGE

  4. Check the list of services running:

    kubernetes> ./cluster/kubectl.sh get se
NAME         LABELS                                    SELECTOR   IP(S)        PORT(S)
kubernetes   component=apiserver,provider=kubernetes   <none>     10.247.0.1   443/TCP

  5. Check the list of replication controllers:

    kubernetes> ./cluster/kubectl.sh get rc
CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR   REPLICAS

Deploy Java EE Application (multiple configuration files)

Pods, and the IP addresses assigned to them, are ephemeral. If a pod dies then Kubernetes will recreate that pod because of its self-healing features, but it might recreate it on a different host. Even if it is on the same host, a different IP address could be assigned to it. And so any application cannot rely upon the IP address of the pod.

Kubernetes services is an abstraction which defines a logical set of pods. A service is typically back-ended by one or more physical pods (associated using labels), and it has a permanent IP address that can be used by other pods/applications. For example, WildFly pod can not directly connect to a MySQL pod but can connect to MySQL service. In essence, Kubernetes service offers clients an IP and port pair which, when accessed, redirects to the appropriate backends.

kubernetes service
Figure 4. Kubernetes Service
Note
 In this case, all the pods are running on a single node. This is because, that is the default number for a Kubernetes cluster. The pod can very be on another node if more number of nodes are configured to start in the cluster.

Any Service that a Pod wants to access must be created before the Pod itself, or else the environment variables will not be populated.

The order of Service and the targeted Pods does not matter. However Service needs to be started before any other Pods consuming the Service are started.

Start MySQL Pod

  1. Start MySQL Pod:

    ./cluster/kubectl.sh create -f ../../attendees/kubernetes/app-mysql-pod.yaml
pods/mysql-pod

    It uses the following configuration file:

    apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
  labels:
    name: mysql-pod
    context: docker-k8s-lab
spec:
  containers:
    -
      name: mysql
      image: mysql:latest
      env:
        -
          name: "MYSQL_USER"
          value: "mysql"
        -
          name: "MYSQL_PASSWORD"
          value: "mysql"
        -
          name: "MYSQL_DATABASE"
          value: "sample"
        -
          name: "MYSQL_ROOT_PASSWORD"
          value: "supersecret"
      ports:
        -
          containerPort: 3306

  2. Get status of the Pod:

    kubernetes> ./cluster/kubectl.sh get -w po
NAME        READY     STATUS    RESTARTS   AGE
mysql-pod   0/1       Pending   0          4s
NAME        READY     STATUS    RESTARTS   AGE
mysql-pod   0/1       Running   0          44s
mysql-pod   1/1       Running   0         44s

    -w watches for changes to the requested object. Wait for the MySQL pod to be in Running status.

Start MySQL service

  1. Start MySQL Service:

    ./cluster/kubectl.sh create -f ../../attendees/kubernetes/app-mysql-service.yaml
services/mysql-service

    It uses the following configuration file:

    apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  labels:
    name: mysql-pod
    context: docker-k8s-lab
spec:
  ports:
    # the port that this service should serve on
    - port: 3306
  # label keys and values that must match in order to receive traffic for this service
  selector:
    name: mysql-pod
    context: docker-k8s-lab

    Once again, the label “context: docker-k8s-lab” is used. This simplifies querying the created pods later on.

  2. Get status of the Service:

    ./cluster/kubectl.sh get -w se
NAME            LABELS                                    SELECTOR                                IP(S)          PORT(S)
kubernetes      component=apiserver,provider=kubernetes   <none>                                  10.247.0.1     443/TCP
mysql-service   context=docker-k8s-lab,name=mysql-pod     context=docker-k8s-lab,name=mysql-pod   10.247.63.43   3306/TCP

    If multiple services are running, then it can be narrowed by specifying the labels:

    ./cluster/kubectl.sh  get -w po -l context=docker-k8s-lab,name=mysql-pod
NAME        READY     STATUS    RESTARTS   AGE
mysql-pod   1/1       Running   0          4m

    This is also the selector label used by Service to target Pods.

    When a Service is run on a node, the kubelet adds a set of environment variables for each active Service. It supports both Docker links compatible variables and simpler {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT variables, where the Service name is upper-cased and dashes are converted to underscores.

    Our service name is “mysql-service” and so MYSQL_SERVICE_SERVICE_HOST and MYSQL_SERVICE_SERVICE_PORT variables are available to other pods.

Kubernetes also allows services to be resolved using DNS configuration. Send a Pull Request for adding this functionality to the lab as explained in #62.

Start WildFly Replication Controller

  1. Start WildFly replication controller:

    ./cluster/kubectl.sh create -f ../../attendees/kubernetes/app-wildfly-rc.yaml
replicationcontrollers/wildfly-rc

    It uses the following configuration file:

    apiVersion: v1
kind: ReplicationController
metadata:
  name: wildfly-rc
  labels:
    name: wildfly
    context: docker-k8s-lab
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: wildfly
    spec:
      containers:
      - name: wildfly-rc-pod
        image: arungupta/wildfly-mysql-javaee7:k8s
        ports:
        - containerPort: 8080

  2. Check status of the Pod inside Replication Controller:

    ./cluster/kubectl.sh get po
NAME               READY     STATUS    RESTARTS   AGE
mysql-pod          1/1       Running   0          1h
wildfly-rc-w2kk5   1/1       Running   0          6m

  3. Get IP address of the Pod:

    ./cluster/kubectl.sh get -o template po wildfly-rc-w2kk5 --template={{.status.podIP}}
10.246.1.23

Access the application (using node)

  1. Log in to node:

    vagrant ssh minion-1

  2. Access the application using curl http://10.246.1.23:8080/employees/resources/employees/ and replace IP address with the one obtained earlier:

    Last login: Thu Jul 16 00:24:36 2015 from 10.0.2.2
[vagrant@kubernetes-minion-1 ~]$ curl http://10.246.1.23:8080/employees/resources/employees/
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><collection><employee><id>1</id><name>Penny</name></employee><employee><id>2</id><name>Sheldon</name></employee><employee><id>3</id><name>Amy</name></employee><employee><id>4</id><name>Leonard</name></employee><employee><id>5</id><name>Bernadette</name></employee><employee><id>6</id><name>Raj</name></employee><employee><id>7</id><name>Howard</name></employee><employee><id>8</id><name>Priya</name></employee></collection>

Access the application (using proxy)

Send a PR for arun-gupta#80

Deploy Java EE Application (one configuration file)

Kubernetes allow multiple resources to be specified in a single configuration file. This allows to create a “Kubernetes Application” that can consists of multiple resources easily.

Previous section showed how to deploy the Java EE application using multiple configuration files. This application can be delpoyed using a single configuration file as well.

  1. Start the application using the configuration file:

    apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
  labels:
    name: mysql-pod
    context: docker-k8s-lab
spec:
  containers:
    -
      name: mysql
      image: mysql:latest
      env:
        -
          name: "MYSQL_USER"
          value: "mysql"
        -
          name: "MYSQL_PASSWORD"
          value: "mysql"
        -
          name: "MYSQL_DATABASE"
          value: "sample"
        -
          name: "MYSQL_ROOT_PASSWORD"
          value: "supersecret"
      ports:
        -
          containerPort: 3306
----
apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  labels:
    name: mysql-pod
    context: docker-k8s-lab
spec:
  ports:
    # the port that this service should serve on
    - port: 3306
  # label keys and values that must match in order to receive traffic for this service
  selector:
    name: mysql-pod
    context: docker-k8s-lab
----
apiVersion: v1
kind: ReplicationController
metadata:
  name: wildfly-rc
  labels:
    name: wildfly
    context: docker-k8s-lab
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: wildfly
    spec:
      containers:
      - name: wildfly-rc-pod
        image: arungupta/wildfly-mysql-javaee7:k8s
        ports:
        - containerPort: 8080

    Notice that each section, one each for MySQL Pod, MySQL Service, and WildFly Replication Controller, is separated by ----.

  2. Start the application:

    ./cluster/kubectl.sh create -f ../../attendees/kubernetes/app.yaml
pods/mysql-pod
services/mysql-service
replicationcontrollers/wildfly-rc

  3. Access the application using Access the application (using node) or Access the application (using proxy).

Rescheduling Pods

Replication Controller ensures that specified number of pod “replicas” are running at any one time. If there are too many, the replication controller kills some pods. If there are too few, it starts more.

WildFly Replication Controller is already running with one Pod. Lets delete this Pod and see how a new Pod is automatically rescheduled.

  1. Find the Pod’s name:

    ./cluster/kubectl.sh get po
NAME               READY     STATUS    RESTARTS   AGE
wildfly-rc-w2kk5   1/1       Running   0          6m

  2. Delete the Pod:

    ./cluster/kubectl.sh delete po wildfly-rc-w2kk5
pods/wildfly-rc-w2kk5

    Status of the Pods can be seen in another shell:

    ./cluster/kubectl.sh get -w po
NAME               READY     STATUS    RESTARTS   AGE
wildfly-rc-w2kk5   1/1       Running   0          2m
NAME               READY     STATUS    RESTARTS   AGE
wildfly-rc-xz6wu   0/1       Pending   0         2s
wildfly-rc-xz6wu   0/1       Pending   0         2s
wildfly-rc-xz6wu   0/1       Pending   0         12s
wildfly-rc-xz6wu   0/1       Running   0         14s
wildfly-rc-xz6wu   1/1       Running   0         22s

    Notice how Pod with name “wildfly-rc-w2kk5” was deleted and a new Pod with the name “wildfly-rc-xz6wu” was created.

Scaling Pods

Replication Controller allows dynamic scaling up and down of Pods.

  1. Scale up the number of Pods:

    ./cluster/kubectl.sh scale --replicas=2 rc wildfly-rc
scaled

  2. Status of the Pods can be seen in another shell:

    ./cluster/kubectl.sh get -w po
NAME               READY     STATUS    RESTARTS   AGE
wildfly-rc-bgtkg   1/1       Running   0          3m
NAME               READY     STATUS    RESTARTS   AGE
wildfly-rc-bymu7   0/1       Pending   0          2s
wildfly-rc-bymu7   0/1       Pending   0         2s
wildfly-rc-bymu7   0/1       Pending   0         2s
wildfly-rc-bymu7   0/1       Running   0         3s
wildfly-rc-bymu7   1/1       Running   0         12s

    Notice a new Pod with the name “wildfly-rc-bymu7” is created.

  3. Scale down the number of Pods:

    ./cluster/kubectl.sh scale --replicas=1 rc wildfly-rc
scaled

  4. Status of the Pods using -w is not shown correctly #11338. But status of the Pods can be seen correctly as:

    ./cluster/kubectl.sh get po
NAME               READY     STATUS    RESTARTS   AGE
wildfly-rc-bgtkg   1/1       Running   0          9m

    Notice only one Pod is running now.

Application Logs

  1. Get a list of the Pods:

    ./cluster/kubectl.sh get po
NAME               READY     STATUS    RESTARTS   AGE
mysql-pod          1/1       Running   0          18h
wildfly-rc-w2kk5   1/1       Running   0          16h

  2. Get logs for the WildFly Pod:

    ./cluster/kubectl.sh logs wildfly-rc-w2kk5
=> Starting WildFly server
=> Waiting for the server to boot
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/wildfly

  . . .

Logs can be obtained for any Kubernetes resources using this way. Alternatively, the logs can also be seen by logging into the node:

  1. Log in to the node VM:

    > vagrant ssh minion-1
Last login: Fri Jun  5 23:01:36 2015 from 10.0.2.2
[vagrant@kubernetes-minion-1 ~]$

  2. Log in as root:

    [vagrant@kubernetes-minion-1 ~]$ su -
Password:
[root@kubernetes-minion-1 ~]#

    Default root password for VM images created by Vagrant is ‘vagrant’.

  3. See the list of Docker containers running on this VM:

    docker ps

  4. View WildFly log as:

    docker logs $(docker ps | grep arungupta/wildfly | awk '{print $1}')

  5. View MySQL log as:

    docker logs <CID>

Delete Kubernetes Resources

Individual resources (service, replication controller, or pod) can be deleted by using delete command instead of create command. Alternatively, all services and replication controllers can be deleted using a label as:

kubectl delete -l se,po context=docker-k8s-lab

Stop Kubernetes Cluster

> ./cluster/kube-down.sh
Bringing down cluster using provider: vagrant
==> minion-1: Forcing shutdown of VM...
==> minion-1: Destroying VM and associated drives...
==> master: Forcing shutdown of VM...
==> master: Destroying VM and associated drives...
Done

Debug Kubernetes Master

  1. Log in to the master as:

    vagrant ssh master
Last login: Wed Jul 15 20:36:32 2015 from 10.0.2.2
[vagrant@kubernetes-master ~]$

  2. Log in as root:

    [vagrant@kubernetes-master ~]$ su -
Password:
[root@kubernetes-master ~]#

    Default root password for VM images created by Vagrant is ‘vagrant’.

  3. Check the containers running on master:

    CONTAINER ID        IMAGE                                                                               COMMAND                CREATED             STATUS              PORTS               NAMES
dc59a764953c        gcr.io/google_containers/etcd:2.0.12                                                "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_etcd-container.fa2ab1d9_etcd-server-kubernetes-master_default_7b64ecafde589b94a342982699601a19_2b69c4d5
b722e22d3ddb        gcr.io/google_containers/kube-scheduler:d1107ff3b8fcdcbf5a9d78d9d6dbafb1            "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_kube-scheduler.7501c229_kube-scheduler-kubernetes-master_default_98b354f725c1589ea5a12119795546ae_b81b9740
38a73e342866        gcr.io/google_containers/kube-controller-manager:fafaf8100ccc963e643b55e35386d713   "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_kube-controller-manager.db050993_kube-controller-manager-kubernetes-master_default_f5c25224fbfb2de87e1e5c35e6b3a293_dcd4cb5d
01001de6409e        gcr.io/google_containers/kube-apiserver:cff9e185796caa8b281e7d961aea828b            "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_kube-apiserver.7e06f4e1_kube-apiserver-kubernetes-master_default_829f8c23fd5fc7951253cac7618447fc_b39c0a5d
0f8ccb144ece        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_kube-scheduler-kubernetes-master_default_98b354f725c1589ea5a12119795546ae_eb1efcac
0b8f527456c0        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_kube-apiserver-kubernetes-master_default_829f8c23fd5fc7951253cac7618447fc_5dd4dee7
39d9c41ab1a2        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_kube-controller-manager-kubernetes-master_default_f5c25224fbfb2de87e1e5c35e6b3a293_522972ae
d970ddff7046        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_etcd-server-kubernetes-master_default_7b64ecafde589b94a342982699601a19_fa75b27f