Crucible Production Kubernetes

Terraform, Ansible and kubernetes code to install a production instance of Crucible

This stack utilizes Rancher k3s distribution with a rancher front end for web role based access to the cluster.

The kubernetes cluster utilizes the kubernetes ingress nginx controller, with a longhorn persistent storage backend and MetalLB for IP allocation.

Requirements

• vSphere Server with Distributed Port Groups.
• Ubuntu 20.04 template, with snapshot.
• Ubuntu 20.04 Desktop Jump Box.

Ubuntu Jump Box & Quick Start

1. Create an Ubuntu 20.04 Desktop VM as a jump box.

   • 30GB+ HD
   • 4 processors
   • 6GB RAM
   • Install Ubuntu Desktop, git and vscode.

2. Clone this repository: git clone https://github.com/sei-noconnor/k3s-production.git

3. Navigate to: cd k3s-production.

4. Rename env.example to env and set the variables in this file, many required defaults are intentionally missing, pay close attention to TOKEN and PASS variables.

5. Run sudo ./prep.sh. This will install the binaries needed for the appliance, including terraform, ansible and the kubernetes binaries

ENV File

The env file is a set a variables for customizing the stack to your unique infrastructure. It is laid out like an ini file and similar in function to docker-compose env file. The ./setup.sh script will replace these values in the relevant files.

Networking

A standard network will need to be available and a block of approximately 10 IPs is required 7 for kubernetes, 3 for various ingress addresses.

Set the variables in the [network] section of the env file.

DNS

This stack uses an kubernetes ingress nginx. Hostname's must be used when accessing the applications. While most Crucible applications use path based routing to limit entires in DNS there are third party applications that need entires as well. If you are setting up an initial Proof of Concept a host file entry can be used.

Deploying Kubernetes VMs

This repo contains terraform to deploy VMs needed for kubernetes. it utilizes your existing ubuntu template. You may wish to modify the VMs configuration such as processors and RAM, or last octet of IP. edit the terraform/variables.auto.tfvars at the bottom you can modify the vm configuration.

1. within the terraform directory run terraform init
2. run terraform plan and verify that the correct resources will be created.
3. run terraform apply again verify the resources and confirm you want to apply.
4. your VMs will begin to be provisioned and will be accessible at their specified IPs listed in the output of the terraform.

Installing Kubernetes with Ansible.

This repo contains an ansible playbook to install k3s master, slave and worker nodes. modify k3s-ansible/inventory/sample/hosts.ini with the correct values based on the terraform output.

1. withins the k3s-ansible directory run ansible-playbook -i inventory/sample/hosts.ini site.yml -K
2. You will be prompted to enter the BECOME PASSWORD this should be the password
3. once ansible completes run kubectl get nodes to verify the kubernetes cluster.

Certificate

Certificates are essential to the stack and must be valid, without proper certificates the deployment will fail.

Certificates can be generated by setting the GENERATE_CERTS value to true in the env. you will still need to provide your vcenter root-ca certificate.

If you are providing your own certificate be sure to place certificates in common/certs with the following file names.

Type	Filename
host	host.pem
host key	host-key.pem
root CA	root-ca.pem
vSphere	vsphere.pem

Deploying Crucible

You should now have a kubernetes cluster deployed with Terraform and configured with Ansible. Verify your env file has all the values needed.

from the root of the k3s-production directory run ./setup.sh

Once complete you should have a working Crucible stack. Navigate to any of the URLs and login.

URLs

---

Replace crucible.io with the <DOMAIN> environment URL if changed.

Application	URL
Alloy	crucible.io/alloy
Alloy API	crucible.io/alloy/api
Caster	crucible.io/caster
Caster API	crucible.io/caster/api
Identity	crucible.io/identity
Identity API	crucible.io/identity/api
Player	crucible.io/player
Player API	crucible.io/player/api
Steamfitter	crucible.io/steamfitter
Steamfitter API	crucible.io/steamfitter/api
VM	crucible.io/vm
VM API	crucible.io/vm/api
VM Console	crucible.io/vm/console

3rd Party URLs

---

3rd Party Applications	URL
GitLab	gitlab.crucible.io
StackStorm	crucible.io/stackstorm