ubuntu-20-devops-tools

This repo contains support files to create a devops environment on Ubuntu 20.04. The tools are all open-source.

VirtualBox Ubuntu 20.04 LTS Setup

Please follow the instructions in this video to set up your VM. The instructions are for a Windows host, but should also work for Mac/Linux. I suggest watching this even if you know what you are doing. Instructions to install Ubuntu on Oracle Virtual Box If using Virtual Box 7, I recommend NOT using the automatic install mechanism. You should choose the minimal install, as you can install desktop tools on an as-need basis. Depeding on how tool projects move and publish resources, the script may fail as urls change, and downloads become unavailable. As of October 2022, this script ran without errors with the -e flag set. You can of course use this script on bare-metal Ubuntu 20.04, and with minor modifications it will run on any debian linux flavour.

Additional information needed for the Virtual Box setup

Link to downlod Ubuntu 20.04 ISO

• 4096mb memory (more is OK).
• 35Gb disk (more is OK) OS and tools will use about 15gb.
• 2 Virtual CPU's (more is OK).
• Video memory set to 64mb may be better for large screens.
• Setting up bidirectional clipboard sharing in VirtualBox (YourVM)->Settings->General->Advanced->Shared Clipboard may be useful. This will only work after the VMWare guest estensions are installed.
• I suggest not scaling the display in VirtualBox, nor in Ubuntu's Display Settings, keep it at 100%. Most applications (like Chrome, VSCode, Terminal) offer a Zoom mode or Font Size, which is preferrable from a performance standpoint.

Todo before installing VirtualBox Guest additions (inside Ubuntu)

In a terminal run: (Ctrl+Alt+T to launch Terminal)

sudo apt install -y build-essential dkms linux-headers-$(uname -r)

After completing the Virtual Box setup (Make sure you reboot to active Guest Additions)

Install git

In a terminal run:

sudo apt-get install -y git

Install VS Code

In a terminal run:

sudo snap install --classic code

Click 'start', then find code, start VSCode, add to Favorites on your toolbar

Clone this repo

In a terminal run:

cd ~ && \
mkdir git && \
cd git && \
git clone https://github.com/brunzefb/ubuntu-20-devops-tools.git

Run script to install the tooling

Review the provision.sh script first, then

cd $HOME/git/ubuntu-20-devops-tools
bash provision.sh

What does the script install in terms of standard Ubuntu packages that is interesting?

Package name	Comments
build_essential	Compiler needed to build tools.
wget	Tool to do a REST get from an HTTP(s) server.
figlet	Tool to display large text titles.
git-lfs	Git extension to allow handling large files.
default-mysql-client	The mysql cli tool to acces MySql databases.
jq	Json query tool.
tmux	Terminal virtualizer, keeps ssh sessions from timing out.
vim	Editor.
dnsutils	Dns lookup tools like dig and nslookup
openssl	To mess with certificates, and other crypt stuff.
zip	Command line zip tool.
unzip	Command line unzip tool.
ssh	Ssh client.
xclip	Command line tool that allows copying to clipboard, alias=clip
python3 pip	To install python libs.
zsh	Installs an alternate shell to bash.
curl	Command line REST utility.

Other devops tools that are installed:

Tool Name	Version	Comments
git	latest	Clones the git repo, and recompiles and installs it. Newer version needed for sparse checkouts.
aws_cli	2.4.27	AWS command line utility.
kubectl	1.22.2	To control Kubernetes/EKS clusters. For K8s v 1.22.
helm	3.8.1	To install helm charts (software packages) for K8s.
terraform	1.1.7	To provision cloud infrastructure.
terragrunt	0.36.6	To provision cloud infrastructure. Wrapper around terraform.
aws-iam-authenticator	0.5.5	Needed to authenticate to EKS cluster.
cloud-nuke	0.11.3	To destroy cloud infrastructure. Use at your own risk!
sops	3.7.2	Mozilla sops. Tool to encrypt/decrypt secrets.
krew	0.4.2	Plugin manager for Kubernetes. See installed plugins below.
packer	1.7.9	Tool to create Amazon Machine Images (Ami's) programmatically.
argo	3.2.6	Tool to manage kubernetes argo workflows.
argocd	2.3.1	Tool to manage argo-cd (a Kubernetes deployment tool).
kustomize	4.5.2	Tool to customize manifests deployed to K8s
redis-cli	latest	Compiled from source. To interact with redis cache servers.
oh-my-zsh	latest	Shell tooling and themes for zsh.
kubescape	2.0.149	Open source K8s security scanner.
docker	20.10.17	To manage containers and build images.
exa	010.1	Prettier, clearer ls utility.
crossplane cli	1.10.0	K8s cluster provisioning tool.
fluxctl	n/a	Tool to manage flux deployments.
keepass2	2.44	Open source password manager.

Installed Krew plugins (subset of useful ones shown)

Name	Comments
kubectl neat	Cleans up yaml rendered by -oyaml.
kubectl ns	Switches default namespaces. Shows current ns.
kubectl deprecations	Shows K8s API deprecations.
kubectl node-shell	Allows you to 'ssh' into a worker node by name.
kubectl skew	Shows versions of your cluster vs kubectl version.
kubectl view-secret	Dumps secrets with built-in base64 decoding.
kubectl score	Linter for manifests.
kubectl df-pv	Shows disk free on persistent volumes.
kubectl pv-migrate	Copy one PVC to another.
kubectl ctx	Show and change the K8s context.
kubectl viewnode	Shows pods running on nodes.
kubectl ttsum	Shows tains/tolerations.
kubectl unused-volumes	Shows unreferenced volumes.

Installed Visual Studio Code Extensions

Name	What it does
Packer	Provides syntax highlighting for .pkr.hcl files.
vscode-base64	Allows in-place base64 encoding/decoding. Useful in dealing with secrets.
unique-lines	Removes duplicate lines.
xml	Allows pretty-printing xml.
json-pretty-printer	Allows pretty-printing json.
open-in-terminal	Allows you to click on a folder, run the open in terminal command. Opens terminal with the folder as current directory.
terraform	Terraform .hcl syntax highligher.
vscode-kubernetes-tools	Gives you a right-click kubectl apply functionality from vs code explorer.
python	Pyhon language support.
subtle-brackets	Allows you to visualize which bracket matches with corresponding one in an unobtrusive way.
vscode-hex dump	Shows your content as hex dump.
jumpy	Allows crazy keyboard navigation.
better-align	Allows alignment of code.
tabspacer	Helps convert tabs to spaces and vice versa.
git lens	Mostly useful for git diffs directly in vscode.
yaml	Yaml language support for vscode.
vscode-base64	Encode/decode base64 strings

Other features

• Syntax highlighting in the terminal, see here.
• Supercharged 'cd' command, zsh-z aliased to 'j', see here.
• useful git aliases (stored in ~/.gitconfig).
• zsh-kubectl-prompt -- useful prompt to indicate cluster and namespace. The blue prompt on the right gets out of the way when you type. If you dislike the prompt, simply run unset RPROMPT in the terminal.

Useful git aliases (you can see them all in ~/.gitconfig)

Also see the standard zsh git shortcuts here

Name	What it does
git addall	Equivalent to git add . --all.
git st	Equivalent to git status.
git cm "msg"	Equivalent to git -am "msg".
git branch-name	equivalent to git rev-parse --abbrev-ref HEAD.
git publish	Pushes new branch to origin, equivalent to git push -u origin [current branch].
git unpublish	Deletes the branch on the server, equivalent to git push -u :branch.
git co	Equivalent to git checkout.
git conflicts	Shows conflicted files, diff --name-only --diff-filter=U.
git squash	Does interactive rebase from where you branched from develop branch.
git current	Shows the sha1 of the HEAD commit.
git cob	Equivalent to git checkout -b.

Git with ssh

The script will create a ~/.ssh/id_rsa (private key) and ~/.ssh/id_rsa.pub (public key). The .zshrc will start an ssh agent and load the key, but for things to work you must upload your public key to Github or git provider. More info about github and ssh can be found here. To use your own key, just edit the ~/.ssh/id_rsa file, and replace it with your own personal private key content.

Opening a ZSH shell by default in vscode.

VSCode will open a bash shell by default. You can either type zsh to get the zshell one.

1. Open Visual Studio Code.
2. Press CTRL + SHIFT + P to open the Command Palette.
3. Search for “Terminal: Select Default Profile” (previously “Terminal: Select Default Shell”)
4. Pick zsh

Python script utilites

To use these, you should make sure they are executable (chmod +x) and copy them to a place like ~/.local/bin

Script name	What it does	Example
dns4elb	This script allows you to check if an amazon elb dns name corresponds to a dns name. When using ingress on AWS EKS, the ingress points to a ELB/ALB loadbalancer name, but also usually has a DNS name. The external-dns deployment takes care of creating and pointing the DNS name to the ELB/ALB. To make sure the assignent happened, you can use this utility, which looks up the IP's and compares them.	dns4lb foo.myhost.com c9287718dbf8d453db55e7628ce240b8-173467794.us-east-2.elb.amazonaws.com
wait4dns	This is useful when your company DNS updates are slow. After publishing a new service, Route53 will publish the new name, but this takes some time to propagate to your ISP's domain name server or company DNS if you are on a VPN. Propagation can take a while (sometimes 15 mins or more), and you typically want to know when the record is available so that you can use the service. wait4dns does an automatic dns lookup every second, and exits when the name is available. If the dns entry is there, it will exit immediately.	wait4dns google.ca