This document walks you through steps for a number of scenarios covering:
- Adding a workload cluster
- Adding an application to a workload cluster (including associated AWS resources)
- Connecting to a workload cluster to check reconciliation status
- Updating an application running in a workload cluster
- Upgrading a workload cluster
- Removing an application from a workload cluster
- Removing a workload cluster
For illustration purposes, these scenarios are based on the example of a product catalog application developed by the commercial department of an organisation.
- The department
runs two clusters
commercial-staging
andcommercial-prod
. - The product catalog application comprises two microservices which are owned by different teams.
- API server: this provides the backend, and makes use of an AWS DynamoDB table to store its state.
- Front end server: provides a web front end service that can be accessed via a browser.
Pre-prepared manifest files for the application workloads are included in the
repos/apps-manifests
directory as follows:
reops/apps-manifests/product-catalog-api-manifests
: contains manifests for two distinct versions of the API server, labelledV1
andV2
.reops/apps-manifests/product-catalog-fe-manifests
: contains manifests for the front end server. You can see that both of these include overlays for both thecommercial-staging
andcommercial-prod
clusters.
In this section you will create a new workload cluster called commercial-staging
. To achieve this, you need to use the supplied templates to create manifest files for the new cluster in
gitops-system
, and update clusters-config/kustomization.yaml
Once done, you then push the
changes so that flux can pick them up and act on them.
You can make the required changes quickly using the script add-cluster.sh
:
cd ~/environment
eks-multi-cluster-gitops/bin/add-cluster.sh ./gitops-system commercial-staging
Once done, commit and push the changes as follows:
cd gitops-system
git add .
git commit -m "Add cluster commercial-staging"
git push
Use
kubectl get kustomization -n flux-system
to monitor the creation of resources for the new cluster.
You can repeat the same process to create the commercial-prod
cluster.
The add-cluster.sh
script performs the following steps. You can choose to execute these steps instead of running the script. Please ensure your working directory is set to ~/environment
before executing.
-
Instantiate the
cluster-configs
template: This creates a folder for the newcommercial-staging
cluster undercluster-configs
, and copies the template. It then replaces all occurances ofcluster-name
in the template withcommercial-staging
.mkdir -p gitops_system/clusters-config/commercial-staging cp -R gitops_system/clusters-config/template/* gitops_system/clusters-config/commercial-staging grep -RiIl 'cluster-name' gitops_system/clusters-config/commercial-staging| xargs sed -i "s/cluster-name/commercial-staging/g"
-
Instantiate the
cluster
template: This create a folder for thecommcercial-staging
cluster underclusters
, and copies the template. It then replaces all occurances ofcluster-name
in the template withcommercial-staging
.mkdir -p gitops_system/clusters/commercial-staging cp -R gitops_system/clusters/template/* gitops_system/clusters/commercial-staging grep -RiIl 'cluster-name' gitops_system/clusters/commercial-staging | xargs sed -i "s/cluster-name/commercial-staging/g"
-
Instantiate the
workloads
template. This creates a folder for thecommcercial-staging
cluster underworkloads
and copies the template. It then replaces all occurences ofcluster-name
in the template withcommercial-staging
.mkdir -p gitops_system/workloads/commercial-staging cp -R gitops_system/workloads/template/* gitops_system/workloads/commercial-staging grep -RiIl 'cluster-name' gitops_system/workloads/commercial-staging | xargs sed -i "s/cluster-name/commercial-staging/g"
-
Add
commercial-staging
toclusters-config/kustomization.yaml
. This forces Flux to pick up the new cluster config.yq -i e ".resources += [\"commercial-staging\"]" gitops_system/clusters-config/kustomization.yaml
In this section you will add an application product-catalog-api
to the cluster commercial-staging
. To achieve this, you need to use the supplied template to
create manifest files for the new application in gitops-workloads/commercial-staging
, and update commercial-staging/kustomization.yaml
. Once done, you then push the
changes so that flux can pick them up and act on them.
First, prepare a repo for product-catalog-api-manifests
as follows:
cd ~/environment
gh repo create --private --clone product-catalog-api-manifests
cp -r eks-multi-cluster-gitops/repos/apps-manifests/product-catalog-api-manifests/v1/* product-catalog-api-manifests/
cd product-catalog-api-manifests
git add .
git commit -m "baseline version"
git branch -M main
git push --set-upstream origin main
Next, make the required changes in gitops-workloads
using the script add-cluster-app.sh
:
cd ~/environment
eks-multi-cluster-gitops/bin/add-cluster-app.sh \
./gitops-workloads \
commercial-staging product-catalog-api staging main \
eks-multi-cluster-gitops/initial-setup/secrets-template/git-credentials.yaml \
~/.ssh/gitops ~/.ssh/gitops.pub \
"github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=" \
./sealed-secrets-keypair-public.pem
Once done, commit and push the changes as follows:
cd gitops-workloads
git add .
git commit -m "Add product-catalog-api to commercial-staging"
If this is the first commit on gitops-workloads
, you need to rename the branch to main
, and push the changes as follows:
git branch -M main
git push --set-upstream origin main
Otherwise, you just push the changes as follows:
git push
To view the reconciliation of resources in the workload cluster, and verify that application
resources have been created, see the section Connect to a workload cluster. You can also see the resources created in the workload cluster through the EKS console. In this case, you have to access the EKS console using the IAM entity whose ARN is set in EKS_CONSOLE_IAM_ENTITY_ARN
environment variable during the initial setup.
You can repeat the same process to add the application to the commercial-prod
cluster
(using the same manifests repo).
You can also repeat this process to add the front-end application product-catalog-fe
to
each cluster. You will
find pre-prepared manifests for this in the product-catalog-fe-manifests
repo. Note these
include overlays for each of commercial-staging
and commercial-prod
.
The add-cluster-app.sh
script performs the following steps. You can choose to execute these steps instead of running the script. Please ensure your working directory is set to ~/environment
before executing.
- Ensure a directory for the application exists under
gitops-workloads/commercial-staging
: create a directory if it does not yet exist (i.e. this is the first app in this cluster).
mkdir -p gitops-workloads/commercial-staging/product-catalog-api
- Ensure a
kustomization.yaml
file exists: check if the file exists, and if not then create one by copying the template (only happens for the first app that is added to the cluster).
if [[ ! -f gitops-workloads/commercial-staging/kustomization.yaml ]]
then
cp gitops-workloads/template/kustomization.yaml gitops-workloads/commercial-staging
fi
- Instantiate the workloads application template: this copies the application template
from
gitops-workloads/template/app-template
togitops-workloads/commercial-staging/product-catalog-api
, and then updates the cluster name and app name to the correct values.
cp -R gitops-workloads/template/app-template/* gitops-workloads/commercial-staging/product-catalog-api
grep -RiIl 'cluster-name' gitops-workloads/commercial-staging/product-catalog-api | xargs sed -i "s/cluster-name/commercial-staging/g"
grep -RiIl 'app-name' gitops-workloads/commercial-staging/product-catalog-api | xargs sed -i "s/app-name/product-catalog-api/g"
grep -RiIl 'branch-name' gitops_workloads/commercial-staging/product-catalog-api | xargs sed -i "s/branch-name/main/g"
grep -RiIl 'overlay-dir-name' gitops-workloads/commercial-staging/product-catalog-api | xargs sed -i "s/overlay-dir-name/staging/g"
- Prepare the sealed secret for the application: this prepares a sealed secret for the
application, which contains the Git credentials needed to access the repo. A
temporary file is used to store a copy of the Git credentials template (which is
a K8s
Secret
). This file is updated to use the correct name, public/private key pair, and known hosts.kubeseal
is then used to create a sealed secret, using a supplied.pem
file containing a key to encrypt the sealed secret which is stored ingitops-workloads/commercial-staging/product-catalog-api/git-secret.yaml
.
tmp_git_creds=$(mktemp /tmp/git-creds.yaml.XXXXXXXXX)
cp eks-multi-cluster-gitops/initial-setup/secrets-template/git-credentials.yaml $tmp_git_creds
APP_NAME=product-catalog-api yq e '.metadata.name=strenv(APP_NAME)' -i $tmp_git_creds
KEY=$(cat ~/.ssh/gitops | base64 -w 0) yq -i '.data.identity = strenv(KEY)' $tmp_git_creds
CERT=$(cat ~/.ssh/gitops.pub | base64 -w 0) yq -i '.data."identity.pub" = strenv(CERT)' $tmp_git_creds
HOSTS=$(echo "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=" | base64 -w 0) yq -i '.data.known_hosts = strenv(HOSTS)' $tmp_git_creds
kubeseal --cert ./sealed-secrets-keypair-public.pem --format yaml <$tmp_git_creds >gitops-workloads/commercial-staging/product-catalog-api/git-secret.yaml
rm $tmp_git_creds
- Add entry to
kustomization.yaml
: this forces Flux to pick up the new application.
yq -i e ".resources += [\"product-catalog-api\"]" gitops-workloads/commercial-staging/kustomization.yaml
To connect to <cluster-name>
workload cluster using kubeconfig
stored as a Secret
- In a terminal window that is currently configured for the management cluster, obtain a config file for the workload cluster using:
unset KUBECONFIG
kubectl -n flux-system get secret <cluster-name>-eks-connection -n flux-system -o jsonpath="{.data.value}" | base64 -d > wl-kube.conf
export KUBECONFIG=wl-kube.conf
kubectl config current-context
(Replace <cluster-name>
with the cluster name).
Note: A convenience script eks-multi-cluster-gitops/bin/connect-to-cluster.sh
is provided to quickly connect to a cluster.
In a bash terminal source the script to generate the kubeconfig file and export the KUBECONFIG
environment variable.
source eks-multi-cluster-gitops/bin/connect-to-cluster.sh <cluster-name>
- To monitor the bootstrapping of the workload clusters (and subseuently for the
deployment of the applications into it), list the
Kustomization
resources using the following command:
kubectl get kustomization -n flux-system
In this section you will push a new version of the application product-catalog-api
in
the cluster commercial-staging
. To achieve this, you need to update the release tag in the app manifests in gitops-workloads/commercial-staging/product-catalog-api
. Once done, you then push the changes so that flux can pick them up and act on them.
The new version of the application makes use of a DynamoDB table which is created by Crossplane. So, the service account used by the application pods needs to be attached to an IAM role with the required permissions.
First, the policy document specified in the Policy
CRD that exists in gitops-workloads/commercial-staging/product-catalog-api/app-iam.yaml
has to be changed as follows:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1657328236602",
"Action": "dynamodb:*",
"Effect": "Allow",
"Resource": "arn:aws:dynamodb:${AWS_REGION}:${ACCOUNT_ID}:table/products-staging"
}
]
}
Next, commit this change:
cd ~/environment/gitops-workloads
git add .
git commit -m "add the IAM permissions required for v2"
git push
Next, update the repo for product-catalog-api
to the new version (v2):
cd ~/environment
cp -r eks-multi-cluster-gitops/repos/apps-manifests/product-catalog-api-manifests/v2/* product-catalog-api-manifests/
cd product-catalog-api-manifests
git add .
git commit -m "Updated version"
git push origin main
Next, tag the current commit as version 2.0 and push the tag:
git tag -a v2.0 -m "Version 2.0"
git push origin v2.0
Update the release tag in gitops-workloads/commercial-staging/product-catalog-api
. You
can do this using the update-cluster-app.sh
script as follows:
cd ~/environment
eks-multi-cluster-gitops/bin/update-cluster-app.sh ./gitops-workloads commercial-staging product-catalog-api v2.0
Finally, commit this change:
cd gitops-workloads
git add .
git commit -m "Updated product-catalog-api to v2.0 in commercial-staging"
git push
Monitor the reconciliation in the cluster. You can verify that the application deployment has been updated by checking the deployment history:
kubectl rollout history deployment/product-catalog-api-staging -n product-catalog-api
You can also verify that the container image in the deployment has been updated:
kubectl describe deployment/product-catalog-api-staging -n product-catalog-api
You can verify the creation of the DynamoDB table using the DynamoDB console, or via the AWS CLI:
aws dynamodb list-tables
The update-cluster-app.sh
script makes a single change as follows. Please ensure your working directory is set to ~/envionment
before executing.
yq -i e ".spec.ref.tag = \"v2.0\"" gitops-workloads/commercial-staging/product-catalog-api/git-repo.yaml
In this section you will upgrade the cluster commercial-staging
to a newer version
of Kubernetes. As described in Amazon EKS documentaiton on "Updating Kubernetes version", a cluster upgrade is a process of multiple steps. First you'll have to upgrade the Amazon EKS Cluster itslef, and after a successful completion of the cluster upgrade, you'll have to upgrade the nodes to the same Kubernetes version. In our sample, we are using Managed Node Group, so we will perform the upgrade of the node, by updating the Managed Node Group that was created with the cluster.
-
Open
gitops-system/clusters-config/commercial-staging/def/eks-cluster.yaml
. -
Change the value for
spec.parameters.eks-k8s-version
(e.g. from1.23
to1.24
). -
Commit changes.
cd ~/environment/gitops-system/ git add . git commit -m "upgrading commercial-staging cluster" git push
-
Confirm that the cluster is updating using the following commands:
eksctl get cluster eksctl get cluster --name <cluster-name>
Alternatively, you can check the cluster status on the EKS console.
-
Change the
gitops-system/clusters-config/commercial-staging/def/eks-cluster.yaml
file again. This time change the value forspec.parameters.mng-k8s-version
from 1.23 to 1.24. -
Commit again the changes.
cd ~/environment/gitops-system/ git add . git commit -m "updating managed node group version for commercial-staging cluster" git push
-
Confirm that the cluster is updating using the following commands:
eksctl get nodegroup --cluster <cluster-name> -o yaml
Alternatively, you can check the managed node group status on the EKS console.
In this section you will delete the application product-catalog-api
from
the cluster commercial-staging
. To achieve this, you need to remove the
entry for this application from gitops-workloads/commercial-staging/kustomization.yaml
.
Once done, you then push the changes so that flux can pick them up and act on them.
You also need to tidy up the gitops-workloads/commercial-staging
directory by removing
the product-catalog-api
directory.
You can make the required changes quickly using the script remove-cluster-app.sh
:
cd ~/environment
eks-multi-cluster-gitops/bin/remove-cluster-app.sh ./gitops-workloads commercial-staging product-catalog-api
Once done, commit and push the changes as follows:
cd gitops-workloads
git add .
git commit -m "Removed product-catalog-api from cluster commercial-staging"
git push
You can monitor the reconciliation of resources in the cluster using the method described previously.
The remove-cluster-app.sh
script performs the following steps. You can choose to execute these steps instead of running the script. Please ensure your working directory is set to ~/envionment
before executing.
- Remove
product-catalog-api
fromgitops-workloads/commrical-staging/kustomization.yaml
: this forces Flux to remove the application resources from the cluster.
yq -i e "del ( .resources[] | select (. == \"product-catalog-api\" ))" gitops-workloads/commercial-staging/kustomization.yaml
- Tidy up
gitops-workloads/commercial-staging
: remove the application folder from thegitops-workloads/commrical-staging
as it is no longer needed.
rm -rf gitops-workloads/commercial-staging/product-catalog-api
In this section you will delete the commercial-staging
cluster. To achieve this, you
need to remove the entry for this cluster from gitops-system/clusters-config/kustomization.yaml
.
this triggers Flux (via Crossplane) to remove the cluster.
You also need to tidy up various directories by removing the various cluster manifests.
Before proceeding, you should ensure that all applications running in the cluster have been removed.
You can make the required repo changes quickly using the script remove-cluster.sh
:
cd ~/environment
eks-multi-cluster-gitops/bin/remove-cluster.sh ./gitops-system ./gitops-workloads commercial-staging
Once done, commit and push the changes as follows:
cd gitops-system
git add .
git commit -m "Removed cluster commercial-staging"
git push
cd ../gitops-workloads
git add .
git commit -m "Removed cluster commercial-staging"
git push
You can monitor the reconciliation of resources in the management cluster using
kubectl get kustomization -n flux-system
The remove-cluster.sh
script performs the following steps. You can choose to execute these steps instead of running the script. Please ensure your working directory is set to ~/environment
before executing.
- Remove
commercial-staging
fromgitops-system/clusters-config/kustomization.yaml
: this forces Flux to remove the application resources from the cluster.
yq -i e "del ( .resources[] | select (. == \"commercial-staging\" ))" gitops-system/clusters-config/kustomization.yaml
- Tidy up
gitops-system
andgitops-workloads
repos:
rm -rf gitops-system/clusters-config/commercial-staging
rm -rf gitops-system/clusters/commercial-staging
rm -rf gitops-system/workloads/commercial-staging
rm -rf gitops-workloads/commercial-staging