From 21580ba90f5da04772225369278809274ecd9453 Mon Sep 17 00:00:00 2001 From: Luca Bertelli <53040714+banshee86vr@users.noreply.github.com> Date: Fri, 23 Feb 2024 22:50:52 +0100 Subject: [PATCH] feat: Clean + refactor (#8) --- README.md | 26 +++---- {argo/workflow => argo-workflow}/README.md | 44 +++++------ {argo/workflow => argo-workflow}/cd.yaml | 0 {argo/workflow => argo-workflow}/ci.yaml | 0 .../images/1_ci_submit.png | 0 .../images/2_ci_graph.png | 0 .../images/3_cd_submit.png | 0 .../images/4_cd_graph.png | 0 {argo/workflow => argo-workflow}/lang/go.yaml | 0 argo/README.md | 73 ------------------- argo/events/README.md | 1 - hello-world-app/README.md | 4 +- 12 files changed, 28 insertions(+), 120 deletions(-) rename {argo/workflow => argo-workflow}/README.md (70%) rename {argo/workflow => argo-workflow}/cd.yaml (100%) rename {argo/workflow => argo-workflow}/ci.yaml (100%) rename {argo/workflow => argo-workflow}/images/1_ci_submit.png (100%) rename {argo/workflow => argo-workflow}/images/2_ci_graph.png (100%) rename {argo/workflow => argo-workflow}/images/3_cd_submit.png (100%) rename {argo/workflow => argo-workflow}/images/4_cd_graph.png (100%) rename {argo/workflow => argo-workflow}/lang/go.yaml (100%) delete mode 100644 argo/README.md delete mode 100644 argo/events/README.md diff --git a/README.md b/README.md index 3d09b70..c8c0248 100644 --- a/README.md +++ b/README.md @@ -4,34 +4,26 @@ You can read the article at: -- On my personal portfolio website: +- My portfolio website: - On Medium: -## How to use vCluster, ArgoCD Events and Argo Workflow to manage short-live test ephemeral environments +## How to use vCluster and Argo Workflow to manage ephemeral test environments Project structure: ```text . -├── argo -│ ├── events -│ └── workflow -│ └── lang -├── hello-world-app -└── vcluster +├── argo-workflow +│ └── lang +└── hello-world-app ``` -- `argo` folder: All configurations made for - - `events`: Argo Events webhook, trigger (for Argo Workflow), and sensor - - `workflow`: CI/CD pipeline triggered by Argo Events - - `lang`: ArgoWorkflow Templates for supported languages +- `argo-workflow`: CI/CD pipeline templates + - `lang`: ArgoWorkflow Templates for supported languages - `hello-world-app` folder: Go Hello world application that prints a beautiful octopus 🐙 in ASCII code -- `vcluster` folder: All configurations made for creating VCluster where deploy the hello world application Key highlights from the article include: -1. **vCluster Usage**: The article introduces vCluster as a pivotal tool for creating lightweight, ephemeral Kubernetes clusters. It's so interesting how vCluster can be employed to instantiate and manage test environments on-demand, covering the scenarios where temporary clusters are essential for testing purposes. +1. **vCluster Usage**: The article introduces vCluster as a pivotal tool for creating lightweight, ephemeral Kubernetes clusters. It's interesting how vCluster can be employed to instantiate and manage test environments on-demand, covering the scenarios where temporary clusters are essential for testing. -2. **Argo Events Integration**: The article explores the integration of Argo Events, a tool designed for event-driven architecture in Kubernetes. It details how Argo Events can be utilized to trigger and manage events received from the SCM to manage the lifecycle of short-lived test environments. This allows for dynamic and automated responses to changes in the testing and security requirements. - -3. **Argo Workflow Implementation**: The article delves into the integration of Argo Workflow, a workflow engine for Kubernetes. It provides insights into how Argo Workflow can be configured to orchestrate the deployment, testing, and teardown processes efficiently. The tool enables the final user to define also another kind of process drawing DAGs (). This feature supports complex scenarios where there is a requirement to maximize parallelism when running tasks. +2. **Argo Workflow Implementation**: The article delves into the integration of Argo Workflow, a workflow engine for Kubernetes, providing a way to configure the orchestration of the deployment, testing, and teardown processes efficiently. The tool also enables the final user to define another kind of process drawing DAGs (). This feature supports complex scenarios where there is a requirement to maximize parallelism when running tasks. diff --git a/argo/workflow/README.md b/argo-workflow/README.md similarity index 70% rename from argo/workflow/README.md rename to argo-workflow/README.md index ac4ae02..d6a945c 100644 --- a/argo/workflow/README.md +++ b/argo-workflow/README.md @@ -1,6 +1,6 @@ # Configuration of Argo Workflow entities -## Prerequisites +## Requirements - Minikube - `kubectl` command-line tool installed and configured to connect to your Kubernetes cluster. @@ -13,7 +13,7 @@ ```bash minikube start helm repo add argo https://argoproj.github.io/argo-helm -helm install argo-workflows argo/argo-workflows +helm install argo-workflows argo/argo-workflows -n argo --create-namespace ``` This command installs Argo Workflows in the default namespace of your Kubernetes cluster. @@ -53,11 +53,11 @@ Argo Workflows provides a web-based UI for managing and monitoring workflows. To kubectl port-forward svc/argo-server -n argo 2746:2746 ``` -Now you can access the Argo Workflows UI by navigating to `http://localhost:2746` in your web browser. +Now, you can access the Argo Workflows UI by navigating to `http://localhost:2746` in your web browser. ### 5. Add privileges to Argo service accounts -> Add this privileges to the Argo service accounts are recommended only for demo purposes. **IT'S STRONGLY NOT RECOMMENDED TO REPLICATE THIS CONFIGURATION IN PRODUCTION EVINRONMENTS.** +> Add these privileges to the Argo service accounts are recommended only for demo purposes. **IT'S STRONGLY NOT RECOMMENDED TO REPLICATE THIS CONFIGURATION IN PRODUCTION ENVIRONMENTS.** This command adds `cluster-admin` clusterrole to `argo:argo-server` and `argo:default`. In this way, Argo Workflow can manage every kind of resource in every namespace of the cluster. @@ -66,11 +66,11 @@ kubectl create clusterrolebinding argo-admin-server --clusterrole=cluster-admin kubectl create clusterrolebinding argo-admin-default --clusterrole=cluster-admin --serviceaccount=argo:default -n argo ``` -> In production evironments it's strongly recommended to create a dedicated role to these service accounts allowing only required verbs on the resources managed by the workflows. +> In production environments, creating a dedicated role for these service accounts is strongly recommended, allowing only required verbs on the resources managed by the workflows. ### 6. Prepare secrets required by the pipelines -Just in case of a private Git repository you can run this command to allow the clone command executed by the pipeline `ci.yaml`: +Just in case of a private Git repository, you can run this command to allow the clone command executed by the pipeline `ci.yaml`: ```bash kubectl create secret generic github-token -n argo --from-literal=token=......... @@ -111,18 +111,16 @@ Alternatively, you can submit the workflow using the UI: ![Submit CI workflow via UI](images/1_ci_submit.png) -The CI pipeline performs these steps: +The CI pipeline performs these steps inside the [ci.yaml](https://github.com/banshee86vr/ephemeral-test-environment/blob/main/argo-workflow/ci.yaml) manifest: 1. **Cloning Repository**: Fetches the source code from the git repository. -2. **Building Application**: Utilizes the GoLang template to compile the Go application. +2. **Building Application**: Utilizes the GoLang template [go.yaml](https://github.com/banshee86vr/ephemeral-test-environment/blob/main/argo-workflow/lang/go.yaml) to compile the Go application. 3. **Building and Pushing Docker Image**: Packages the application into a Docker image and pushes it to the registry. -After the completion of all steps, you can check the correct status of every step: +After the completion of all steps, you can check the correct status of every step and locate the updated Docker image in your registry: ![CI workflow graph](images/2_ci_graph.png) -If all steps have been completed, you can find a new version of the Docker image in your registry. - ### 9. Submit the CD pipeline To submit the CD pipeline, you can use the [official APIs](https://argo-workflows.readthedocs.io/en/latest/rest-api/): @@ -135,17 +133,15 @@ Alternatively, you can submit the workflow using the UI: ![Submit CD workflow via UI](images/3_cd_submit.png) -The CD pipeline performs these steps: +The CD pipeline performs these steps inside the [cd.yaml](https://github.com/banshee86vr/ephemeral-test-environment/blob/main/argo-workflow/cd.yaml) manifest: -1. **Preparing an ephemeral environment**: Prepares an ephemeral environment using vCluster where the user can test the application inside an isolated Kubernetes cluster -2. **Deploy the application**: Deploy the application Helm chart on the vCluster just created +1. **Preparing an ephemeral environment**: Prepares a temporary environment using vCluster where the user can test the application inside an isolated Kubernetes cluster. +2. **Deploy the application**: Deploy the application Helm chart on the vCluster just created. After the completion of all steps, you can check the correct status of every step: ![CD workflow graph](images/4_cd_graph.png) -If all steps have been completed, you can check the status of your application deployed on the vCluster just created - ### 10. Access to the application To check how to access the application deployed on vCluster, you can run these commands to list all vCluster and to access it: @@ -157,24 +153,18 @@ $ vcluster list ------------------+----------+-----------------+---------+---------+-----------+-------------------------------+---------+--------- demo-pr-request | minikube | demo-pr-request | Running | 0.19.0 | | xxxx-xx-xx xx:xx:xx +0100 CET | 1h8m49s | OSS -$ ➜ vcluster connect demo-pr-request --namespace demo-pr-request -- kubectl get pod -n demo-pr-request +$ vcluster connect demo-pr-request --namespace demo-pr-request -- kubectl get pod -n demo-pr-request NAME READY STATUS RESTARTS AGE demo-pr-request-hello-world-7f6d78645f-bjmjc 1/1 Running 0 7s ``` -As reported [here](https://www.vcluster.com/docs/using-vclusters/access) you can expose in different ways the ephemeral vCluster created. - -- **Via Ingress**: An Ingress Controller with SSL passthrough support will provide the best user experience, but there is a workaround if this feature is not natively supported. - - - Kubernetes Nginx - - Traefik Proxy - - Emissary +As reported [here](https://www.vcluster.com/docs/using-vclusters/access), you can expose the ephemeral vCluster created differently. - Make sure your ingress controller is installed and healthy on the cluster that will host your virtual clusters. More details [here](https://www.vcluster.com/docs/using-vclusters/access#via-ingress) +- **Via Ingress**: An Ingress Controller with SSL passthrough support will provide the best user experience. Ensure your ingress controller is installed and healthy on the cluster hosting your virtual clusters. More details [here](https://www.vcluster.com/docs/using-vclusters/access#via-ingress) - **Via LoadBalancer service**: The easiest way is to use the flag `--expose` in vcluster create to tell vCluster to use a LoadBalancer service. It depends on the specific implementation of the host Kubernetes cluster. -- **Via NodePort service**: You can also expose the vCluster via a NodePort service. In this case, you have to create a NodePort service and change the `values.yaml` file to use for the creation of the vCluster. More details [here](https://www.vcluster.com/docs/using-vclusters/access#via-nodeport-service) -- **From Host **Cluster**: To access the virtual cluster from within the host cluster, you can directly connect to the vCluster service. Make sure you can access that service and then create a kube config in the following form: +- **Via NodePort service**: You can also expose the vCluster via a NodePort service. In this case, you must create a NodePort service and change the `values.yaml` file to use for the creation of the vCluster. More details [here](https://www.vcluster.com/docs/using-vclusters/access#via-nodeport-service) +- **From Host Cluster**: To access the virtual cluster from within the host cluster, you can directly connect to the vCluster service. Make sure you can access that service and then create a kube config in the following form: ```bash vcluster connect my-vcluster -n my-vcluster --server=my-vcluster.my-vcluster --insecure --update-current=false diff --git a/argo/workflow/cd.yaml b/argo-workflow/cd.yaml similarity index 100% rename from argo/workflow/cd.yaml rename to argo-workflow/cd.yaml diff --git a/argo/workflow/ci.yaml b/argo-workflow/ci.yaml similarity index 100% rename from argo/workflow/ci.yaml rename to argo-workflow/ci.yaml diff --git a/argo/workflow/images/1_ci_submit.png b/argo-workflow/images/1_ci_submit.png similarity index 100% rename from argo/workflow/images/1_ci_submit.png rename to argo-workflow/images/1_ci_submit.png diff --git a/argo/workflow/images/2_ci_graph.png b/argo-workflow/images/2_ci_graph.png similarity index 100% rename from argo/workflow/images/2_ci_graph.png rename to argo-workflow/images/2_ci_graph.png diff --git a/argo/workflow/images/3_cd_submit.png b/argo-workflow/images/3_cd_submit.png similarity index 100% rename from argo/workflow/images/3_cd_submit.png rename to argo-workflow/images/3_cd_submit.png diff --git a/argo/workflow/images/4_cd_graph.png b/argo-workflow/images/4_cd_graph.png similarity index 100% rename from argo/workflow/images/4_cd_graph.png rename to argo-workflow/images/4_cd_graph.png diff --git a/argo/workflow/lang/go.yaml b/argo-workflow/lang/go.yaml similarity index 100% rename from argo/workflow/lang/go.yaml rename to argo-workflow/lang/go.yaml diff --git a/argo/README.md b/argo/README.md deleted file mode 100644 index 83e02a3..0000000 --- a/argo/README.md +++ /dev/null @@ -1,73 +0,0 @@ -# Trigger Argo Workflow Pipelines with Argo Events - -Folders structure: - -``` -. -├── events -│ └── README.md -└── workflow - ├── README.md - ├── ci.yaml - ├── cd.yaml - ├── images - │ └── ... - └── lang - └── go.yaml -``` - -This CI/CD pipeline automates the build and deployment process for applications, focusing on a Dockerized GoLang application for this experiment. It consists of the following phases: - -1. **Cloning Repository**: Fetches the source code from the git repository. -2. **Building Application**: Utilizes the GoLang template to compile the Go application. -3. **Building and Pushing Docker Image**: Packages the application into a Docker image and pushes it to the registry. -4. **Preparing an ephemeral environment**: Prepares an ephemeral environment using vCluster where the user can test the application inside an isolated Kubernetes cluster -5. **Deploy the application**: Deploy the application Helm chart on the vCluster just created - -## Folder Structure -- `ci.yaml`: Argo Workflow Template for the Docker image build and push phases: - 1. Clone the git repository. - 2. Build the application using the GoLang template. - 3. Build and push the Docker image to the registry. -- `cd.yaml`: Argo Workflow Template for the deploy phases: - 1. Create a new vCluster - 2. Clone the repository with the application Helm chart - 3. Deploy on vCluster the application Helm chart -- `lang/go.yaml`: Argo Workflow Template to build Go applications. - -## Introduction -This experiment provides an overview of Argo Workflow Pipelines and Argo Events, including their concepts, creation, configuration, and integration for effective workflow management and event-driven architecture. - -### Argo Workflow Pipelines -Argo Workflows is an open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes. It enables users to define complex workflows as code, automating and orchestrating tasks efficiently. - -#### Key Concepts: -- **Workflow**: A directed acyclic graph (DAG) of tasks and dependencies, representing the workflow logic. -- **Steps**: Individual units of work within a workflow, typically containerized tasks. -- **Templates**: Reusable specifications for defining steps in workflows, allowing for modular and maintainable workflows. -- **Artifacts**: Data or files produced by steps within the workflow. -- **Parameters**: Inputs to templates or steps, allowing for dynamic workflow behavior. -- **Outputs**: Results produced by workflows, including artifacts and status information. - -#### Creation and Configuration: -1. **Define Workflow**: Write YAML or JSON specifications defining the workflow, including steps, dependencies, inputs, and outputs. -2. **Configure Resources**: Specify Kubernetes resources required for workflow execution, such as containers, volumes, and environment variables. -3. **Handle Errors and Retries**: Implement error handling and retry mechanisms to ensure robustness and reliability. -4. **Customize Workflow Logic**: Utilize advanced features like loops, conditionals, and parameterization to create flexible workflows. -5. **Optimize Performance**: Fine-tune resource utilization, parallelism, and scheduling for optimal performance and resource efficiency. - -### Argo Events -Argo Events extends Kubernetes with event-driven architecture capabilities, enabling users to trigger workflows and other actions in response to events within the Kubernetes ecosystem or external systems. - -#### Key Concepts: -- **Event Sources**: Sources of events, including Kubernetes resources, messaging systems, webhooks, and custom triggers. -- **Event Types**: Different types of events, such as resource creation, updates, or custom events defined by users. -- **Event Sensors**: Components responsible for detecting and processing events, triggering associated actions. -- **Handlers**: Actions or workflows executed in response to events, such as invoking Argo Workflows pipelines. - -#### Creation and Configuration: -1. **Define Event Sources**: Configure event sources to monitor for relevant events, specifying filters and event types. -2. **Set Up Event Sensors**: Deploy event sensors to detect events from configured sources and route them to appropriate handlers. -3. **Implement Handlers**: Define handlers to execute desired actions or workflows in response to detected events. -4. **Monitor and Debug**: Monitor event processing and troubleshoot issues using logs, metrics, and observability tools. -5. **Integrate with Workflows**: Integrate Argo Events with Argo Workflows pipelines to automate workflows based on detected events, enabling event-driven automation and orchestration. \ No newline at end of file diff --git a/argo/events/README.md b/argo/events/README.md deleted file mode 100644 index d7815bc..0000000 --- a/argo/events/README.md +++ /dev/null @@ -1 +0,0 @@ -# Configuration of Argo Events diff --git a/hello-world-app/README.md b/hello-world-app/README.md index 0328de6..fde36d1 100644 --- a/hello-world-app/README.md +++ b/hello-world-app/README.md @@ -1,6 +1,6 @@ # Go Hello World Web Application - Print a beautiful ASCII octopus -This is a basic web application written in Go that displays an ASCII octopus when accessed calling the exposed HTTP listener. +This folder contains a basic web application in Go that displays an ASCII octopus when accessed by calling the exposed HTTP listener. ## Prerequisites @@ -54,7 +54,7 @@ Make sure you have Go installed on your machine. You can download it from [here] ## Docker -Alternatively, you can use Docker to run the application in a container and deploying it on Kubernetes. +Alternatively, you can use Docker to run the application in a container and deploy it on Kubernetes. 1. Build the Docker image: