Skip to content

Commit

Permalink
feat: update documentation (#13)
Browse files Browse the repository at this point in the history 
merge after: 
- #12
- #9

---------

Co-authored-by: Silvija Tovernic <[email protected]>
Co-authored-by: Marko Paulic <[email protected]>
  • Loading branch information
@stovernic
3 people authored Sep 23, 2024
1 parent 8a2875c commit 5d7d2d4
Show file tree
Hide file tree
Showing 4 changed files with 266 additions and 65 deletions.
65 changes: 65 additions & 0 deletions README-terraform.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# wp10-image-factory
Frame Contract WP10 Image Factory

# Contents
- `terraform-init` - terraform code for setting up an Azure Storage Account that can be used for saving the terraform state
- `terraform` - terraform modules for deploying the resource group, virtual network, subnet, VM

# Requirements
- Access to Azure Resource Manager
- `terraform >=1.0.0`, tested with `1.9.5`
- Azure CLI, tested with `2.64.0`
- Azure credentials saved as environment variables:
```
export AZURE_CLIENT_ID=
export AZURE_CLIENT_SECRET=
export AZURE_TENANT_ID=
export AZURE_SUBSCRIPTION_ID=
```

# Usage
Login to Azure and select subscription
```
az login --service-principal -u $AZURE_CLIENT_ID -p AZURE_CLIENT_SECRET --tenant AZURE_TENANT_ID
az account set --subscription AZURE_SUBSCRIPTION_ID
```

Setting up a Storage Account for the terraform state backend using terraform-init
```
# Change directory to
cd terraform-init
terraform init
terraform plan
terraform apply
```
Mark down the output containing Azure Storage Account details and create `config.azurerm.tfbackend` from `config.azurerm.tfbackend.tempalte`

Deploying network and runner modules
```
# Change into main terraform directory
cd terraform
# Source helper functions
source scripts/helpers.sh
# Retrieve Storage Account Access key
export RESOURCE_GROUP_NAME=$(extract_value "resource_group_name" config.azurerm.tfbackend)
export STORAGE_ACCOUNT_NAME=$(extract_value "storage_account_name" config.azurerm.tfbackend)
export ARM_ACCESS_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv)
# Decide on a terraform workspace
terraform workspace list
terraform workspace new # Create a terraform workspace, can be used for staging
terraform workspace select # Or select an existing workspace
# Initialize terraform with Azure Storage Account backend
terraform init --backend-config=config.azurerm.tfbackend
# Plan, review and apply
terraform plan -out main.tfplan
terraform apply main.tfplan
```

The SSH key associated with the VM will be saved in the current working directory as `private_key.pem`
205 changes: 140 additions & 65 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,65 +1,140 @@
# wp10-image-factory
Frame Contract WP10 Image Factory

# Contents
- `terraform-init` - terraform code for setting up an Azure Storage Account that can be used for saving the terraform state
- `terraform` - terraform modules for deploying the resource group, virtual network, subnet, VM

# Requirements
- Access to Azure Resource Manager
- `terraform >=1.0.0`, tested with `1.9.5`
- Azure CLI, tested with `2.64.0`
- Azure credentials saved as environment variables:
```
export AZURE_CLIENT_ID=
export AZURE_CLIENT_SECRET=
export AZURE_TENANT_ID=
export AZURE_SUBSCRIPTION_ID=
```

# Usage
Login to Azure and select subscription
```
az login --service-principal -u $AZURE_CLIENT_ID -p AZURE_CLIENT_SECRET --tenant AZURE_TENANT_ID
az account set --subscription AZURE_SUBSCRIPTION_ID
```

Setting up a Storage Account for the terraform state backend using terraform-init
```
# Change directory to
cd terraform-init
terraform init
terraform plan
terraform apply
```
Mark down the output containing Azure Storage Account details and create `config.azurerm.tfbackend` from `config.azurerm.tfbackend.tempalte`

Deploying network and runner modules
```
# Change into main terraform directory
cd terraform
# Source helper functions
source scripts/helpers.sh
# Retrieve Storage Account Access key
export RESOURCE_GROUP_NAME=$(extract_value "resource_group_name" config.azurerm.tfbackend)
export STORAGE_ACCOUNT_NAME=$(extract_value "storage_account_name" config.azurerm.tfbackend)
export ARM_ACCESS_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv)
# Decide on a terraform workspace
terraform workspace list
terraform workspace new # Create a terraform workspace, can be used for staging
terraform workspace select # Or select an existing workspace
# Initialize terraform with Azure Storage Account backend
terraform init --backend-config=config.azurerm.tfbackend
# Plan, review and apply
terraform plan -out main.tfplan
terraform apply main.tfplan
```

The SSH key associated with the VM will be saved in the current working directory as `private_key.pem`
# WP10 Image Factory

The WP10 Image Factory is a self-service system designed to facilitate the creation of VM and container images. This repository serves as the primary resource for deploying the underlying infrastructure through GitHub Actions workflows that automate the deployment and configuration processes.

Additionally, it provides reusable GitHub Actions workflows that users can invoke from their own repositories for image-building purposes.

## Table of Contents
- [WP10 Image Factory](#wp10-image-factory)
- [Table of Contents](#table-of-contents)
- [Repository Structure](#repository-structure)
- [Prerequisites](#prerequisites)
- [Repository Secrets and variables](#repository-secrets-and-variables)
- [Building Platform Images](#building-platform-images)
- [Building Runner Host Image](#building-runner-host-image)
- [Building Runner Container Image](#building-runner-container-image)
- [Platform Deployment](#platform-deployment)
- [Executing the Workflow](#executing-the-workflow)
- [Platform Configuration](#platform-configuration)
- [Executing the Workflow](#executing-the-workflow-1)
- [Image Factory Reusable Workflows](#image-factory-reusable-workflows)
- [Reusable Docker Image Workflow](#reusable-docker-image-workflow)
- [Reusable Packer Image Workflow](#reusable-packer-image-workflow)
- [Example User Repository](#example-user-repository)

## Repository Structure

- [`ansible/`](./ansible/): Contains Ansible configuration and playbooks for setting up the VM.
- [`docs/`](./docs/): Documentation files related to the project.
- [`images/`](./images/): Holds the Docker and Packer image configurations.
- [`docker/`](./images/docker/): Custom Docker image for GitHub runners.
- [`packer/`](./images/packer/): Packer template for building VM images.
- [`terraform/`](./terraform/): Terraform scripts for deploying various Azure components (resource group, vnet, subnet, runner host vm etc.)
- [`modules/`](./terraform/modules/): Reusable Terraform modules for Azure resources.
- [`scripts/`](./terraform/scripts/): Helper scripts for automation.
- [`terraform-init/`](./terraform-init/): Terraform scripts for setting up an Azure Storage Account used as Terraform state storage

<br />

## Prerequisites

To use this repository, the following prerequisites are needed:

- An Azure subscription
- An Azure service principal with credentials (Client ID, Client Secret, and Tenant ID)
- GitHub Actions secrets and variables configured in the repo

### Repository Secrets and variables

The following GitHub secrets need to be configured in the repository to run GitHub Actions workflows:

- `ACR_PASSWORD`: Password for Azure Container Registry.
- `ACR_USERNAME`: Username for Azure Container Registry.
- `AZURE_CLIENT_ID`: Client ID of the Azure service principal.
- `AZURE_CLIENT_SECRET`: Client Secret of the Azure service principal.
- `AZURE_SUBSCRIPTION_ID`: Subscription ID for the Azure account.
- `AZURE_TENANT_ID`: Tenant ID of the Azure service principal.
- `AZURE_VM_SSH_KEY`: SSH key for Azure VM access.
- `AZURE_IF_RUNNER_IP`: IP address of the image factory runner.
- `JUMPHOST_IP`: IP address of the Azure jump host.
- `RUNNER_HOST_IP`: IP address for the VM hosting the Image Factory runners in the air-gapped environment.
- `RUNNER_HOST_IP_DEVEL`: IP address for the VM hosting the Image Factory runners in the development environment (public subnet).
- `SSH_KEY`: SSH key for accessing the runner host VM in the air-gapped environment.
- `SSH_KEY_DEVEL`: SSH key for accessing the runner host VM in the development environment.
- `GH_PAT_TOKEN`: Personal Access Token for GitHub used for Image Factory runner registration.
- `REGISTRY`: Azure Container Registry registry URL.

The following GitHub repository variables need to be configured in the repository to run GitHub Actions workflows:

- `ACR_RUNNER_IMAGE_NAME`: Name of the runner image in Azure Container Registry.
- `AZURE_ACG`: Azure Compute Gallery name.
- `AZURE_LOCATION`: Azure region where resources will be deployed.
- `AZURE_RESOURCE_GROUP`: Resource group in Azure for the deployment.
- `GH_OWNER`: GitHub owner (user or organization) where Image Factory runners will be connected to.
- `GH_REPO`: GitHub repository name where Image Factory runners will be connected to.
- `VM_ENV`: Environment setting for the runner host VM. Valid values are `"development"` and `"air-gapped"`.

<br />

## Building Platform Images

The following images are used in the main platform deployment:
- Runner Host Image (virtual machine)
- Runner (container)

Platform Images are built using GitHub Actions workflows.

### Building Runner Host Image

The [`packer-build-if-runner.yml`](.github/workflows/packer-build-if-vm.yml) workflow is used to build the runner host VM image with Packer and push it to the Azure Compute Gallery (ACG).
Packer image and code used in this GitHub action can be found in the [`images/packer`](./images/packer) folder.

### Building Runner Container Image

The [`docker-build-if-runner.yml`](.github/workflows/docker-build-if-runner.yml) workflow is used to build the runner container image with Docker and push it to the Azure Container Registry (ACR).
Docker image and all resources needed in the Docker build action can be found in the [`images/docker`](./images/docker) folder.

<br />

## Platform Deployment

The platform deployment is managed through a `.github/workflows/terraform-deploy.yml` GitHub Actions workflow that leverages Terraform to provision and configure the necessary infrastructure on Azure.

Detailed information about the Terraform configurations and scripts can be found in the following locations:
- Terraform initialization scripts: [`terraform-init/`](./terraform-init/)
- Main Terraform scripts: [`terraform/`](./terraform/)

### Executing the Workflow

To deploy the platform, ensure the necessary GitHub secrets and variables are configured as described in the Prerequisites section. The workflow requires manual execution and user inputs, which can be provided in the GitHub Actions tab.

<br />

## Platform Configuration

Further platform configuration is done in `.github/workflows/ansible-configure-vm.yml` GitHub Actions workflow. This workflow uses Ansible to configure the deployed runner host VM.

For more detailed information about the Ansible playbooks and configurations, refer to the following locations:
- Ansible playbooks and configurations: [`ansible/`](./ansible/)

### Executing the Workflow

To configure the platform, ensure the necessary GitHub secrets and variables are configured as described in the Prerequisites section. The workflow can be found in the GitHub Actions tab and can be executed manually once infractructure is deployed with terraform.

<br />

## Image Factory Reusable Workflows

This repository provides two reusable GitHub Actions workflows that customers can call from their repositories to build Docker and Packer images.

### Reusable Docker Image Workflow

The [`reusable-workflow-docker-image.yml`](./.github/workflows/reusable-workflow-docker-image.yml) workflow is used to build Docker images. Customers can integrate this workflow into their repositories to automate the Docker image-building process.

### Reusable Packer Image Workflow

The [`reusable-workflow-packer-image.yml`](./.github/workflows/reusable-workflow-packer-image.yml) workflow is used to build Packer images. Customers can use this workflow in their repositories to automate the creation of VM images with Packer.

### Example User Repository

An example user repository that demonstrates how to call these reusable workflows can be found [here](https://github.com/comsysto/wp10-image-factory-user).
23 changes: 23 additions & 0 deletions ansible/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Ansible

This directory contains Ansible code for setting up the runner host VM for the WP10 Image Factory.

## Ansible Playbook

The main Ansible playbook [`configure-image-factory-vm.yml`](./configure-image-factory-vm.yml) is responsible for calling the [`podman`](./roles/podman) role to configure the runner host VM.

## Podman Role

The [`podman`](./roles/podman) role configures Podman on the runner host VM. It sets up necessary directories, templates configuration files and GitHub PAT, logs into the Azure Container Registry, pulls the runner image, and creates the runner container. It also generates a systemd unit file for the runner container, ensuring it is managed by systemd, which enables automatic restarts and ensures that the container starts on system boot.

## GitHub Workflow

The Ansible playbook is executed through the GitHub Actions workflow [`ansible-configure-if-vm.yml`](../.github/workflows/ansible-configure-if-vm.yml). This workflow handles both development and air-gapped environments, and templates the necessary SSH keys and inventory files accordingly.

### Workflow Steps

1. Checkout Repository
2. Template SSH Key and Inventory (based on the environment)
3. Template Ansible Variables
4. Install and Configure Ansible
5. Run Ansible Playbook
38 changes: 38 additions & 0 deletions images/docker/image-factory-runner/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# GitHub Actions Runner Docker Image

This directory contains the Dockerfile and supporting scripts to build a Docker image for the container that will be provided to Image Factory users as a GitHub Actions runner. The image is based on Ubuntu 22.04 and includes all necessary dependencies and configurations to run GitHub Actions workflows.

## Dockerfile Overview

The Dockerfile performs the following tasks:

1. **Base Image**: Starts with an Ubuntu 22.04 base.
2. **Install Dependencies**: Updates the package list, upgrades the system, and installs required packages like `curl`, `git`, `podman`, `qemu`, and more.
3. **Create User**: Creates a `gha` user with a home directory at `/opt/gha`.
4. **Setup Podman**: Configures rootless `podman` for the `gha` user.
5. **Download GitHub Actions Runner**: Downloads and extracts the GitHub Actions runner binaries.
6. **Install Additional Tools**: Installs tools such as Packer, Azure CLI, and Trivy.
7. **Set Permissions and Entrypoint**: Sets necessary file permissions and specifies the start script as the entry point.

### Entrypoint

The [`start-github-runner.sh`](./scripts/start-github-runner.sh) script performs the following actions:

1. Reads the GitHub Personal Access Token (PAT) from the `/pat/.token` file.
2. Logs into the specified container registry using the provided credentials.
3. Configures Podman to use the container registry.
4. Requests a runner registration token from GitHub.
5. Deletes the PAT token from the runner filesystem.
6. Sets up QEMU for emulating different CPU architectures.
7. Configures the runner with GitHub using the registration token.
8. Starts the runner in an unattended, ephemeral, and replaceable mode.

## GitHub Actions Workflow

The GitHub Actions workflow [`.github/workflows/docker-build-if-runner.yml`](../../.github/workflows/docker-build-if-runner.yml) is triggered on pushes to this directory or manual dispatch and performs the following steps:

1. **Checkout Repository**: Checks out the repository to the runner.
2. **Log in to Azure Container Registry**: Logs in to the Azure Container Registry using the provided credentials.
3. **Build Docker Image**: Builds the Docker image using the Dockerfile in this directory.
4. **Push Docker Image**: Pushes the built image to the Azure Container Registry.
5. **Tag and Push as Latest**: Optionally tags the image as `latest` and pushes it.

0 comments on commit 5d7d2d4

Please sign in to comment.