This project generates self-hosted build agents based on the official Microsoft-hosted build agents images, in an Aure DevOps Pipeline. The resulting Azure Managed Image will be associated to the existing Virtual Machine Scale Set so that new VM's will be using the newly generated image. This Virtual Machine Scale Set is managed by Azure DevOps as a Azure Virtual Machine Scale Set Agent.
Currently supports Windows Server 2019 and Ubuntu 2004 images.
- buildagent-generation.yml
- Checkout the latest
mainbranch from actions/virtual-environments - Build the VM with Packer
- Clean up remaining temporary Azure resources
- Turn VM disk into Azure Managed Image
- Update Virtual Machine Scale Set with new Managed Image
- Checkout the latest
- managedimage-cleanup.yml
- Remove unused Azure Managed Images
The pipeline requires Azure resources for the temporary building of the VM image, Azure resources for running the resulting Agent Pool, and some configuration in Azure DevOps.
The Azure resources are created with the Azure PowerShell Module
- Connect to Azure
Connect-AzAccount -UseDeviceAuthentication
- Create resource group that will store the Packer temporary resources
New-AzResourceGroup -Name "DevOps-PackerResources" -Location "West Europe"
- Create an Azure Storage Account to store the generated VHD
New-AzStorageAccount -ResourceGroupName "DevOps-PackerResources" -AccountName "devopspacker" -Location "West Europe" -SkuName "Standard_LRS"
- Create Azure AD Service Principal, output client secret and client id
$sp = New-AzADServicePrincipal -DisplayName "DevOps-Packer"
$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($sp.Secret)
$plainPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)
$plainPassword
$sp.ApplicationId
- Make the Service Principal a Contributor on the subscription
New-AzRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $sp.ApplicationId
- Make the Service Principal a Storage Blob Data Contributor on the subscription
New-AzRoleAssignment -RoleDefinitionName "Storage Blob Data Contributor" -ServicePrincipalName $sp.ApplicationId
Generate a GitHub Personal Access Token that allows Packer to download Packages from GitHub.
- Go to github.com and log in
- Click on your profile image to fold out the menu, and select
Settings - On the Settings page, select
Developer settingsfrom the menu - On the Developer settings page, select
Personal access tokensfrom the menu - Click
Generateto create a new token. - Give it a name and make sure you select at least
read:packagesunderneathscopes
Keep the generated token safe, for usage later in this guide.
To use an Azure Virtual Machine Scale Set as an Azure DevOps Scale Set Agent it has to adhere to a certain set of requirements. The documentation contains all the required information, but at the time of writing the following things were important:
- VM size: at least Standard_D4s_v4
- Overprovisioning: no, Azure DevOps will decide whether or not new VM's (and thus Agents) need to be provisioned
- Upgrade policy: manual
The Virtual Machine Scale Set from the previous step needs to be registered as an Agent Pool in Azure DevOps. The instructions are very clear:
- Add an Agent Pool of type "Azure virtual machine scale set"
- Use a service connection to select the scale set from the previous step (only supported via
Secretauthentication, notCertificateorManaged Identityauthentication) - Give the agent pool a name
- Enter the required configuration values
Create a Variable Group in the Azure DevOps project running the pipeline, and give it a name. It needs to contain the following variables with their appropriate value:
| Variable | Description |
|---|---|
| AZURE_AGENTS_RESOURCE_GROUP | Resource Group that contains the Virtual Machine Scale Sets to be used as Scale Set Agents in Azure DevOps |
| AZURE_LOCATION | Azure location where Packer will create the temporary resources |
| AZURE_RESOURCE_GROUP | Resource group containing the Azure Storage Account that will be used by Packer. The resulting Azure Managed Image will also be put in this Resource Group |
| AZURE_STORAGE_ACCOUNT | Storage Account that Packer will use to store the temporary OSDisk and the resulting sysprepped .vhd |
| AZURE_SUBSCRIPTION | Subscription ID of the Azure Subscription that is used to host the temporary resources. |
| AZURE_TENANT | Tenant ID of the Azure tenant that has the Azure Resource Groups and Subscription. |
| CLIENT_ID | Id of the Azure AD application that has appriopriate permissions on the Subscription to create temporary resources and finalizing the Scale Set configuration. See output from scripts above. |
| CLIENT_SECRET | Application secret to be used fot the connection in combination with the Client Id. See output from scripts above. |
| GITHUB_TOKEN | A Personal access token for GitHub to fetch packages, having at least the read:packages scope. |
| VMSS_Windows2019 | Name of the Azure Virtual Machine Scale Set that will run Build Agents on Windows Server 2019 |
| VMSS_Ubuntu2004 | Name of the Azure Virtual Machine Scale Set that will run Build Agents on Ubuntu 20.04 |
- Build Agent Image: which image to build, choice between
Windows Server 2019andUbuntu 20.04 - Variable Group: name of the Variable Group containing the variables necessary for execution
- Agent Pool: the Agent Pool to use for running the pipeline
- Variable Group: name of the Variable Group containing the variables necessary for execution
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Important to know: while building the image, Packer will spin up a VM in Azure to run the installation instructions, sys-prep that image after completion and cleanup all the temporary resources.
Azure virtual machine scale set agents are a form of self-hosted agents that can be autoscaled to meet demands. This elasticity reduces the need to run dedicated agents all the time.
Generating the images takes a long time, so don't be surprised. A Windows Server 2019 image takes about 6 to 7h's to generate, a Ubuntu 20.04 image takes about 4h's.
Generating the image through Packer takes longer than 1h (see previous bullet point), and thus can't be run on the free tier of the Microsoft Hosted Agents (limited to 1h runs). It can only be run on the paid tier of the Microsoft Hosted Agent, or it needs an existing self-hosted agent to run.
In this project, the initial run of the project is done on a paid Microsoft Hosted Agent and then switched over to the newly generated self-hosted agent scale set. At some point this worked, but currently Microsoft is more strictly enforcing the 6 hour runtime limit on Microsoft Hosted Agents. This can be worked around by first creating a self hosted Linux build agent using this process and then using the freshly generated agent to create a Windows image, or be setting up a basic self hosted agent first and use that to generate the full blown build agent.
This might be resolved in the near future when changes are made to the images regarding .NET runtime installation, which should significantly reduce the build time.
See documentation for YAML-based pipelines and Classic pipelines