From 4101425bc301771416630226f9a1cb402d5d614b Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Mon, 17 Oct 2022 19:00:23 +0200 Subject: [PATCH 01/35] Document oidc-agent, fedcloudclient and terraform --- .../_index.md | 336 ++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md new file mode 100644 index 0000000000..b55a0699b3 --- /dev/null +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -0,0 +1,336 @@ +--- +title: "Using Terraform with oidc-agent and fedcloudclient" +type: docs +weight: 10 +description: > + Step by step guide to Terraform with oidc-agent and fedcloudclient +--- + +## Overview + +This tutorial describes how to create a Virtual Machine in the EGI Federation, +leveraging [oidc-agent](https://indigo-dc.gitbook.io/oidc-agent/) to retrieve +ODIC tokens from [EGI Check-in](../../aai/check-in), +[fedcloudclient](https://fedcloudclient.fedcloud.eu/) to simplify interacting +with the [EGI Cloud Compute service](../../compute/cloud-compute), +[terraform](https://www.terraform.io) and [Ansible](https://www.ansible.com/) to +simplify deploying an infrastructure. +[EGI Dynamic DNS](../../compute/cloud-compute/dynamic-dns) is also used to +assign a domain name to the virtual machine. + +## Step 1: Signing up + +Create an EGI account with [Check-in](../../aai/check-in/signup). + +## Step 2: Enrolling to a Virtual Organisation + +Once your EGI account is ready you need to join a +[Virtual Organisation (VO)](https://confluence.egi.eu/display/EGIG/Virtual+organisation). +Here are the steps to +[join a VO](../../aai/check-in/joining-virtual-organisation/). Explore the list +of available VOs in the +[Operations Portal](https://operations-portal.egi.eu/vo/a/list). We have a +dedicated VO called +[vo.access.egi.eu](https://operations-portal.egi.eu/vo/view/voname/vo.access.egi.eu) +for piloting purposes. If you are not sure about which VO to enrol to, please +request access to the _vo.access.egi.eu_ VO with your EGI account by visiting +the [enrolment URL](https://aai.egi.eu/registry/co_petitions/start/coef:240). +Check [AppDB](https://appdb.egi.eu/store/vo/vo.access.egi.eu) to see the list of +Virtual Appliances and Resource Providers participating in the +_vo.access.egi.eu_ VO. AppDB is one of the service in the +[EGI Architecture](../../getting-started/architecture/). + +> This tutorial will assume you are using `vo.access.egi.eu`, adapt as required +> for your specific environment. + +## Step 3: Creating a VM + +Once your membership to a VO has been approved you are ready to create your +first Virtual Machine. + +The OpenID Connect (OIDC) protocol is used to authenticate users and authorise +access to [Cloud Compute](../../compute/cloud-compute/) resources that are +integrated with [EGI Check-in](../../aai/check-in/). + +While it's not mandatory, a convenient way to manage the OIDC token is to use +[oidc-agent](#setting-up-oidc-agent). + +### Setting up oidc-agent + +> [oidc-agent](https://indigo-dc.gitbook.io/oidc-agent/) is a set of tools to +> manage OpenID Connect tokens and make them easily usable from the command +> line. + +Install `oidc-agent` according to +[official documentation](https://indigo-dc.gitbook.io/oidc-agent/installation), +once `oidc-agent` is installed it can be used to retrieve an OIDC access token +from EGI Check-in. + +```shell +# Generate configuration for EGI Check-in +oidc-gen --pub --issuer https://aai.egi.eu/auth/realms/egi \ + --scope "email \ + eduperson_entitlement \ + eduperson_scoped_affiliation \ + eduperson_unique_id" egi +# List existing configuration +oidc-add -l +# Request an OIDC access token +oidc-token egi +# Setting a variable for an access token to be used with OpenStack +# XXX access tokens are short lived, relaunch command to obtain a new token +export OS_ACCESS_TOKEN=`oidc-token egi` +``` + +It's possible to automatically start `oidc-agent` in your shell initialisation, +example that can be added to `~/.bash_profile` or `~/.zshrc`: + +```bash +if command -v oidc-agent-service &> /dev/null + eval `oidc-agent-service use` + # for fedcloudclient, once egi account got created + export OIDC_AGENT_ACCOUNT=egi +fi +``` + +See [full documentation](https://indigo-dc.gitbook.io/oidc-agent/). + +### Identifying a suitable cloud site + +It's possible to deploy an OpenStack Virtual Machine (VM) on any of the sites +supporting the Virtual Organisations (VO) you are a member of. + +[`fedcloudlcient`](https://fedcloudclient.fedcloud.eu/) is an high-level Python +package for a command-line client designed for interaction with the OpenStack +services in the EGI infrastructure. The client can access various EGI services +and can perform many tasks for users including managing access tokens, listing +services, and mainly execute commands on OpenStack sites in EGI infrastructure. + +`fedcloudclient` can leverage [`oidc-agent`](#setting-up-oidc-agent) if it's +installed and properly configured. + +`fedcloudclient` and +[`opentackclient`](https://docs.openstack.org/python-openstackclient/latest/) +will be used to interact with the EGI Cloud Compute service. Both of them can be +installed in a +[python virtualenv](https://docs.python.org/3/tutorial/venv.html): + +```shell +# Creating a python 3 virutal env +python3 -m venv ~/.virtualenvs/fedcloud +# Activating the virutal env +source ~/.virtualenvs/fedcloud +# Installing required packages +pip install -U openstackclient +pip install -U fedcloudclient +``` + +```shell +# Listing the VO membership related to your OIDC access token +fedcloud token list-vos +``` + +In order to look for sites supporting a particular VO, you can use the +[EGI Application Database](https://appdb.egi.eu/browse/vos/cloud). + +`vo.access.egi.eu` is a VO for piloting activities, you can enrol via +[EGI Check-in](https://aai.egi.eu/registry/co_petitions/start/coef:240). + +You can retrieve information from the AppDB about the sites supporting the +[vo.access.egi.eu VO](https://appdb.egi.eu/store/vo/vo.access.egi.eu). + +### Deploying the Virtual Machine + +In the following example, the `IN2P3-IRES` site supporting the +`vo.access.egi.eu` VO will be used. + +#### Creating the VM manually + +```shell +# Setting environement up +export EGI_SITE='IN2P3-IRES' +export EGI_VO='vo.access.egi.eu' +# Export variables for OpenStack access +eval `fedcloud site env` +# Selecting an image +fedcloud select image --image-specs "Name =~ 'EGI.*22'" +export IMAGE_ID="..." +# Selecting a flavor +fedcloud select flavor --flavor-specs "RAM>=2096" --flavor-specs "Disk > 10" --vcpus 2 +export FLAVOR_ID="..." +# Selecting network +fedcloud select network --network-specs default +export NETWORK_ID="..." +# Registering an ssh key +openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey +# Identifying and configuring security groups +openstack security group list +openstack security group rule list default +openstack security group rule list http +# Creating the server (the Virtual Machine) +openstack server create --flavor $FLAVOR_ID \ + --image $IMAGE_ID \ + --nic net-id=$NETWORK_ID \ + --security-group default --security-group http \ + --key-name mykey vm1-scoreboard +export SERVER_ID="..." +# Listing VMS +fedcloud openstack --site $EGI_SITE server list +# List the created VM +fedcloud openstack --site $EGI_SITE server show $SERVER_ID +# Listing network and find external one +openstack network list +# Creating a floating IP on external/public network (ext-net for this site) +openstack floating ip create ext-net +export FLOATING_IP='XXX.XXX.XXX.XXX' +# Assigning the public floating IP to the VM +openstack server add floating ip $SERVER_ID $FLOATING_IP +# Accessing the VM +ssh ubuntu@$FLOATING_IP +``` + +##### Using the EGI Dynamic DNS service + +Once connected to the VM, it's possible to use the +[EGI Dynamic DNS service](https://docs.egi.eu/users/compute/cloud-compute/dynamic-dns/) +to get a registered domain name, that can also be used for getting a +[Let's Encrypt certificate](https://letsencrypt.org/). + +```shell +# Accessing the VM using its IP Address +ssh ubuntu@$FLOATING_IP +sudo apt update && sudo apt upgrade +# Login to NSupdate service with EGI +# https://docs.egi.eu/users/compute/cloud-compute/dynamic-dns/ +https://nsupdate.fedcloud.eu/ +# Register a new VM, and get URL with secret +# From the VM +curl "https://:@nsupdate.fedcloud.eu/nic/update" +# Accessing the VM using its domain name +ssh ubuntu@ +``` + +#### Creating the VM with terraform + +Instead of creating the server manually, it is possible to use +[terraform with EGI Cloud Compute](../../compute/cloud-compute/openstack/#terraform). + +The +[Terraform OpenStack provider](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs) +provides official documentation. + +Terraform provides +[installation instructions](https://www.terraform.io/downloads) for all usual +platforms. + +Once terraform is installed locally, you can make use of it. + +Setting up the environment (OS\_\* variables will be used by terraform): + +```shell +source ~/.virtualenvs/fedcloudclient/bin/activate +export EGI_VO='vo.access.egi.eu' +export EGI_SITE='IN2P3-IRES' +eval `fedcloud site env` +# Need an OS_TOKEN for terraform +# XXX this breaks using openstackclient +# rely on fedcloudclient or unset OS_TOKEN before using openstackclient +export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" --vo "$EGI_VO" -j | jq -r '.[0].Result.id') +``` + +Configure flavor, image, network variables for the site you want to use, see +example of [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars): + +```terraform +# Internal network +internal_net_id = "7ae7b0ca-f122-4445-836a-5fb7af524dcb" + +# Public IP pool for floating IPs +public_ip_pool = "ext-net" + +# Flavor: m1.medium +flavor_id = "ab1fbd4c-324d-4155-bd0f-72f077f0ebce" + +# Image: CentOS 7 with docker +image_id = "8901c279-8aee-4b5b-be03-d4460cfe4008" + +# Security groups +security_groups = ["default"] +``` + +The initial configuration of the VM is done using a `cloud-init.yaml` file. +Replace ``, ``, `` by the +proper values. + +```yaml +--- +# cloud-config +runcmd: + - [ + curl, + "https://:@nsupdate.fedcloud.eu/nic/update", + ] + +users: + - name: egi + gecos: EGI + primary_group: egi + groups: users + shell: /bin/bash + sudo: ALL=(ALL) NOPASSWD:ALL + ssh_import_id: + - gh: + +packages: + - vim + +package_update: true +package_upgrade: true +package_reboot_if_required: true +``` + +```shell +# Initialise working directory, install dependencies +terraform init +# Review plan of actions for creating the infrastructure +# Use relevant site-specific config file +terraform plan --var-file="${EGI_SITE}.tfvars" +# Create the infrastructure +# Manual approval can be skipped using -auto-approve +# The SERVER_ID will be printed (openstack_compute_instance_v2.scoreboard) +terraform apply --var-file="${EGI_SITE}.tfvars" +# Wait a few minutes for the setup to be finalised and connect to the server +ssh egi@$NSUPATE_HOSTNAME +``` + +##### Debugging terraform + +The token used by Terraform for accessing OpenStack is short lived, it will have +to be renewed from time to time. + +```shell +# Creating a new token to access the OpenStack endpoint +export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" --vo "$EGI_VO" -j | jq -r '.[0].Result.id') +``` + +It is possible to print a verbose/debug output to get details on interactions +with the OpenStack endpoint. + +```shell +# Debugging +OS_DEBUG=1 TF_LOG=DEBUG terraform apply -auto-approve --var-file="${EGI_SITE}.tfvars" +``` + +##### Destroying the resources created by terraform + +```shell +# Debugging +# Destroying the created infrastructure +terraform destroy --var-file="${EGI_SITE}.tfvars" +``` + +#### Integrating Terraform with Ansible + +## Asking for help + +If you find issues please do not hesitate to [contact us](../../../support/). From f350b5fdcbbce065d6c662de00fc032e085f7c71 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 14:11:32 +0200 Subject: [PATCH 02/35] Use a requirements.txt file --- .../oidc-agent-fedcloudclient-terraform/_index.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index b55a0699b3..51dc0f4177 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -115,14 +115,21 @@ will be used to interact with the EGI Cloud Compute service. Both of them can be installed in a [python virtualenv](https://docs.python.org/3/tutorial/venv.html): +Document required python dependencies in a `requirements.txt` file: + +```requirements +openstackclient +fedcloudclient +ansible +``` + ```shell # Creating a python 3 virutal env python3 -m venv ~/.virtualenvs/fedcloud # Activating the virutal env source ~/.virtualenvs/fedcloud -# Installing required packages -pip install -U openstackclient -pip install -U fedcloudclient +# Installing required python packages +pip install -r requirements.txt ``` ```shell From bb8955eb0cbc3b590bbb4446c1f029da8006c6ac Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 14:11:54 +0200 Subject: [PATCH 03/35] Document main.tf for terraform --- .../_index.md | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 51dc0f4177..680279680b 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -296,6 +296,54 @@ package_upgrade: true package_reboot_if_required: true ``` +Create terraform configuration in `main.tf`: + +```terraform +# Terraform versions and providers +terraform { + required_version = ">= 0.14.0" + required_providers { + openstack = { + source = "terraform-provider-openstack/openstack" + version = "~> 1.35.0" + } + } +} + +# Allocate a floating IP from the public IP pool +resource "openstack_networking_floatingip_v2" "egi_vm_floatip_1" { + pool = var.public_ip_pool +} + +# Creating the VM +resource "openstack_compute_instance_v2" "egi_vm" { + name = "egi_test_vm" + image_id = var.image_id + flavor_id = var.flavor_id + security_groups = var.security_groups + user_data = file("cloud-init.yaml") + network { + uuid = var.internal_net_id + } +} + +# Attach the floating public IP to the created instance +resource "openstack_compute_floatingip_associate_v2" "egi_ui_fip_1" { + instance_id = "${openstack_compute_instance_v2.egi_ui.id}" + floating_ip = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" +} + +# Create inventory file for Ansible +resource "local_file" "hosts_cfg" { + content = templatefile("${path.module}/hosts.cfg.tpl", + { + ui = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" + } + ) + filename = "./inventory/hosts.cfg" +} +``` + ```shell # Initialise working directory, install dependencies terraform init From 8ebf88311d29fb0cd6df48861ecebac4494b5fe9 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 14:12:21 +0200 Subject: [PATCH 04/35] Document using ansible to test access --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 680279680b..c1f40cd62f 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -354,7 +354,10 @@ terraform plan --var-file="${EGI_SITE}.tfvars" # Manual approval can be skipped using -auto-approve # The SERVER_ID will be printed (openstack_compute_instance_v2.scoreboard) terraform apply --var-file="${EGI_SITE}.tfvars" -# Wait a few minutes for the setup to be finalised and connect to the server +# Wait a few minutes for the setup to be finalised +# Test if ansible can reach the vm +ansible all -m ping +# Connect to the server using ssh ssh egi@$NSUPATE_HOSTNAME ``` From d55fb9a3448a14ab37e9cf846bb06c8858e04c19 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 14:15:56 +0200 Subject: [PATCH 05/35] Document configuring ansible --- .../oidc-agent-fedcloudclient-terraform/_index.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index c1f40cd62f..a52da0ae4c 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -344,6 +344,16 @@ resource "local_file" "hosts_cfg" { } ``` +Configure Ansible environment in a `ansible.cfg` file: + +```ansible +[defaults] +# Use user created using cloud-init.yml +remote_user = egi +# Use generated inventory file +inventory = ./inventory/hosts.cfg +``` + ```shell # Initialise working directory, install dependencies terraform init From cef42f750b136dab1797ea7ce1895a42ad7ba109 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 15:11:32 +0200 Subject: [PATCH 06/35] Link to VM in appdb, update ssh key configuration --- .../_index.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index a52da0ae4c..e23f2c8016 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -258,16 +258,20 @@ public_ip_pool = "ext-net" # Flavor: m1.medium flavor_id = "ab1fbd4c-324d-4155-bd0f-72f077f0ebce" -# Image: CentOS 7 with docker -image_id = "8901c279-8aee-4b5b-be03-d4460cfe4008" +# Image: EGI CentOS 8 +# https://appdb.egi.eu/store/vappliance/egi.centos.8 +image_id = "38ced5bf-bbfd-434b-ae41-3ab35d929aba" +# Image: EGI Ubuntu 22.04 +# https://appdb.egi.eu/store/vappliance/egi.ubuntu.22.04 +# image_id = "fc6c83a3-845f-4f29-b44d-2584f0ca4177" # Security groups security_groups = ["default"] ``` The initial configuration of the VM is done using a `cloud-init.yaml` file. -Replace ``, ``, `` by the -proper values. +Replace ``, ``, `` (the +content of your SSH public key) by the proper values. ```yaml --- @@ -285,8 +289,8 @@ users: groups: users shell: /bin/bash sudo: ALL=(ALL) NOPASSWD:ALL - ssh_import_id: - - gh: + ssh_authorized_keys: + - packages: - vim From a48722517ea20dbefa0d7b787626ed7d648db27a Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 15:40:41 +0200 Subject: [PATCH 07/35] Update snippets, document identifying images/flavors/network/security groups --- .../_index.md | 147 ++++++------------ 1 file changed, 51 insertions(+), 96 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index e23f2c8016..0fee0bde8f 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -16,7 +16,8 @@ with the [EGI Cloud Compute service](../../compute/cloud-compute), [terraform](https://www.terraform.io) and [Ansible](https://www.ansible.com/) to simplify deploying an infrastructure. [EGI Dynamic DNS](../../compute/cloud-compute/dynamic-dns) is also used to -assign a domain name to the virtual machine. +assign a domain name to the virtual machine, which can then be used to get a +valid TLS certificate from [Let's Encrypt](https://letsencrypt.org/). ## Step 1: Signing up @@ -68,18 +69,18 @@ from EGI Check-in. ```shell # Generate configuration for EGI Check-in -oidc-gen --pub --issuer https://aai.egi.eu/auth/realms/egi \ +$ oidc-gen --pub --issuer https://aai.egi.eu/auth/realms/egi \ --scope "email \ eduperson_entitlement \ eduperson_scoped_affiliation \ eduperson_unique_id" egi # List existing configuration -oidc-add -l +$ oidc-add -l # Request an OIDC access token -oidc-token egi +$ oidc-token egi # Setting a variable for an access token to be used with OpenStack # XXX access tokens are short lived, relaunch command to obtain a new token -export OS_ACCESS_TOKEN=`oidc-token egi` +$ export OS_ACCESS_TOKEN=`oidc-token egi` ``` It's possible to automatically start `oidc-agent` in your shell initialisation, @@ -125,16 +126,16 @@ ansible ```shell # Creating a python 3 virutal env -python3 -m venv ~/.virtualenvs/fedcloud +$ python3 -m venv ~/.virtualenvs/fedcloud # Activating the virutal env -source ~/.virtualenvs/fedcloud +$ source ~/.virtualenvs/fedcloud # Installing required python packages -pip install -r requirements.txt +$ pip install -r requirements.txt ``` ```shell # Listing the VO membership related to your OIDC access token -fedcloud token list-vos +$ fedcloud token list-vos ``` In order to look for sites supporting a particular VO, you can use the @@ -151,72 +152,6 @@ You can retrieve information from the AppDB about the sites supporting the In the following example, the `IN2P3-IRES` site supporting the `vo.access.egi.eu` VO will be used. -#### Creating the VM manually - -```shell -# Setting environement up -export EGI_SITE='IN2P3-IRES' -export EGI_VO='vo.access.egi.eu' -# Export variables for OpenStack access -eval `fedcloud site env` -# Selecting an image -fedcloud select image --image-specs "Name =~ 'EGI.*22'" -export IMAGE_ID="..." -# Selecting a flavor -fedcloud select flavor --flavor-specs "RAM>=2096" --flavor-specs "Disk > 10" --vcpus 2 -export FLAVOR_ID="..." -# Selecting network -fedcloud select network --network-specs default -export NETWORK_ID="..." -# Registering an ssh key -openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey -# Identifying and configuring security groups -openstack security group list -openstack security group rule list default -openstack security group rule list http -# Creating the server (the Virtual Machine) -openstack server create --flavor $FLAVOR_ID \ - --image $IMAGE_ID \ - --nic net-id=$NETWORK_ID \ - --security-group default --security-group http \ - --key-name mykey vm1-scoreboard -export SERVER_ID="..." -# Listing VMS -fedcloud openstack --site $EGI_SITE server list -# List the created VM -fedcloud openstack --site $EGI_SITE server show $SERVER_ID -# Listing network and find external one -openstack network list -# Creating a floating IP on external/public network (ext-net for this site) -openstack floating ip create ext-net -export FLOATING_IP='XXX.XXX.XXX.XXX' -# Assigning the public floating IP to the VM -openstack server add floating ip $SERVER_ID $FLOATING_IP -# Accessing the VM -ssh ubuntu@$FLOATING_IP -``` - -##### Using the EGI Dynamic DNS service - -Once connected to the VM, it's possible to use the -[EGI Dynamic DNS service](https://docs.egi.eu/users/compute/cloud-compute/dynamic-dns/) -to get a registered domain name, that can also be used for getting a -[Let's Encrypt certificate](https://letsencrypt.org/). - -```shell -# Accessing the VM using its IP Address -ssh ubuntu@$FLOATING_IP -sudo apt update && sudo apt upgrade -# Login to NSupdate service with EGI -# https://docs.egi.eu/users/compute/cloud-compute/dynamic-dns/ -https://nsupdate.fedcloud.eu/ -# Register a new VM, and get URL with secret -# From the VM -curl "https://:@nsupdate.fedcloud.eu/nic/update" -# Accessing the VM using its domain name -ssh ubuntu@ -``` - #### Creating the VM with terraform Instead of creating the server manually, it is possible to use @@ -235,18 +170,36 @@ Once terraform is installed locally, you can make use of it. Setting up the environment (OS\_\* variables will be used by terraform): ```shell -source ~/.virtualenvs/fedcloudclient/bin/activate -export EGI_VO='vo.access.egi.eu' -export EGI_SITE='IN2P3-IRES' +# Activate virtualenv +$ source ~/.virtualenvs/fedcloudclient/bin/activate +# Export variable for VO and SITE to avoid having to repeat them +$ export EGI_VO='vo.access.egi.eu' +$ export EGI_SITE='IN2P3-IRES' eval `fedcloud site env` -# Need an OS_TOKEN for terraform -# XXX this breaks using openstackclient -# rely on fedcloudclient or unset OS_TOKEN before using openstackclient -export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" --vo "$EGI_VO" -j | jq -r '.[0].Result.id') +# Get an OS_TOKEN for terraform +# XXX this breaks using openstackclient: use fedcloudclient +# or unset OS_TOKEN before using openstackclient +$ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \ + --vo "$EGI_VO" -j | jq -r '.[0].Result.id') ``` -Configure flavor, image, network variables for the site you want to use, see -example of [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars): +Identify and configure flavor, image, network variables for the site you want to +use, using the information gathered via `fedcloudclient`, see the example +[`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars): + +```shell +# Identifying an image +$ fedcloud select image --image-specs "Name =~ 'EGI.*22'" +# Identikfying a flavor +$ fedcloud select flavor --flavor-specs "RAM>=2096" \ + --flavor-specs "Disk > 10" --vcpus 2 +# Identifying a network +$ fedcloud select network --network-specs default +# Identifying security groups +$ fedcloud openstack security group list +$ fedcloud openstack security group rule list default +$ fedcloud openstack security group rule list http +``` ```terraform # Internal network @@ -348,33 +301,36 @@ resource "local_file" "hosts_cfg" { } ``` -Configure Ansible environment in a `ansible.cfg` file: +Configure a basic Ansible environment in the `ansible.cfg` file: ```ansible [defaults] # Use user created using cloud-init.yml remote_user = egi -# Use generated inventory file +# Use inventory file generated by terraform inventory = ./inventory/hosts.cfg ``` ```shell # Initialise working directory, install dependencies -terraform init +$ terraform init # Review plan of actions for creating the infrastructure # Use relevant site-specific config file -terraform plan --var-file="${EGI_SITE}.tfvars" +$ terraform plan --var-file="${EGI_SITE}.tfvars" # Create the infrastructure # Manual approval can be skipped using -auto-approve # The SERVER_ID will be printed (openstack_compute_instance_v2.scoreboard) -terraform apply --var-file="${EGI_SITE}.tfvars" +$ terraform apply --var-file="${EGI_SITE}.tfvars" # Wait a few minutes for the setup to be finalised # Test if ansible can reach the vm -ansible all -m ping +$ ansible all -m ping # Connect to the server using ssh -ssh egi@$NSUPATE_HOSTNAME +$ ssh egi@$NSUPATE_HOSTNAME ``` +> From here you can extend the `cloud-init.yaml` and/or use Ansible locally to +> configure the remote machine, as well as doing manual work via SSH. + ##### Debugging terraform The token used by Terraform for accessing OpenStack is short lived, it will have @@ -382,7 +338,8 @@ to be renewed from time to time. ```shell # Creating a new token to access the OpenStack endpoint -export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" --vo "$EGI_VO" -j | jq -r '.[0].Result.id') +$ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \ + --vo "$EGI_VO" -j | jq -r '.[0].Result.id') ``` It is possible to print a verbose/debug output to get details on interactions @@ -390,7 +347,7 @@ with the OpenStack endpoint. ```shell # Debugging -OS_DEBUG=1 TF_LOG=DEBUG terraform apply -auto-approve --var-file="${EGI_SITE}.tfvars" +$ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" ``` ##### Destroying the resources created by terraform @@ -398,11 +355,9 @@ OS_DEBUG=1 TF_LOG=DEBUG terraform apply -auto-approve --var-file="${EGI_SITE}.tf ```shell # Debugging # Destroying the created infrastructure -terraform destroy --var-file="${EGI_SITE}.tfvars" +$ terraform destroy --var-file="${EGI_SITE}.tfvars" ``` -#### Integrating Terraform with Ansible - ## Asking for help If you find issues please do not hesitate to [contact us](../../../support/). From 89ac2b7b6f827bb3a386710026caccffe343a679 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 16:03:53 +0200 Subject: [PATCH 08/35] Update commands for listing security groups and configuring network --- .../_index.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 0fee0bde8f..100c7b1f0d 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -184,8 +184,8 @@ $ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \ ``` Identify and configure flavor, image, network variables for the site you want to -use, using the information gathered via `fedcloudclient`, see the example -[`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars): +use, using the information gathered via `fedcloudclient`, they should be +documented in a `$EGI_SITE.tfvars` file, as documented below. ```shell # Identifying an image @@ -195,12 +195,19 @@ $ fedcloud select flavor --flavor-specs "RAM>=2096" \ --flavor-specs "Disk > 10" --vcpus 2 # Identifying a network $ fedcloud select network --network-specs default +$ fedcloud openstack --site "$EGI_SITE" network list # Identifying security groups -$ fedcloud openstack security group list -$ fedcloud openstack security group rule list default -$ fedcloud openstack security group rule list http +$ fedcloud openstack --site "$EGI_SITE" security group list +$ fedcloud openstack --site "$EGI_SITE" security group rule list default +$ fedcloud openstack --site "$EGI_SITE" security group rule list http ``` +> The network configuration can be tricky and is usually dependant on the site. +> For `IN2P3-IRES`, one has to request a floating IP from the public network IP +> pool `ext-net` and assign this floating IP to the created instance. + +See the example [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars): + ```terraform # Internal network internal_net_id = "7ae7b0ca-f122-4445-836a-5fb7af524dcb" From 910fc88d31fa4e939ddd7855766aac788874f301 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 16:12:25 +0200 Subject: [PATCH 09/35] Drop useless example --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 100c7b1f0d..a8ec6ad3fd 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -198,8 +198,8 @@ $ fedcloud select network --network-specs default $ fedcloud openstack --site "$EGI_SITE" network list # Identifying security groups $ fedcloud openstack --site "$EGI_SITE" security group list +# Listing rules from a specific security group $ fedcloud openstack --site "$EGI_SITE" security group rule list default -$ fedcloud openstack --site "$EGI_SITE" security group rule list http ``` > The network configuration can be tricky and is usually dependant on the site. From 2c9961454fbc41e0c92494ed09ef78ad8132caeb Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 17:48:29 +0200 Subject: [PATCH 10/35] Review content --- .../_index.md | 200 +++++++++++++----- 1 file changed, 145 insertions(+), 55 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index a8ec6ad3fd..3f46921223 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -1,9 +1,10 @@ --- -title: "Using Terraform with oidc-agent and fedcloudclient" +title: "Automating with oidc-agent, fedcloudclient, terraform and Ansible" type: docs -weight: 10 +weight: 150 description: > - Step by step guide to Terraform with oidc-agent and fedcloudclient + Step by step guide to automating the deployment using Ansible with Terraform, + oidc-agent and fedcloudclient --- ## Overview @@ -68,17 +69,17 @@ once `oidc-agent` is installed it can be used to retrieve an OIDC access token from EGI Check-in. ```shell -# Generate configuration for EGI Check-in +# Generating configuration for EGI Check-in $ oidc-gen --pub --issuer https://aai.egi.eu/auth/realms/egi \ --scope "email \ eduperson_entitlement \ eduperson_scoped_affiliation \ eduperson_unique_id" egi -# List existing configuration +# Listing existing configuration $ oidc-add -l -# Request an OIDC access token +# Requesting an OIDC access token $ oidc-token egi -# Setting a variable for an access token to be used with OpenStack +# Exporting a variable with a Check-in OIDC access token to be used with OpenStack # XXX access tokens are short lived, relaunch command to obtain a new token $ export OS_ACCESS_TOKEN=`oidc-token egi` ``` @@ -94,14 +95,15 @@ if command -v oidc-agent-service &> /dev/null fi ``` -See [full documentation](https://indigo-dc.gitbook.io/oidc-agent/). +When using `oidc-agent-service`, +[fedcloudclient](#installing-fedcloudclient-and-ansible) will be able to +automatically request a new access token from `oidc-agent`. -### Identifying a suitable cloud site +See [full documentation](https://indigo-dc.gitbook.io/oidc-agent/). -It's possible to deploy an OpenStack Virtual Machine (VM) on any of the sites -supporting the Virtual Organisations (VO) you are a member of. +### Installing fedcloudclient and ansible -[`fedcloudlcient`](https://fedcloudclient.fedcloud.eu/) is an high-level Python +[`fedcloudclient`](https://fedcloudclient.fedcloud.eu/) is an high-level Python package for a command-line client designed for interaction with the OpenStack services in the EGI infrastructure. The client can access various EGI services and can perform many tasks for users including managing access tokens, listing @@ -111,48 +113,62 @@ services, and mainly execute commands on OpenStack sites in EGI infrastructure. installed and properly configured. `fedcloudclient` and -[`opentackclient`](https://docs.openstack.org/python-openstackclient/latest/) -will be used to interact with the EGI Cloud Compute service. Both of them can be -installed in a -[python virtualenv](https://docs.python.org/3/tutorial/venv.html): +[`opentackclient`](https://docs.openstack.org/python-openstackclient/latest/), +the official OpenStack python client, will be used to interact with the EGI +Cloud Compute service. -Document required python dependencies in a `requirements.txt` file: +Required python dependencies are documented in a `requirements.txt` file +(Ansible will be used at a later stage, but is installed at the same time): -```requirements +```requirements.txt openstackclient fedcloudclient ansible ``` +For keeping the main system tidy and isolating the environment, the python +packages will be installed in a dedicated +[python virtualenv](https://docs.python.org/3/tutorial/venv.html): + ```shell -# Creating a python 3 virutal env +# Creating an arbitrary directory where to store python virtual environments +$ mkdir -p ~/.virtualenvs +# Creating a python 3 virtual environment $ python3 -m venv ~/.virtualenvs/fedcloud -# Activating the virutal env +# Activating the virtual environment $ source ~/.virtualenvs/fedcloud -# Installing required python packages +# Installing required python packages in the virtual environment $ pip install -r requirements.txt ``` +### Identifying a suitable cloud site + +It's possible to deploy an OpenStack Virtual Machine (VM) on any of the sites +supporting the Virtual Organisations (VO) you are a member of. + +Once [fedcloudclient](#installing-fedcloudclient-openstackclient-and-ansible) is +installed it's possible to get information about the OIDC token accessed via +[oidc-agent](#setting-up-oidc-agent). + ```shell -# Listing the VO membership related to your OIDC access token +# Verifying that an OS_ACCESS_TOKEN is available +$ echo $OS_ACCESS_TOKEN +# Listing the VO membership related to the OIDC access token $ fedcloud token list-vos ``` In order to look for sites supporting a particular VO, you can use the [EGI Application Database](https://appdb.egi.eu/browse/vos/cloud). -`vo.access.egi.eu` is a VO for piloting activities, you can enrol via -[EGI Check-in](https://aai.egi.eu/registry/co_petitions/start/coef:240). - You can retrieve information from the AppDB about the sites supporting the [vo.access.egi.eu VO](https://appdb.egi.eu/store/vo/vo.access.egi.eu). -### Deploying the Virtual Machine - -In the following example, the `IN2P3-IRES` site supporting the -`vo.access.egi.eu` VO will be used. +> In the following example, the `IN2P3-IRES` site supporting the +> `vo.access.egi.eu` VO will be used, see +> [Step 2: Enrolling to a Virtual Organisation](#step-2-enrolling-to-a-virtual-organisation) +> to request access. -#### Creating the VM with terraform +### Deploying the Virtual Machine with terraform Instead of creating the server manually, it is possible to use [terraform with EGI Cloud Compute](../../compute/cloud-compute/openstack/#terraform). @@ -183,22 +199,21 @@ $ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \ --vo "$EGI_VO" -j | jq -r '.[0].Result.id') ``` -Identify and configure flavor, image, network variables for the site you want to -use, using the information gathered via `fedcloudclient`, they should be -documented in a `$EGI_SITE.tfvars` file, as documented below. +Identify flavor, image, network and security groups for the site you want to +use, using the information gathered with `fedcloudclient`. ```shell -# Identifying an image +# Selecting an image $ fedcloud select image --image-specs "Name =~ 'EGI.*22'" -# Identikfying a flavor +# Selecting a flavor $ fedcloud select flavor --flavor-specs "RAM>=2096" \ --flavor-specs "Disk > 10" --vcpus 2 -# Identifying a network -$ fedcloud select network --network-specs default +# Identifying available networks $ fedcloud openstack --site "$EGI_SITE" network list +$ fedcloud select network --network-specs default # Identifying security groups $ fedcloud openstack --site "$EGI_SITE" security group list -# Listing rules from a specific security group +# Listing rules of a specific security group $ fedcloud openstack --site "$EGI_SITE" security group rule list default ``` @@ -206,7 +221,10 @@ $ fedcloud openstack --site "$EGI_SITE" security group rule list default > For `IN2P3-IRES`, one has to request a floating IP from the public network IP > pool `ext-net` and assign this floating IP to the created instance. -See the example [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars): +The chosen flavor, image, network and security group should be documented in a +`$EGI_SITE.tfvars` file that will be passed as an argument to terraform +commands. See the example [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars), to be +adjusted for the requirements and according to the site: ```terraform # Internal network @@ -230,6 +248,25 @@ security_groups = ["default"] ``` The initial configuration of the VM is done using a `cloud-init.yaml` file. + +This `curl` call in the `cloud-init.yaml` configuration below, will register the +IP of the virtual machine in the DNS zone managed using the +[EGI Dynamic DNS service](https://nsupdate.fedcloud.eu/), allowing to access the +virtual machine using a fully qualified host name and allowing to retrieve a +[Let's Encrypt certificate](https://letsencrypt.org/). + +> Please look at the +> [EGI Dynamic DNS documentation](../../compute/cloud-compute/dynamic-dns/) for +> instructions on creating the configuration for a new host. + +The `users` block in the `cloud-init.yaml` configuration below, will create a +new user with password-less [sudo](https://www.sudo.ws/) access. + +> While this `egi` user can only be accessed via the specified SSH key(s), +> setting a user password and requesting password verification for using sudo +> should be considered, as a compromise of this user account would mean a +> compromise of the complete virtual machine. + Replace ``, ``, `` (the content of your SSH public key) by the proper values. @@ -260,7 +297,43 @@ package_upgrade: true package_reboot_if_required: true ``` -Create terraform configuration in `main.tf`: +The main terraform configuration file is using variables that have to be +described in a `vars.tf` file: + +```terraform +# Terraform variables definition +# Values to be provided in a *.tfvars file passed on the command line + +variable "internal_net_id" { + type = string + description = "The id of the internal network" +} + +variable "public_ip_pool" { + type = string + description = "The name of the public IP address pool" +} + +variable "image_id" { + type = string + description = "VM image id" +} + +variable "flavor_id" { + type = string + description = "VM flavor id" +} + +variable "security_groups" { + type = list(string) + description = "List of security groups" +} + +``` + +To be more reusable, the `main.tf` configuration file is referencing variables +described in a `vars.tf` file created previously, and will take the values from +the `$EGI_SITE.tfvars` file passed as an argument to the terraform command. ```terraform # Terraform versions and providers @@ -308,15 +381,9 @@ resource "local_file" "hosts_cfg" { } ``` -Configure a basic Ansible environment in the `ansible.cfg` file: - -```ansible -[defaults] -# Use user created using cloud-init.yml -remote_user = egi -# Use inventory file generated by terraform -inventory = ./inventory/hosts.cfg -``` +Now that all the files have been created, it's possible to deploy the +infrastructure, currently only a single VM, but it can easily be extended to a +more complex setup, using terraform: ```shell # Initialise working directory, install dependencies @@ -329,16 +396,39 @@ $ terraform plan --var-file="${EGI_SITE}.tfvars" # The SERVER_ID will be printed (openstack_compute_instance_v2.scoreboard) $ terraform apply --var-file="${EGI_SITE}.tfvars" # Wait a few minutes for the setup to be finalised -# Test if ansible can reach the vm -$ ansible all -m ping # Connect to the server using ssh $ ssh egi@$NSUPATE_HOSTNAME ``` -> From here you can extend the `cloud-init.yaml` and/or use Ansible locally to -> configure the remote machine, as well as doing manual work via SSH. +> From here you can extend the `cloud-init.yaml` and/or use +> [Ansible](#testing-ansible-access) to configure the remote machine, as well as +> doing manual work via SSH. + +#### Testing Ansible access + +The [terraform deployment](#deploying-the-virtual-machine-with-terraform) +generated an +[Ansible inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html), +`inventory/hosts.cfg`, that can directly be used by Ansible. + +Configure a basic Ansible environment in the `ansible.cfg` file: + +```ansible +[defaults] +# Use user created using cloud-init.yml +remote_user = egi +# Use inventory file generated by terraform +inventory = ./inventory/hosts.cfg +``` + +Then you can verify that the Virtual Machine is accessible by Ansible: + +```shell +# Test if ansible can reach the vm +$ ansible all -m ping +``` -##### Debugging terraform +#### Debugging terraform The token used by Terraform for accessing OpenStack is short lived, it will have to be renewed from time to time. @@ -357,7 +447,7 @@ with the OpenStack endpoint. $ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" ``` -##### Destroying the resources created by terraform +#### Destroying the resources created by terraform ```shell # Debugging From c12bb391aec3379ce5e12aaa07af74568242bf5b Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 18:08:08 +0200 Subject: [PATCH 11/35] Reorganise sections --- .../_index.md | 265 ++++++++++-------- 1 file changed, 147 insertions(+), 118 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 3f46921223..5acb7d16fd 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -181,24 +181,70 @@ Terraform provides [installation instructions](https://www.terraform.io/downloads) for all usual platforms. -Once terraform is installed locally, you can make use of it. +Once terraform is installed locally, we will create a deployment as documented +in the following sections. -Setting up the environment (OS\_\* variables will be used by terraform): +#### Setting up the environment + +The `OS_*` variables that will be used by terraform can be generated using +`fedcloudclient`. ```shell -# Activate virtualenv +# Activating the virtual environment $ source ~/.virtualenvs/fedcloudclient/bin/activate -# Export variable for VO and SITE to avoid having to repeat them +# Exporting variable for VO and SITE to avoid having to repeat them $ export EGI_VO='vo.access.egi.eu' $ export EGI_SITE='IN2P3-IRES' eval `fedcloud site env` -# Get an OS_TOKEN for terraform +# Obtaining an OS_TOKEN for terraform # XXX this breaks using openstackclient: use fedcloudclient # or unset OS_TOKEN before using openstackclient $ export OS_TOKEN=$(fedcloud openstack token issue --site "$EGI_SITE" \ --vo "$EGI_VO" -j | jq -r '.[0].Result.id') ``` +#### Describing the terraform variables + +The main terraform configuration file, +[main.tf](#creating-the-main-terraform-deployment-file) is using variables that +have to be described in a `vars.tf` file: + +```terraform +# Terraform variables definition +# Values to be provided in a *.tfvars file passed on the command line + +variable "internal_net_id" { + type = string + description = "The id of the internal network" +} + +variable "public_ip_pool" { + type = string + description = "The name of the public IP address pool" +} + +variable "image_id" { + type = string + description = "VM image id" +} + +variable "flavor_id" { + type = string + description = "VM flavor id" +} + +variable "security_groups" { + type = list(string) + description = "List of security groups" +} +``` + +The SITE and VO specific values for those variables will be +[identified](#identifying-the-cloud-resources) and documented in a +[`$EGI_SITE.tfvars` file](#documenting-the-cloud-resources-for-the-selected-site). + +#### Identifying the cloud resources + Identify flavor, image, network and security groups for the site you want to use, using the information gathered with `fedcloudclient`. @@ -217,14 +263,20 @@ $ fedcloud openstack --site "$EGI_SITE" security group list $ fedcloud openstack --site "$EGI_SITE" security group rule list default ``` -> The network configuration can be tricky and is usually dependant on the site. -> For `IN2P3-IRES`, one has to request a floating IP from the public network IP -> pool `ext-net` and assign this floating IP to the created instance. +#### Documenting the cloud resources for the selected site The chosen flavor, image, network and security group should be documented in a `$EGI_SITE.tfvars` file that will be passed as an argument to terraform -commands. See the example [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars), to be -adjusted for the requirements and according to the site: +commands. + +> The network configuration can be tricky and is usually dependant on the site. +> For `IN2P3-IRES`, one has to request a floating IP from the public network IP +> pool `ext-net` and assign this floating IP to the created instance, for +> another site it may not be needed, and in that case the `main.tf` will have to +> be adjusted accordingly. + +See the example [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars), to be adjusted +according to the requirements and to the selected site and VO: ```terraform # Internal network @@ -247,9 +299,68 @@ image_id = "38ced5bf-bbfd-434b-ae41-3ab35d929aba" security_groups = ["default"] ``` +#### Creating the main terraform deployment file + +To be more reusable, the `main.tf` configuration file is referencing variables +described in the [vars.tf](#describing-the-terraform-variables) file created +previously, and will take the values from the +[`$EGI_SITE.tfvars`](#documenting-the-cloud-resources-for-the-selected-site) +file passed as an argument to the terraform command. + +```terraform +# Terraform versions and providers +terraform { + required_version = ">= 0.14.0" + required_providers { + openstack = { + source = "terraform-provider-openstack/openstack" + version = "~> 1.35.0" + } + } +} + +# Allocate a floating IP from the public IP pool +resource "openstack_networking_floatingip_v2" "egi_vm_floatip_1" { + pool = var.public_ip_pool +} + +# Creating the VM +resource "openstack_compute_instance_v2" "egi_vm" { + name = "egi_test_vm" + image_id = var.image_id + flavor_id = var.flavor_id + security_groups = var.security_groups + user_data = file("cloud-init.yaml") + network { + uuid = var.internal_net_id + } +} + +# Attach the floating public IP to the created instance +resource "openstack_compute_floatingip_associate_v2" "egi_ui_fip_1" { + instance_id = "${openstack_compute_instance_v2.egi_ui.id}" + floating_ip = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" +} + +# Create inventory file for Ansible +resource "local_file" "hosts_cfg" { + content = templatefile("${path.module}/hosts.cfg.tpl", + { + ui = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" + } + ) + filename = "./inventory/hosts.cfg" +} +``` + +#### Initial configuration of the VM using cloud-init + +> [cloud-init](https://cloudinit.readthedocs.io/) is the industry standard +> multi-distribution method for cross-platform cloud instance initialization. + The initial configuration of the VM is done using a `cloud-init.yaml` file. -This `curl` call in the `cloud-init.yaml` configuration below, will register the +The `curl` call in the `cloud-init.yaml` configuration below, will register the IP of the virtual machine in the DNS zone managed using the [EGI Dynamic DNS service](https://nsupdate.fedcloud.eu/), allowing to access the virtual machine using a fully qualified host name and allowing to retrieve a @@ -297,89 +408,7 @@ package_upgrade: true package_reboot_if_required: true ``` -The main terraform configuration file is using variables that have to be -described in a `vars.tf` file: - -```terraform -# Terraform variables definition -# Values to be provided in a *.tfvars file passed on the command line - -variable "internal_net_id" { - type = string - description = "The id of the internal network" -} - -variable "public_ip_pool" { - type = string - description = "The name of the public IP address pool" -} - -variable "image_id" { - type = string - description = "VM image id" -} - -variable "flavor_id" { - type = string - description = "VM flavor id" -} - -variable "security_groups" { - type = list(string) - description = "List of security groups" -} - -``` - -To be more reusable, the `main.tf` configuration file is referencing variables -described in a `vars.tf` file created previously, and will take the values from -the `$EGI_SITE.tfvars` file passed as an argument to the terraform command. - -```terraform -# Terraform versions and providers -terraform { - required_version = ">= 0.14.0" - required_providers { - openstack = { - source = "terraform-provider-openstack/openstack" - version = "~> 1.35.0" - } - } -} - -# Allocate a floating IP from the public IP pool -resource "openstack_networking_floatingip_v2" "egi_vm_floatip_1" { - pool = var.public_ip_pool -} - -# Creating the VM -resource "openstack_compute_instance_v2" "egi_vm" { - name = "egi_test_vm" - image_id = var.image_id - flavor_id = var.flavor_id - security_groups = var.security_groups - user_data = file("cloud-init.yaml") - network { - uuid = var.internal_net_id - } -} - -# Attach the floating public IP to the created instance -resource "openstack_compute_floatingip_associate_v2" "egi_ui_fip_1" { - instance_id = "${openstack_compute_instance_v2.egi_ui.id}" - floating_ip = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" -} - -# Create inventory file for Ansible -resource "local_file" "hosts_cfg" { - content = templatefile("${path.module}/hosts.cfg.tpl", - { - ui = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" - } - ) - filename = "./inventory/hosts.cfg" -} -``` +#### Launching the terraform deployment Now that all the files have been created, it's possible to deploy the infrastructure, currently only a single VM, but it can easily be extended to a @@ -404,30 +433,6 @@ $ ssh egi@$NSUPATE_HOSTNAME > [Ansible](#testing-ansible-access) to configure the remote machine, as well as > doing manual work via SSH. -#### Testing Ansible access - -The [terraform deployment](#deploying-the-virtual-machine-with-terraform) -generated an -[Ansible inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html), -`inventory/hosts.cfg`, that can directly be used by Ansible. - -Configure a basic Ansible environment in the `ansible.cfg` file: - -```ansible -[defaults] -# Use user created using cloud-init.yml -remote_user = egi -# Use inventory file generated by terraform -inventory = ./inventory/hosts.cfg -``` - -Then you can verify that the Virtual Machine is accessible by Ansible: - -```shell -# Test if ansible can reach the vm -$ ansible all -m ping -``` - #### Debugging terraform The token used by Terraform for accessing OpenStack is short lived, it will have @@ -455,6 +460,30 @@ $ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" $ terraform destroy --var-file="${EGI_SITE}.tfvars" ``` +#### Testing Ansible access + +The [terraform deployment](#deploying-the-virtual-machine-with-terraform) +generated an +[Ansible inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html), +`inventory/hosts.cfg`, that can directly be used by Ansible. + +Configure a basic Ansible environment in the `ansible.cfg` file: + +```ansible +[defaults] +# Use user created using cloud-init.yml +remote_user = egi +# Use inventory file generated by terraform +inventory = ./inventory/hosts.cfg +``` + +Then you can verify that the Virtual Machine is accessible by Ansible: + +```shell +# Test if ansible can reach the vm +$ ansible all -m ping +``` + ## Asking for help If you find issues please do not hesitate to [contact us](../../../support/). From 4f2c87a235bcb1c18e6ce4649d50760757c15fab Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:01:12 +0200 Subject: [PATCH 12/35] Fix link --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 5acb7d16fd..b73b1e0cb9 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -275,8 +275,8 @@ commands. > another site it may not be needed, and in that case the `main.tf` will have to > be adjusted accordingly. -See the example [`IN2P3-IRES.tfvars`](IN2P3-IRES.tfvars), to be adjusted -according to the requirements and to the selected site and VO: +See the example `IN2P3-IRES.tfvars` below, to be adjusted according to the +requirements and to the selected site and VO: ```terraform # Internal network From b16130d87280add50f10d2fabc73a1fd90c7d8e2 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:19:29 +0200 Subject: [PATCH 13/35] Fix terminology --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index b73b1e0cb9..18ded98fa9 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -363,7 +363,7 @@ The initial configuration of the VM is done using a `cloud-init.yaml` file. The `curl` call in the `cloud-init.yaml` configuration below, will register the IP of the virtual machine in the DNS zone managed using the [EGI Dynamic DNS service](https://nsupdate.fedcloud.eu/), allowing to access the -virtual machine using a fully qualified host name and allowing to retrieve a +virtual machine using a fully qualified hostname and allowing to retrieve a [Let's Encrypt certificate](https://letsencrypt.org/). > Please look at the From ec1d61e4b1251acab8358d6dd9c8f33359d1c0e2 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:33:49 +0200 Subject: [PATCH 14/35] Fix header level, link to ansible homepage --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 18ded98fa9..a7069c8d00 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -460,12 +460,13 @@ $ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" $ terraform destroy --var-file="${EGI_SITE}.tfvars" ``` -#### Testing Ansible access +### Testing Ansible access The [terraform deployment](#deploying-the-virtual-machine-with-terraform) generated an [Ansible inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html), -`inventory/hosts.cfg`, that can directly be used by Ansible. +`inventory/hosts.cfg`, that can directly be used by +[Ansible](https://www.ansible.com/). Configure a basic Ansible environment in the `ansible.cfg` file: From c8e32feb364216ae512a752db965eadaa5b74c81 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:34:23 +0200 Subject: [PATCH 15/35] Rephrase comment --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index a7069c8d00..7b159c6a36 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -481,7 +481,7 @@ inventory = ./inventory/hosts.cfg Then you can verify that the Virtual Machine is accessible by Ansible: ```shell -# Test if ansible can reach the vm +# Confirming ansible can reach the VM $ ansible all -m ping ``` From b2d6c98963ce93a52c72bf55c99bc9f201f50e07 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:44:21 +0200 Subject: [PATCH 16/35] Fix file excerpt type --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 7b159c6a36..242de5238d 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -470,7 +470,7 @@ generated an Configure a basic Ansible environment in the `ansible.cfg` file: -```ansible +```ini [defaults] # Use user created using cloud-init.yml remote_user = egi From 7d8973553d27d1f400f211b6a1a092d9bff88c87 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:53:32 +0200 Subject: [PATCH 17/35] Update ansible part --- .../oidc-agent-fedcloudclient-terraform/_index.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 242de5238d..bac3806835 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -429,9 +429,10 @@ $ terraform apply --var-file="${EGI_SITE}.tfvars" $ ssh egi@$NSUPATE_HOSTNAME ``` -> From here you can extend the `cloud-init.yaml` and/or use -> [Ansible](#testing-ansible-access) to configure the remote machine, as well as -> doing manual work via SSH. +> From here you can extend the +> [cloud-init.yaml](#initial-configuration-of-the-vm-using-cloud-init) and/or +> use [Ansible](#step-3-using-ansible) to configure the remote machine, as well +> as doing manual work via SSH. #### Debugging terraform @@ -460,7 +461,10 @@ $ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" $ terraform destroy --var-file="${EGI_SITE}.tfvars" ``` -### Testing Ansible access +## Step 3: Using Ansible + +[Ansible](https://www.ansible.com/) can be used to manage the configuration of +the crated virtual machine. The [terraform deployment](#deploying-the-virtual-machine-with-terraform) generated an @@ -485,6 +489,9 @@ Then you can verify that the Virtual Machine is accessible by Ansible: $ ansible all -m ping ``` +Once this works, you can create advanced playbooks to manage your deployed +host(s). + ## Asking for help If you find issues please do not hesitate to [contact us](../../../support/). From 9cd58caa835ea8683391273970fe857ee3de9660 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:53:56 +0200 Subject: [PATCH 18/35] Step 3 bis is step 4 --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index bac3806835..479d59746c 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -461,7 +461,7 @@ $ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" $ terraform destroy --var-file="${EGI_SITE}.tfvars" ``` -## Step 3: Using Ansible +## Step 4: Using Ansible [Ansible](https://www.ansible.com/) can be used to manage the configuration of the crated virtual machine. From b2ba1ff19b68ac7e9e9864597814c1577f351910 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Tue, 18 Oct 2022 19:55:11 +0200 Subject: [PATCH 19/35] Rename step 1 --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 479d59746c..42cbb61978 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -20,7 +20,7 @@ simplify deploying an infrastructure. assign a domain name to the virtual machine, which can then be used to get a valid TLS certificate from [Let's Encrypt](https://letsencrypt.org/). -## Step 1: Signing up +## Step 1: Signing up for an EGI Check-in account Create an EGI account with [Check-in](../../aai/check-in/signup). From 868628a233b6a6e4683c6764473a36d6ea4b3e57 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Wed, 19 Oct 2022 09:50:34 +0200 Subject: [PATCH 20/35] Let it ing --- .../oidc-agent-fedcloudclient-terraform/_index.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 42cbb61978..59b01f7a64 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -415,17 +415,17 @@ infrastructure, currently only a single VM, but it can easily be extended to a more complex setup, using terraform: ```shell -# Initialise working directory, install dependencies +# Initialising working directory, install dependencies $ terraform init -# Review plan of actions for creating the infrastructure +# Reviewing plan of actions for creating the infrastructure # Use relevant site-specific config file $ terraform plan --var-file="${EGI_SITE}.tfvars" -# Create the infrastructure +# Creating the infrastructure # Manual approval can be skipped using -auto-approve # The SERVER_ID will be printed (openstack_compute_instance_v2.scoreboard) $ terraform apply --var-file="${EGI_SITE}.tfvars" # Wait a few minutes for the setup to be finalised -# Connect to the server using ssh +# Connecting to the server using ssh $ ssh egi@$NSUPATE_HOSTNAME ``` @@ -456,7 +456,6 @@ $ OS_DEBUG=1 TF_LOG=DEBUG terraform apply --var-file="${EGI_SITE}.tfvars" #### Destroying the resources created by terraform ```shell -# Debugging # Destroying the created infrastructure $ terraform destroy --var-file="${EGI_SITE}.tfvars" ``` From ff39af39e0dbe62aeed111a2bb84786ab35391b2 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Wed, 19 Oct 2022 09:57:37 +0200 Subject: [PATCH 21/35] Add some more links --- .../oidc-agent-fedcloudclient-terraform/_index.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 59b01f7a64..660ba793fe 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -353,6 +353,11 @@ resource "local_file" "hosts_cfg" { } ``` +The last resource is relying on +[templatefile](https://www.terraform.io/language/functions/templatefile) to +populate the inventory file that will later be used by +[ansible](#step-4-using-ansible). + #### Initial configuration of the VM using cloud-init > [cloud-init](https://cloudinit.readthedocs.io/) is the industry standard @@ -488,8 +493,9 @@ Then you can verify that the Virtual Machine is accessible by Ansible: $ ansible all -m ping ``` -Once this works, you can create advanced playbooks to manage your deployed -host(s). +Once this works, you can +[create advanced playbooks](https://docs.ansible.com/ansible/latest/user_guide/index.html) +to configure your deployed host(s). ## Asking for help From 413ed4fb3fd304066f608fa501836c27e2347141 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Wed, 19 Oct 2022 10:01:39 +0200 Subject: [PATCH 22/35] Motto links --- .../oidc-agent-fedcloudclient-terraform/_index.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 660ba793fe..c5800cd34a 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -365,8 +365,10 @@ populate the inventory file that will later be used by The initial configuration of the VM is done using a `cloud-init.yaml` file. -The `curl` call in the `cloud-init.yaml` configuration below, will register the -IP of the virtual machine in the DNS zone managed using the +The `curl` call from the +[runcmd](https://cloudinit.readthedocs.io/en/latest/topics/modules.html#runcmd) +block in the `cloud-init.yaml` configuration below, will register the IP of the +virtual machine in the DNS zone managed using the [EGI Dynamic DNS service](https://nsupdate.fedcloud.eu/), allowing to access the virtual machine using a fully qualified hostname and allowing to retrieve a [Let's Encrypt certificate](https://letsencrypt.org/). @@ -375,8 +377,10 @@ virtual machine using a fully qualified hostname and allowing to retrieve a > [EGI Dynamic DNS documentation](../../compute/cloud-compute/dynamic-dns/) for > instructions on creating the configuration for a new host. -The `users` block in the `cloud-init.yaml` configuration below, will create a -new user with password-less [sudo](https://www.sudo.ws/) access. +The +[users](https://cloudinit.readthedocs.io/en/latest/topics/modules.html#users-and-groups) +block in the `cloud-init.yaml` configuration below, will create a new user with +password-less [sudo](https://www.sudo.ws/) access. > While this `egi` user can only be accessed via the specified SSH key(s), > setting a user password and requesting password verification for using sudo From af0161bcdd4ce30a5c3eb128c3ea36cb8786c067 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Wed, 19 Oct 2022 11:30:18 +0200 Subject: [PATCH 23/35] Fix link fragments --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index c5800cd34a..bb6599c084 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -146,8 +146,8 @@ $ pip install -r requirements.txt It's possible to deploy an OpenStack Virtual Machine (VM) on any of the sites supporting the Virtual Organisations (VO) you are a member of. -Once [fedcloudclient](#installing-fedcloudclient-openstackclient-and-ansible) is -installed it's possible to get information about the OIDC token accessed via +Once [fedcloudclient](#installing-fedcloudclient-and-ansible) is installed it's +possible to get information about the OIDC token accessed via [oidc-agent](#setting-up-oidc-agent). ```shell @@ -440,7 +440,7 @@ $ ssh egi@$NSUPATE_HOSTNAME > From here you can extend the > [cloud-init.yaml](#initial-configuration-of-the-vm-using-cloud-init) and/or -> use [Ansible](#step-3-using-ansible) to configure the remote machine, as well +> use [Ansible](#step-4-using-ansible) to configure the remote machine, as well > as doing manual work via SSH. #### Debugging terraform From 0a5291c5e7b91a12d43a266703cb9a651cbe5c2e Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Wed, 19 Oct 2022 11:39:19 +0200 Subject: [PATCH 24/35] Ansible: escalate privileges using sudo --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index bb6599c084..b6a308868c 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -488,6 +488,10 @@ Configure a basic Ansible environment in the `ansible.cfg` file: remote_user = egi # Use inventory file generated by terraform inventory = ./inventory/hosts.cfg + +[privilege_escalation] +# Escalate privileges using password-less sudo +become = yes ``` Then you can verify that the Virtual Machine is accessible by Ansible: From 7a9ec83923b530890d7cbdf7f7e435a53678cb65 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 09:43:56 +0200 Subject: [PATCH 25/35] Update content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Enol Fernández --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index b6a308868c..38a94b52e8 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -113,7 +113,7 @@ services, and mainly execute commands on OpenStack sites in EGI infrastructure. installed and properly configured. `fedcloudclient` and -[`opentackclient`](https://docs.openstack.org/python-openstackclient/latest/), +[`openstackclient`](https://docs.openstack.org/python-openstackclient/latest/), the official OpenStack python client, will be used to interact with the EGI Cloud Compute service. From 9e9aaff751ac1e08282ff9dbb8ae322914e75442 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 09:49:57 +0200 Subject: [PATCH 26/35] No need to st, check or use any OS_ACCESS_TOKEN --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 38a94b52e8..12fc65104d 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -81,7 +81,8 @@ $ oidc-add -l $ oidc-token egi # Exporting a variable with a Check-in OIDC access token to be used with OpenStack # XXX access tokens are short lived, relaunch command to obtain a new token -$ export OS_ACCESS_TOKEN=`oidc-token egi` +# This is *not* required for following this tutorial, it's an example +$ export OS_ACCESS_TOKEN=$(oidc-token egi) ``` It's possible to automatically start `oidc-agent` in your shell initialisation, @@ -151,8 +152,6 @@ possible to get information about the OIDC token accessed via [oidc-agent](#setting-up-oidc-agent). ```shell -# Verifying that an OS_ACCESS_TOKEN is available -$ echo $OS_ACCESS_TOKEN # Listing the VO membership related to the OIDC access token $ fedcloud token list-vos ``` From 4130442e21ad8b9d4b9b7731e180f4e602818fc3 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 09:56:56 +0200 Subject: [PATCH 27/35] Link to section on setting up the environment --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 12fc65104d..28fdb4333a 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -244,8 +244,9 @@ The SITE and VO specific values for those variables will be #### Identifying the cloud resources -Identify flavor, image, network and security groups for the site you want to -use, using the information gathered with `fedcloudclient`. +Once the [environment is properly configure](#setting-up-the-environment), +`fedcloudclient` is used to gather information and identify flavor, image, +network and security groups for the site you want to use. ```shell # Selecting an image From 80e08252530047b835b0c0d69e96f72e46be6575 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 10:09:37 +0200 Subject: [PATCH 28/35] Use everywhere %() instead of `` --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 28fdb4333a..c61be87a29 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -90,7 +90,7 @@ example that can be added to `~/.bash_profile` or `~/.zshrc`: ```bash if command -v oidc-agent-service &> /dev/null - eval `oidc-agent-service use` + eval $(oidc-agent-service use) # for fedcloudclient, once egi account got created export OIDC_AGENT_ACCOUNT=egi fi @@ -194,7 +194,7 @@ $ source ~/.virtualenvs/fedcloudclient/bin/activate # Exporting variable for VO and SITE to avoid having to repeat them $ export EGI_VO='vo.access.egi.eu' $ export EGI_SITE='IN2P3-IRES' -eval `fedcloud site env` +eval $(fedcloud site env) # Obtaining an OS_TOKEN for terraform # XXX this breaks using openstackclient: use fedcloudclient # or unset OS_TOKEN before using openstackclient From d59541122f419b1eb294daf1e98049abf4e71ef9 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 10:14:27 +0200 Subject: [PATCH 29/35] Ignore generated hugo lock file --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index adcbeaa23e..f0e9dea027 100644 --- a/.gitignore +++ b/.gitignore @@ -3,5 +3,5 @@ !/public/.gitkeep # From NodeJS for docsy node_modules/ - +.hugo_build.lock .DS_Store From 0747679c9704db5d1c25f7fc68b194b71009b660 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 10:39:56 +0200 Subject: [PATCH 30/35] Clarify OIDC_AGENT_ACCOUNT documentation --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index c61be87a29..b84eda75d5 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -91,7 +91,7 @@ example that can be added to `~/.bash_profile` or `~/.zshrc`: ```bash if command -v oidc-agent-service &> /dev/null eval $(oidc-agent-service use) - # for fedcloudclient, once egi account got created + # for fedcloudclient, selecting egi configuration generated with oidc-gen export OIDC_AGENT_ACCOUNT=egi fi ``` From 944879634fc12d8a92a00d50adf3f8ee901ed6b9 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 10:53:10 +0200 Subject: [PATCH 31/35] Link to additional resources (ansible and other repos) --- .../_index.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index b84eda75d5..549ea1e877 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -505,6 +505,29 @@ Once this works, you can [create advanced playbooks](https://docs.ansible.com/ansible/latest/user_guide/index.html) to configure your deployed host(s). +Various Ansible roles are available in the +[egi-qc/ansible-playbooks repository](https://github.com/egi-qc/ansible-playbooks) +and in the +[EGI Federation GitHub organisation](https://github.com/EGI-Federation?q=ansible-role). + +> A +> [style guide for writing Ansible roles](https://github.com/EGI-Federation/ansible-style-guide) +> is providing a skeleton that you can use fore creating new roles. + +## Additional resources + +Additional resources are available, and can help with addressing different use +cases, or be used as a source of inspiration: + +- [egi-qc/deployment-howtos](https://github.com/egi-qc/deployment-howtos): + Deployment recipes extracted from Jenkins builds for the + [UMD](https://go.egi.eu/umd) and [CMD](https://go.egi.eu/cmd) products +- [EGI-ILM/fedcloud-terraform](https://github.com/EGI-ILM/fedcloud-terraform): + providing an advanced helper script allowing to interact with EGI Cloud + Compute. +- [EGI-ILM/automated-containers](https://github.com/EGI-ILM/automated-containers): + providing documentation for automated on-demand execution of Docker containers + ## Asking for help If you find issues please do not hesitate to [contact us](../../../support/). From 8fa8604c04d01a39fc15f80ed3d454ba4a204ac3 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 13:46:41 +0200 Subject: [PATCH 32/35] Document why we set --site for fedcloud openstack. --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 549ea1e877..f2e087887e 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -248,6 +248,12 @@ Once the [environment is properly configure](#setting-up-the-environment), `fedcloudclient` is used to gather information and identify flavor, image, network and security groups for the site you want to use. +> `fedcloud openstack` currently requires an explicit `--site` parameter, this +> will be addressed in a +> [future fedcloud release](https://github.com/tdviet/fedcloudclient/issues/150). +> In the meantime the `$EGI_VO` environment variable can be reused using +> `--site "$EGI_VO"`. + ```shell # Selecting an image $ fedcloud select image --image-specs "Name =~ 'EGI.*22'" From 11fd3553d6d06c53c7f8bda25dc114f3f8e92c3c Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 14:56:05 +0200 Subject: [PATCH 33/35] Update content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md Co-authored-by: Sebastian Luna-Valero --- .../tutorials/oidc-agent-fedcloudclient-terraform/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index f2e087887e..5003080c70 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -251,8 +251,8 @@ network and security groups for the site you want to use. > `fedcloud openstack` currently requires an explicit `--site` parameter, this > will be addressed in a > [future fedcloud release](https://github.com/tdviet/fedcloudclient/issues/150). -> In the meantime the `$EGI_VO` environment variable can be reused using -> `--site "$EGI_VO"`. +> In the meantime the `$EGI_SITE` environment variable can be reused using +> `--site "$EGI_SITE"`. ```shell # Selecting an image From 500b45a5b4b1696b4a08f2f35cddfb165844fc0a Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 20 Oct 2022 15:26:55 +0200 Subject: [PATCH 34/35] Rephrase, link to main.tf creation, reference CentOS 7 image and review resource names in the main.tf --- .../_index.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md index 5003080c70..ac00d98c62 100644 --- a/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md +++ b/content/en/users/tutorials/oidc-agent-fedcloudclient-terraform/_index.md @@ -275,11 +275,12 @@ The chosen flavor, image, network and security group should be documented in a `$EGI_SITE.tfvars` file that will be passed as an argument to terraform commands. -> The network configuration can be tricky and is usually dependant on the site. +> The network configuration can be tricky, and is usually dependant on the site. > For `IN2P3-IRES`, one has to request a floating IP from the public network IP -> pool `ext-net` and assign this floating IP to the created instance, for -> another site it may not be needed, and in that case the `main.tf` will have to -> be adjusted accordingly. +> pool `ext-net`, and assign this floating IP to the created instance. For +> another site it may not be needed, in that case the +> [main.tf](#creating-the-main-terraform-deployment-file) will have to be +> adjusted accordingly. See the example `IN2P3-IRES.tfvars` below, to be adjusted according to the requirements and to the selected site and VO: @@ -294,9 +295,12 @@ public_ip_pool = "ext-net" # Flavor: m1.medium flavor_id = "ab1fbd4c-324d-4155-bd0f-72f077f0ebce" +# Image for EGI CentOS 7 +# https://appdb.egi.eu/store/vappliance/egi.centos.7 +image_id = "09093c70-f2bb-46b8-a87f-00e2cc0c8542" # Image: EGI CentOS 8 # https://appdb.egi.eu/store/vappliance/egi.centos.8 -image_id = "38ced5bf-bbfd-434b-ae41-3ab35d929aba" +# image_id = "38ced5bf-bbfd-434b-ae41-3ab35d929aba" # Image: EGI Ubuntu 22.04 # https://appdb.egi.eu/store/vappliance/egi.ubuntu.22.04 # image_id = "fc6c83a3-845f-4f29-b44d-2584f0ca4177" @@ -343,16 +347,16 @@ resource "openstack_compute_instance_v2" "egi_vm" { } # Attach the floating public IP to the created instance -resource "openstack_compute_floatingip_associate_v2" "egi_ui_fip_1" { - instance_id = "${openstack_compute_instance_v2.egi_ui.id}" - floating_ip = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" +resource "openstack_compute_floatingip_associate_v2" "egi_vm_fip_1" { + instance_id = "${openstack_compute_instance_v2.egi_vm.id}" + floating_ip = "${openstack_networking_floatingip_v2.egi_vm_floatip_1.address}" } # Create inventory file for Ansible resource "local_file" "hosts_cfg" { content = templatefile("${path.module}/hosts.cfg.tpl", { - ui = "${openstack_networking_floatingip_v2.egi_ui_floatip_1.address}" + ui = "${openstack_networking_floatingip_v2.egi_vm_floatip_1.address}" } ) filename = "./inventory/hosts.cfg" From f311e4a5cc4c7fa73beb7eac1aad6a20f166ad49 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Fri, 21 Oct 2022 09:06:51 +0200 Subject: [PATCH 35/35] Adjust CODEOWNERS to avoid bloackage --- CODEOWNERS | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CODEOWNERS b/CODEOWNERS index f49c951276..5a3df589aa 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -7,7 +7,7 @@ # These owners will be the default owners for everything in the repo. # Unless a later match takes precedence, they will be requested for # review when someone opens a pull request. -* @enolfc @glarocca @gwarf @andrea-manzi @thebe14 +* @enolfc @glarocca @gwarf @andrea-manzi # Internal services content/en/internal/ @gwarf @paolini78 @@ -16,7 +16,7 @@ content/en/internal/ @gwarf @paolini78 content/en/providers/ @gwarf @paolini78 # Users -content/en/users/ @enolfc @glarocca @andrea-manzi @gwarf @thebe14 @sebastian-luna-valero +content/en/users/ @enolfc @glarocca @andrea-manzi @gwarf @sebastian-luna-valero # Storage-related services content/en/users/data/ @andrea-manzi @@ -35,10 +35,10 @@ content/en/users/providers/notebooks/ @enolfc @andrea-manzi @sebastian-luna-vale content/en/users/dev-env/binder/ @enolfc @andrea-manzi @sebastian-luna-valero # Training -content/en/users/training/ @glarocca @thebe14 +content/en/users/training/ @glarocca # Tutorials -content/en/users/tutorials/ @thebe14 +content/en/users/tutorials/ @enolfc @sebastian-luna-valero # HTC-related services content/en/users/compute/content-distribution/ @CatalinCondurache