From 9dc1d6800cbf90f3fca871643e6c940e2b6860ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Sun, 16 Jun 2024 12:48:07 +0200 Subject: [PATCH 01/28] add MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- .../deploy-guide/services/infrastructure.md | 35 ++++++++++++------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/docs/guides/deploy-guide/services/infrastructure.md b/docs/guides/deploy-guide/services/infrastructure.md index a55b941675..5f28966683 100644 --- a/docs/guides/deploy-guide/services/infrastructure.md +++ b/docs/guides/deploy-guide/services/infrastructure.md @@ -8,7 +8,19 @@ sidebar_position: 10 Common issues with deploying infrastructure services required by OpenStack are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-guide/openstack). -1. Cron, Fluentd & Kolla Toolbox + +1. Optional: In order to reduce the active observation time for the deployment of the components, + the container images for the following services can be downloaded in advance with the argument `-a pull`. + ``` + osism apply -a pull common + osism apply -a pull loadbalancer + osism apply -a pull redis + osism apply -a pull memcached + osism apply -a pull rabbitmq + osism apply -a pull mariadb + ``` + +2. Cron, Fluentd & Kolla Toolbox The common role of Kolla is used to manage the services `cron`, `fluentd` and `kolla-toolbox`. @@ -18,41 +30,40 @@ are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-gu service. ``` - osism apply -a pull common osism apply common ``` -2. Loadbalancer +3. Loadbalancer + Have a look to the [loadbalancer documentation](../../configuration-guide/loadbalancer.md) and conifigure it before deploying the service. ``` - osism apply -a pull loadbalancer osism apply loadbalancer ``` -3. Redis + It is important to do this deployment before any other deployements in the Kolla + environment, as parts of the other deployments depend on the loadbalancer + service. + +4. Redis ``` - osism apply -a pull redis osism apply redis ``` -4. Memcached +5. Memcached ``` - osism apply -a pull memcached osism apply memcached ``` -5. RabbitMQ +6. RabbitMQ ``` - osism apply -a pull rabbitmq osism apply rabbitmq ``` -6. MariaDB +7. MariaDB ``` - osism apply -a pull mariadb osism apply mariadb ``` From 9c7e377028f6ef3482bfe8d70bc2c7b436d0b0a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Sun, 16 Jun 2024 12:54:53 +0200 Subject: [PATCH 02/28] typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/deploy-guide/services/infrastructure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/deploy-guide/services/infrastructure.md b/docs/guides/deploy-guide/services/infrastructure.md index 5f28966683..49487a84f9 100644 --- a/docs/guides/deploy-guide/services/infrastructure.md +++ b/docs/guides/deploy-guide/services/infrastructure.md @@ -35,7 +35,7 @@ are documented in the [OpenStack Troubleshooting Guide](../../troubleshooting-gu 3. Loadbalancer - Have a look to the [loadbalancer documentation](../../configuration-guide/loadbalancer.md) and conifigure it before deploying the service. + Have a look to the [loadbalancer documentation](../../configuration-guide/loadbalancer.md) and configure it before deploying the service. ``` osism apply loadbalancer ``` From 653fcdaed20fdae1e571fc8b2a961309b41af59b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Mon, 17 Jun 2024 12:53:24 +0200 Subject: [PATCH 03/28] Improve network documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/deploy-guide/services/network.md | 29 ++++++++++---------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/docs/guides/deploy-guide/services/network.md b/docs/guides/deploy-guide/services/network.md index c397f7de16..e503ca5c8f 100644 --- a/docs/guides/deploy-guide/services/network.md +++ b/docs/guides/deploy-guide/services/network.md @@ -5,6 +5,20 @@ sidebar_position: 15 # Network +Two options are available for installing the Openstack network. + +* Open Virtual Network with Open vSwitch: + * the default, recommended + * configuration value `ovn` +* Open vSwitch: + * configuration value `openvswitch` + +The selected option is configured in the following file: +```yaml title="environments/kolla/configuration.yml" +# neutron +neutron_plugin_agent: "ovn" +``` + 1. Open vSwitch (OVS) ``` @@ -12,20 +26,7 @@ sidebar_position: 15 osism apply openvswitch ``` -2. Open Virtual Network (OVN) - - In `environments/kolla/configuration.yml` the parameter `neutron_plugin_agent` is set to - `ovn` if OVN is used as a network plugin. The parameter is set to `ovn` by default in the - Cookiecutter. - - ```yaml title="environments/kolla/configuration.yml" - # neutron - neutron_plugin_agent: "ovn" - ``` - - Otherwise the network plugin is set to `openvswitch`. - If the `neutron_plugin_agent` is set to `openvswitch`, this step does not need to be done. - +2. Optional: Open Virtual Network (OVN) Before the deployment of OVN, the deployment of Open vSwitch must already have been done. ``` From a41068a0911d8cfed686866cb1b8e311ca9e8981 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 20 Jun 2024 19:08:46 +0200 Subject: [PATCH 04/28] update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 83 ++++++++++++++-------- docs/guides/deploy-guide/services/ceph.mdx | 11 ++- 2 files changed, 61 insertions(+), 33 deletions(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index 7949294356..94686f84d2 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -13,6 +13,9 @@ It is **strongly advised** to use the documentation for the version being used. * Quincy - https://docs.ceph.com/en/quincy/rados/configuration/ * Reef - https://docs.ceph.com/en/reef/rados/configuration/ +Its a good idea to review all options in the follwing list. + + ## Unique Identifier The File System ID is a unique identifier for the cluster. @@ -23,6 +26,12 @@ and must be unique. It can be generated with `uuidgen`. fsid: c2120a4a-669c-4769-a32c-b7e9d7b848f4 ``` +## Configure the mon address on the mon nodes + +Set the variable `monitor_address` in the inventory files of the mon hosts to give ceph-ansible the +advise which ip adress should be used to reach the monitor instances. +(`inventory/host_vars/.yml`). + ## Client The `client.admin` keyring is placed in the file `environments/infrastructure/files/ceph/ceph.client.admin.keyring`. @@ -125,7 +134,7 @@ vm.min_free_kbytes=4194303 ceph-control ``` -## Extra pools +## Configuration of custom ceph pools Extra pools can be defined via the `openstack_pools_extra` parameter. @@ -156,29 +165,41 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in ## OSD devices -1. For each Ceph storage node edit the file `inventory/host_vars/.yml` - add a configuration like the following to it. Ensure that no `devices` parameter - is present in the file. - - 1. Parameters - - * With the optional parmaeter `ceph_osd_db_wal_devices_buffer_space_percent` it is possible to - set the percentage of VGs to leave free. The parameter is not set by default. Can be helpful - for SSD performance of some older SSD models or to extend lifetime of SSDs in general. - - ```yaml - ceph_osd_db_wal_devices_buffer_space_percent: 10 - ``` - * It is possible to configure the devices to be used with the parameters `ceph_osd_devices`, - `ceph_db_devices`, `ceph_wal_devices`, and `ceph_db_wal_devices`. This is described below. - * It is always possible to use device names such as `sda` or device IDs such as - `disk/by-id/wwn-` or `disk/by-id/nvme-eui.`. `/dev/` is not - prefixed and is added automatically. - * The `db_size` parameter is optional and defaults to `(VG size - buffer space (if enabled)) / num_osds`. - * The `wal_size` parameter is optional and defaults to `2 GB`. - * The `num_osds` parameter specifies the maximum number of OSDs that can be assigned to a WAL device or DB device. - * The optional parameter `wal_pv` can be used to set the device that is to be used as the WAL device. - * The optional parameter `db_pv` can be used to set the device that is to be used as the DB device. +For more advanced OSD layout requirements leave out the `devices` key +and instead use `lvm_volumes`. Details for this can be found on the +[OSD Scenario](https://docs.ceph.com/projects/ceph-ansible/en/latest/osds/scenarios.html) documentation. + +In order to aid in creating the `lvm_volumes` config entries and provision the LVM devices for them, +OSISM has the two playbooks `ceph-configure-lvm-volumes` and `ceph-create-lvm-devices` available. +TODO: add reference to https://docs.ceph.com/en/latest/rados/operations/pgcalc/ and PG autocaler dryrun + +### Configure the device layout + +For each Ceph storage node edit the file `inventory/host_vars/.yml` +add a configuration like the following to it. Ensure that no `devices` parameter +is present in the file. + +**General information about the parameters** + +* With the optional parmaeter `ceph_osd_db_wal_devices_buffer_space_percent` it is possible to + set the percentage of VGs to leave free. The parameter is not set by default. Can be helpful + for SSD performance of some older SSD models or to extend lifetime of SSDs in general. + + ```yaml + ceph_osd_db_wal_devices_buffer_space_percent: 10 + ``` +* It is possible to configure the devices to be used with the parameters `ceph_osd_devices`, + `ceph_db_devices`, `ceph_wal_devices`, and `ceph_db_wal_devices`. This is described below. +* It is always possible to use device names such as `sda` or device IDs such as + `disk/by-id/wwn-` or `disk/by-id/nvme-eui.`. + The top level dierectory `/dev/` is not prefixed and is added automatically. +* The `db_size` parameter is optional and defaults to `(VG size - buffer space (if enabled)) / num_osds`. +* The `wal_size` parameter is optional and defaults to `2 GB`. +* The `num_osds` parameter specifies the maximum number of OSDs that can be assigned to a WAL device or DB device. +* The optional parameter `wal_pv` can be used to set the device that is to be used as the WAL device. +* The optional parameter `db_pv` can be used to set the device that is to be used as the DB device. + +**Layout variants** 2. OSD only @@ -282,7 +303,9 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in wal_pv: nvme1n1 ``` -2. Push the configuration to your configuration repository and after that do the following +### Rollout the configured layout + +1. Push the configuration to your configuration repository and after that do the following ``` $ osism apply configuration @@ -290,7 +313,7 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in $ osism apply facts ``` -3. After the configuration has been pulled and facts updated, +2. After the configuration has been pulled and facts updated, you can run the LVM configuration playbook: ``` @@ -300,7 +323,7 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in This will generate a new configuration file for each node in `/tmp` on the first manager node named `-ceph-lvm-configuration.yml`. -4. Take the generated configuration file from `/tmp` and **replace the previously +3. Take the generated configuration file from `/tmp` and **replace the previously configuration** for each node. In this example, the following content was in the host vars file before @@ -331,14 +354,14 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in This content from the file in the `/tmp` directory is added in the host vars file. The previous `ceph_osd_devices` is replaced with the new content. -5. Push the updated configuration **again** to your configuration repository and re-run: +4. Push the updated configuration **again** to your configuration repository and re-run: ``` $ osism apply configuration $ osism reconciler sync ``` -6. Finally create the LVM devices. +5. Finally create the LVM devices. ``` $ osism apply ceph-create-lvm-devices @@ -363,7 +386,7 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in osd-block-c6df96be-1264-5815-9cb2-da5eb453a6de ceph-c6df96be-1264-5815-9cb2-da5eb453a6de -wi-a----- <20.00g ``` -7. Everything is now ready for the deployment of the OSDs. +6. Everything is now ready for the deployment of the OSDs. Details on deploying Ceph in the [Ceph deploy guide](../deploy-guide/services/ceph). ### Full examples diff --git a/docs/guides/deploy-guide/services/ceph.mdx b/docs/guides/deploy-guide/services/ceph.mdx index 6a33607ab9..2c61f40a07 100644 --- a/docs/guides/deploy-guide/services/ceph.mdx +++ b/docs/guides/deploy-guide/services/ceph.mdx @@ -11,6 +11,7 @@ import TabItem from '@theme/TabItem'; In OSISM it is also possible to integrate and use existing Ceph clusters. It is not necessary to deploy Ceph with OSISM. If Ceph is deployed with OSISM, it should be noted that OSISM does not claim to provide all possible features of Ceph. + Ceph provided with OSISM is intended to provide the storage for Glance, Nova, Cinder and Manila. In a specific way that has been implemented by OSISM for years. It should be checked in advance whether the way in OSISM the Ceph deployment and the @@ -22,12 +23,14 @@ open source projects, please refer to :::warning -Before starting the Ceph deployment, the configuration and preparation of the -OSD devices must be completed. The steps that are required for this can be found in the -[Ceph Configuration Guide](../../configuration-guide/ceph.md#osd-devices). +Before starting the Ceph deployment, the it is recommended to perform the general ceph configuration. +All the preparing steps are listed in the [Ceph Configuration Guide](../../configuration-guide/ceph). + +At least the [preparation](../../configuration-guide/ceph.md#osd-devices) of the necessary LVM2 volumes for the osd devices must be completed. ::: + 1. Deploy services. * Deploy [ceph-mon](https://docs.ceph.com/en/quincy/man/8/ceph-mon/) services @@ -42,6 +45,8 @@ OSD devices must be completed. The steps that are required for this can be found osism apply ceph-mgrs ``` + * Prepare OSD devices [as described](../../configuration-guide/ceph#osd-devices) in the configuration guide + * Deploy [ceph-osd](https://docs.ceph.com/en/quincy/man/8/ceph-osd/) services ``` From 3f00ca4df9408be0be0db9e87bf5bd8870393ec3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 21 Jun 2024 12:15:07 +0200 Subject: [PATCH 05/28] improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index 94686f84d2..3cf73f81c4 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -36,7 +36,7 @@ advise which ip adress should be used to reach the monitor instances. The `client.admin` keyring is placed in the file `environments/infrastructure/files/ceph/ceph.client.admin.keyring`. -## Swappiness +## Systemctl Parameters, Swappiness and Friends The swappiness is set via the `os_tuning_params` dictionary. The dictionary can only be completely overwritten via an entry in the file `environments/ceph/configuration.yml`. @@ -305,15 +305,16 @@ is present in the file. ### Rollout the configured layout -1. Push the configuration to your configuration repository and after that do the following - +1. Commit and push the configuration to your configuration repository +2. Establish the changed configuration + Make sure that you do not have any open changes on the manager node either, as these will be discarded during this step. ``` $ osism apply configuration $ osism reconciler sync $ osism apply facts ``` -2. After the configuration has been pulled and facts updated, +3. After the configuration has been pulled and facts updated, you can run the LVM configuration playbook: ``` @@ -323,7 +324,7 @@ is present in the file. This will generate a new configuration file for each node in `/tmp` on the first manager node named `-ceph-lvm-configuration.yml`. -3. Take the generated configuration file from `/tmp` and **replace the previously +4. Take the generated configuration file from `/tmp` and **replace the previously configuration** for each node. In this example, the following content was in the host vars file before @@ -354,14 +355,15 @@ is present in the file. This content from the file in the `/tmp` directory is added in the host vars file. The previous `ceph_osd_devices` is replaced with the new content. -4. Push the updated configuration **again** to your configuration repository and re-run: - +5. Commit and push the configuration to your configuration repository **again** +6. Establish the changed configuration + Make sure that you do not have any open changes on the manager node either, as these will be discarded during this step. ``` $ osism apply configuration $ osism reconciler sync ``` -5. Finally create the LVM devices. +7. Finally create the LVM devices. ``` $ osism apply ceph-create-lvm-devices @@ -386,7 +388,7 @@ is present in the file. osd-block-c6df96be-1264-5815-9cb2-da5eb453a6de ceph-c6df96be-1264-5815-9cb2-da5eb453a6de -wi-a----- <20.00g ``` -6. Everything is now ready for the deployment of the OSDs. +7. Everything is now ready for the deployment of the OSDs. Details on deploying Ceph in the [Ceph deploy guide](../deploy-guide/services/ceph). ### Full examples From cc017f73cc792769438f0c2f85a1c336f604c2d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 21 Jun 2024 15:21:31 +0200 Subject: [PATCH 06/28] improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 185 ++++++++++++++---------- 1 file changed, 107 insertions(+), 78 deletions(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index 3cf73f81c4..1c885751f6 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -201,109 +201,138 @@ is present in the file. **Layout variants** - 2. OSD only +OSISM basically utilizes LVM volumes for all OSD setup variants. - The `sda` device will be used as an OSD device without WAL and DB device. + - ```yaml - ceph_osd_devices: - sda: - ``` + - 3. OSD + DB device +This variant does not use a dedicated wal- or db-device. +This is the most simple variant and this variant can be used if you use a all-flash setup with NVMes. - The `nvme0n1` device will be used as an DB device. It is possible to use this DB device for up to 6 OSDs. Each - OSD is provided with 30 GB. +The `sda` device will be used as an OSD device without WAL and DB volume device. - ```yaml - ceph_db_devices: - nvme0n1: - num_osds: 6 - db_size: 30 GB - ``` +```yaml +ceph_osd_devices: + sda: +``` + + + + +The `nvme0n1` device will be used as an source for DB device volumes. +With the configured values the provisioning mechanism creates 6 logical volumes of 30GB size each on the nvme which can +be used for 6 OSD instances. + + ```yaml + ceph_db_devices: + nvme0n1: + num_osds: 6 + db_size: 30 GB + ``` + +The devices `sda` up to `sdf` will use the previously defined DB volumes from `nvme0n1` for the listed OSD instances. + + ```yaml + ceph_osd_devices: + sda: + db_pv: nvme0n1 + ... + sdf: + db_pv: nvme0n1 + ``` + - The `sda` device will be used as an OSD device with `nvme0n1` as DB device. + - ```yaml - ceph_osd_devices: - sda: - db_pv: nvme0n1 - ``` +The `nvme0n1` device will be used as an source for WAL device volumes. +With the configured values the provisioning mechanism creates 6 logical volumes of 2B size each on `nvme0n1` which can +be used for 6 OSD instances. + + +```yaml +ceph_wal_devices: + nvme0n1: + num_osds: 6 + wal_size: 2 GB +``` + +The devices `sda` up to `sdf` will use the previously defined WAL volumes from `nvme0n1` for the listed OSD instances. + +```yaml +ceph_osd_devices: + sda: + wal_pv: nvme0n1 +``` + - 4. OSD + WAL device + - The `nvme0n1` device will be used as an WAL device. It is possible to use this WAL device for up to 6 OSDs. Each - OSD is provided with 2 GB. +The `nvme0n1` device will be used as an source for WAL and DB device volumes. +With the configured values the provisioning mechanism creates 6 logical DB volumes of 30GB and 6 logical WAL volumes of 2B size each +on `nvme0n1` which can be used for 6 OSD instances. - ```yaml - ceph_wal_devices: - nvme0n1: - num_osds: 6 - wal_size: 2 GB - ``` - The `sda` device will be used as an OSD device with `nvme0n1` as WAL device. +The `nvme0n1` device will be used as an DB device and a WAL device. It is possible to use those devices for up +to 6 OSDs. - ```yaml - ceph_osd_devices: - sda: - wal_pv: nvme0n1 - ``` +```yaml +ceph_db_wal_devices: + nvme0n1: + num_osds: 6 + db_size: 30 GB + wal_size: 2 GB +``` - 5. OSD + DB device + WAL device (same device for DB + WAL) +The `sda` device will be used as an OSD device with `nvme0n1` as DB device and `nvme0n1` as WAL device. - The `nvme0n1` device will be used as an DB device and a WAL device. It is possible to use those devices for up - to 6 OSDs. +```yaml +ceph_osd_devices: + sda: + db_pv: nvme0n1 + wal_pv: nvme0n1 +``` - ```yaml - ceph_db_wal_devices: - nvme0n1: - num_osds: 6 - db_size: 30 GB - wal_size: 2 GB - ``` +In the example shown here, both the data structures for the RocksDB and for the write-ahead log are placed on the faster NVMe device. +(This is described in the [Ceph documentation](https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/) as "...whenever a DB device is specified but an explicit WAL device is not, the WAL will be implicitly colocated with the DB on the faster device..."). - The `sda` device will be used as an OSD device with `nvme0n1` as DB device and `nvme0n1` as WAL device. + - ```yaml - ceph_osd_devices: - sda: - db_pv: nvme0n1 - wal_pv: nvme0n1 - ``` + - 6. OSD + DB device + WAL device (different device for DB + WAL) +The `nvme0n1` device will be used as an source for WAL and `nvme1n1` for DB device volumes. +With the configured values the provisioning mechanism creates 6 logical volumes of 30 GB and 6 logical WAL volumes of 2B size each. - The `nvme0n1` device will be used as an DB device. It is possible to use this DB device for up to 6 OSDs. Each - OSD is provided with 30 GB. +```yaml +ceph_db_devices: + nvme0n1: + num_osds: 6 + db_size: 30 GB +``` - ```yaml - ceph_db_devices: - nvme0n1: - num_osds: 6 - db_size: 30 GB - ``` +The `nvme1n1` device will be used as an WAL device. It is possible to use this WAL device for up to 6 OSDs. Each +OSD is provided with 2 GB. - The `nvme1n1` device will be used as an WAL device. It is possible to use this WAL device for up to 6 OSDs. Each - OSD is provided with 2 GB. +```yaml +ceph_wal_devices: + nvme1n1: + num_osds: 6 + wal_size: 2 GB +``` - ```yaml - ceph_wal_devices: - nvme1n1: - num_osds: 6 - wal_size: 2 GB - ``` +The `sda` device will be used as an OSD device with `nvme0n1` as DB device and `nvme1n1` as WAL device. - The `sda` device will be used as an OSD device with `nvme0n1` as DB device and `nvme1n1` as WAL device. +```yaml +ceph_osd_devices: + sda: + db_pv: nvme0n1 + wal_pv: nvme1n1 +``` + - ```yaml - ceph_osd_devices: - sda: - db_pv: nvme0n1 - wal_pv: nvme1n1 - ``` + -### Rollout the configured layout +### Provision the configured layout 1. Commit and push the configuration to your configuration repository 2. Establish the changed configuration From a260e2068dc490f25ed85c39f0cd8f6ada58414d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Tue, 25 Jun 2024 11:48:49 +0200 Subject: [PATCH 07/28] improve ceph deployment documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/deploy-guide/services/ceph.mdx | 121 ++++++++++++--------- 1 file changed, 71 insertions(+), 50 deletions(-) diff --git a/docs/guides/deploy-guide/services/ceph.mdx b/docs/guides/deploy-guide/services/ceph.mdx index 2c61f40a07..c4c19e7451 100644 --- a/docs/guides/deploy-guide/services/ceph.mdx +++ b/docs/guides/deploy-guide/services/ceph.mdx @@ -31,76 +31,92 @@ At least the [preparation](../../configuration-guide/ceph.md#osd-devices) of the ::: -1. Deploy services. +## Deploy ceph services. - * Deploy [ceph-mon](https://docs.ceph.com/en/quincy/man/8/ceph-mon/) services +* Deploy [ceph-mon](https://docs.ceph.com/en/quincy/man/8/ceph-mon/) services - ``` - osism apply ceph-mons - ``` + ``` + osism apply ceph-mons + ``` - * Deploy ceph-mgr services +* Deploy ceph-mgr services - ``` - osism apply ceph-mgrs - ``` + ``` + osism apply ceph-mgrs + ``` - * Prepare OSD devices [as described](../../configuration-guide/ceph#osd-devices) in the configuration guide +* Prepare OSD devices [as described](../../configuration-guide/ceph#osd-devices) in the configuration guide - * Deploy [ceph-osd](https://docs.ceph.com/en/quincy/man/8/ceph-osd/) services +* Deploy [ceph-osd](https://docs.ceph.com/en/quincy/man/8/ceph-osd/) services - ``` - osism apply ceph-osds - ``` + ``` + osism apply ceph-osds + ``` - * Generate pools and keys. This step is only necessary for OSISM >= 7.0.0. +* Configure custom pools [as described](../../configuration-guide/ceph#extra-pools) in the configuration guide - ``` - osism apply ceph-pools - ``` +* Generate pools and the related keys. This step is only necessary for OSISM >= 7.0.0. - * Deploy ceph-crash services + ``` + osism apply ceph-pools + ``` - ``` - osism apply ceph-crash - ``` +* Deploy ceph-crash services - :::info + ``` + osism apply ceph-crash + ``` - It's all done step by step here. It is also possible to do this in a single step. - This speeds up the entire process and avoids unnecessary restarts of individual - services. +:::info - - - ``` - osism apply ceph - ``` +It's all done step by step here. It is also possible to do this in a single step. +This speeds up the entire process and avoids unnecessary restarts of individual +services. + + + +``` +osism apply ceph +``` + +Generate pools and keys. + +``` +osism apply ceph-pools +``` + + +``` +osism apply ceph-base +``` + + + +::: - Generate pools and keys. +## Install Ceph Clients + +1. Get ceph keys. This places the necessary keys in `/opt/configuration`. ``` - osism apply ceph-pools + osism apply copy-ceph-keys ``` - - + +2. Encrypt the fetched keys + It is highly recommended to store the Ceph keys encrypted in the Git repository. ``` - osism apply ceph-base + cd /opt/configuration + make ansible_vault_encrypt_ceph_keys ``` - - - - ::: -2. Get ceph keys. This places the necessary keys in `/opt/configuration`. +3. Add the keys permanently to the repository ``` - osism apply copy-ceph-keys + git add **/ceph.*.keyring + git commit -m "Add the downloaded Keyes to the repository" ``` - After run, these keys must be permanently added to the configuration repository - via Git. - + Here is an overview of the individual keys: ``` environments/infrastructure/files/ceph/ceph.client.admin.keyring environments/kolla/files/overlays/gnocchi/ceph.client.gnocchi.keyring @@ -113,6 +129,8 @@ At least the [preparation](../../configuration-guide/ceph.md#osd-devices) of the environments/kolla/files/overlays/glance/ceph.client.glance.keyring ``` + :::info + If the `osism apply copy-ceph-keys` fails because the keys are not found in the `/share` directory, this can be ignored. The keys of the predefined keys (e.g. for Manila) were then not created as they are not used. If you only use Ceph and do not need the predefined @@ -122,19 +140,22 @@ At least the [preparation](../../configuration-guide/ceph.md#osd-devices) of the ```yaml title="environments/ceph/configuration.yml" ceph_kolla_keys: [] ``` + ::: -3. After the Ceph keys have been persisted in the configuration repository, the Ceph +2. After the Ceph keys have been persisted in the configuration repository, the Ceph client can be deployed. ``` osism apply cephclient ``` -4. Enable and prepare the use of the Ceph dashboard. +## Enable Ceph Dashboard - ``` - osism apply ceph-bootstrap-dashboard - ``` +Enable and prepare the use of the Ceph dashboard. + +``` +osism apply ceph-bootstrap-dashboard +``` ## RGW service From 2687514cbe18112d202c83567cc3985feede5300 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Tue, 25 Jun 2024 12:03:01 +0200 Subject: [PATCH 08/28] add imports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index 1c885751f6..ef56b9f244 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -3,6 +3,9 @@ sidebar_label: Ceph sidebar_position: 30 --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem + # Ceph The official Ceph documentation is located on https://docs.ceph.com/en/latest/rados/configuration/ From ac7d65466d69b5363dcf4feda0cee8d36ce51bee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Tue, 25 Jun 2024 12:06:05 +0200 Subject: [PATCH 09/28] fix import MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index ef56b9f244..eab0e0da49 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -4,7 +4,7 @@ sidebar_position: 30 --- import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem +import TabItem from '@theme/TabItem'; # Ceph From aa455715939ca4c23d632b041707d3e2e1b9adb6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Tue, 25 Jun 2024 12:11:56 +0200 Subject: [PATCH 10/28] add distinct labels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index eab0e0da49..fada30f2e1 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -269,7 +269,7 @@ ceph_osd_devices: ``` - + The `nvme0n1` device will be used as an source for WAL and DB device volumes. With the configured values the provisioning mechanism creates 6 logical DB volumes of 30GB and 6 logical WAL volumes of 2B size each @@ -301,7 +301,7 @@ In the example shown here, both the data structures for the RocksDB and for the - + The `nvme0n1` device will be used as an source for WAL and `nvme1n1` for DB device volumes. With the configured values the provisioning mechanism creates 6 logical volumes of 30 GB and 6 logical WAL volumes of 2B size each. From 20b01fc795528fade593441291c30aef34e19ec4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Tue, 25 Jun 2024 12:20:37 +0200 Subject: [PATCH 11/28] remove change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/deploy-guide/services/network.md | 29 ++++++++++---------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/guides/deploy-guide/services/network.md b/docs/guides/deploy-guide/services/network.md index e503ca5c8f..c397f7de16 100644 --- a/docs/guides/deploy-guide/services/network.md +++ b/docs/guides/deploy-guide/services/network.md @@ -5,20 +5,6 @@ sidebar_position: 15 # Network -Two options are available for installing the Openstack network. - -* Open Virtual Network with Open vSwitch: - * the default, recommended - * configuration value `ovn` -* Open vSwitch: - * configuration value `openvswitch` - -The selected option is configured in the following file: -```yaml title="environments/kolla/configuration.yml" -# neutron -neutron_plugin_agent: "ovn" -``` - 1. Open vSwitch (OVS) ``` @@ -26,7 +12,20 @@ neutron_plugin_agent: "ovn" osism apply openvswitch ``` -2. Optional: Open Virtual Network (OVN) +2. Open Virtual Network (OVN) + + In `environments/kolla/configuration.yml` the parameter `neutron_plugin_agent` is set to + `ovn` if OVN is used as a network plugin. The parameter is set to `ovn` by default in the + Cookiecutter. + + ```yaml title="environments/kolla/configuration.yml" + # neutron + neutron_plugin_agent: "ovn" + ``` + + Otherwise the network plugin is set to `openvswitch`. + If the `neutron_plugin_agent` is set to `openvswitch`, this step does not need to be done. + Before the deployment of OVN, the deployment of Open vSwitch must already have been done. ``` From d49dc6e03c24e8dfe7431c3aa827b0affc2abc30 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Wed, 26 Jun 2024 11:28:29 +0200 Subject: [PATCH 12/28] add general hints MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/index.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/guides/upgrade-guide/index.md b/docs/guides/upgrade-guide/index.md index d59c023135..c69780a2d1 100644 --- a/docs/guides/upgrade-guide/index.md +++ b/docs/guides/upgrade-guide/index.md @@ -5,7 +5,32 @@ sidebar_position: 20 # Upgrade Guide +## General hints + +TODO: Testumgebung, Inkubation, Runbook, Personal, ... + In the examples, the pull of images (if supported by a role) is always run first. While this is optional, it is recommended to speed up the execution of the upgrade action in the second step. This significantly reduces the times required for the restart from a service. + + +## The order of the upgrade steps + + +:::warning + +Always read the [release notes](https://osism.tech/docs/release-notes/) first to learn what has changed and what +adjustments are necessary. Read the release notes even if you are only updating for minor releases e.g. from 7.0.2 to 7.0.5. + +::: + +TODO: validate order + +* Step 1: [Upgrade the Manager](./manager) +* Step 2: [Upgrade Logging & Monitoring](./logging-monitoring) +* Step 3: [Upgrade the Infrastructure](./infrastructure) +* Step 4: [Upgrade the Ceph](./ceph) +* Step 5: [Upgrade the Network](./network) +* Step 6: [Upgrade Docker](./docker) +* Step 7: [Upgrade Openstack](./openstack) From 38614bd1a4e7ba3b0f6ba88f52c737ae1cf9de0b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Wed, 26 Jun 2024 12:49:54 +0200 Subject: [PATCH 13/28] improve upgrade MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/manager.mdx | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/guides/upgrade-guide/manager.mdx b/docs/guides/upgrade-guide/manager.mdx index f34707e88f..cb5faa2352 100644 --- a/docs/guides/upgrade-guide/manager.mdx +++ b/docs/guides/upgrade-guide/manager.mdx @@ -24,7 +24,7 @@ In the example, OSISM release 7.0.5 is used. ``` MANAGER_VERSION=7.0.5 - sed -i -e "s/manager_version: .*/manager_version: ${MANAGER_VERSION}/g" environments/manager/configuration.yml + sed -i -e "s/manager_version: .*/manager_version: ${MANAGER_VERSION:?}/g" environments/manager/configuration.yml ``` 1.2. If `openstack_version` or `ceph_version` are set in `environments/manager/configuration.yml` @@ -65,13 +65,15 @@ In the example, OSISM release 7.0.5 is used. workflows for changes to the configuration repository, only a generic example for Git. ``` - git commit -a -s -m "manager: use OSISM version 7.0.5" + git commit -a -s -m "manager: use OSISM version ${MANAGER_VERSION:?}" git push ``` 2. Update the configuration repository on the manager node. ``` + git pull + git checkout osism apply configuration ``` @@ -104,3 +106,8 @@ In the example, OSISM release 7.0.5 is used. ``` osism set vault password ``` +7. Optional: Tag the upgrade in Git + ``` + git tag ${MANAGER_VERSION:?} + git push --tags + ``` From 9eb34c7228a8f31b5be472caa4e4f24916af6e21 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Wed, 26 Jun 2024 13:31:15 +0200 Subject: [PATCH 14/28] Improve Logging upgrade MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/logging-monitoring.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/guides/upgrade-guide/logging-monitoring.md b/docs/guides/upgrade-guide/logging-monitoring.md index c58148b048..bd6c5713a1 100644 --- a/docs/guides/upgrade-guide/logging-monitoring.md +++ b/docs/guides/upgrade-guide/logging-monitoring.md @@ -5,25 +5,30 @@ sidebar_position: 40 # Logging & Monitoring -1. OpenSearch +1. **Optional:** Pull containers + + ``` + osism apply -a pull opensearch + osism apply -a pull prometheus + osism apply -a pull grafana + ``` + +2. OpenSearch OpenSearch dashboards is also upgraded with the `opensearch` role. ``` - osism apply -a pull opensearch osism apply -a upgrade opensearch ``` -2. Prometheus +3. Prometheus ``` - osism apply -a pull prometheus osism apply prometheus ``` -3. Grafana +4. Grafana ``` - osism apply -a pull grafana osism apply -a upgrade grafana ``` From ea99b5f4e7079da8e0bb62f32b70de5728e030e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Wed, 26 Jun 2024 18:41:05 +0200 Subject: [PATCH 15/28] improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/infrastructure.md | 34 ++++++++++++--------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/docs/guides/upgrade-guide/infrastructure.md b/docs/guides/upgrade-guide/infrastructure.md index 08ee85805c..25ace0c7a1 100644 --- a/docs/guides/upgrade-guide/infrastructure.md +++ b/docs/guides/upgrade-guide/infrastructure.md @@ -5,7 +5,7 @@ sidebar_position: 30 # Infrastructure -1. Kubernetes +1. **Optional:** Kubernetes This is only necessary if the internal Kubernetes cluster has also been deployed. This can be checked by executing `kubectl get nodes` on the manager node. @@ -14,51 +14,57 @@ sidebar_position: 30 osism apply k3s-upgrade ``` -2. Cron, Fluentd & Kolla Toolbox +2. **Optional:** Pull containers + + ``` + osism apply -a pull common + osism apply -a pull loadbalancer + osism apply -a pull redis + osism apply -a pull memcached + osism apply -a pull memcached + osism apply -a pull rabbitmq + osism apply -a pull mariadb + ``` + +3. Cron, Fluentd & Kolla Toolbox The common role of Kolla is used to manage the services `cron`, `fluentd` and `kolla-toolbox`. - It is important to do this upgrade before any other upgrades in the Kolla + It is important to do this upgrade **before any other upgrades** in the Kolla environment, as parts of the other upgrades depend on the `kolla-toolbox` service. ``` - osism apply -a pull common osism apply -a upgrade common ``` -3. Loadbalancer +4. Loadbalancer ``` - osism apply -a pull loadbalancer osism apply -a upgrade loadbalancer ``` -4. Redis +5. Redis ``` - osism apply -a pull redis osism apply -a upgrade redis ``` -5. Memcached +6. Memcached ``` - osism apply -a pull memcached osism apply -a upgrade memcached ``` -6. RabbitMQ +7. RabbitMQ ``` - osism apply -a pull rabbitmq osism apply -a upgrade rabbitmq ``` -7. MariaDB +8. MariaDB ``` - osism apply -a pull mariadb osism apply -a upgrade mariadb ``` From 8fff4172d737b3968ee82612bad47e6a96abc9f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 27 Jun 2024 10:13:22 +0200 Subject: [PATCH 16/28] improve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/guides/upgrade-guide/index.md b/docs/guides/upgrade-guide/index.md index c69780a2d1..631f8c334e 100644 --- a/docs/guides/upgrade-guide/index.md +++ b/docs/guides/upgrade-guide/index.md @@ -30,7 +30,7 @@ TODO: validate order * Step 1: [Upgrade the Manager](./manager) * Step 2: [Upgrade Logging & Monitoring](./logging-monitoring) * Step 3: [Upgrade the Infrastructure](./infrastructure) -* Step 4: [Upgrade the Ceph](./ceph) -* Step 5: [Upgrade the Network](./network) -* Step 6: [Upgrade Docker](./docker) -* Step 7: [Upgrade Openstack](./openstack) +* Step 4: [Upgrade the Network](./network) +* Step 5: [Upgrade Docker](./docker) +* Step 6: [Upgrade Openstack](./openstack) +* Step 7: [Upgrade the Ceph](./ceph) From d995afbd62387bf2c721d6b114c264d74fc0320a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 27 Jun 2024 10:41:56 +0200 Subject: [PATCH 17/28] restructure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/openstack.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/guides/upgrade-guide/openstack.md b/docs/guides/upgrade-guide/openstack.md index 6505bc2755..ebe888e4e5 100644 --- a/docs/guides/upgrade-guide/openstack.md +++ b/docs/guides/upgrade-guide/openstack.md @@ -13,6 +13,20 @@ of the APIs. This downtime is usually less than 1 minute. ::: +1. Pull containers + + ``` + osism apply -a pull keystone + osism apply -a pull glance + osism apply -a pull designate + osism apply -a pull placement + osism apply -a pull cinder + osism apply -a pull neutron + osism apply -a pull nova + osism apply -a pull octavia + osism apply -a pull horizon + ``` + 1. OpenStack client ``` @@ -22,56 +36,48 @@ of the APIs. This downtime is usually less than 1 minute. 2. Keystone ``` - osism apply -a pull keystone osism apply -a upgrade keystone ``` 3. Glance ``` - osism apply -a pull glance osism apply -a upgrade glance ``` 4. Designate ``` - osism apply -a pull designate osism apply -a upgrade designate ``` 5. Placement ``` - osism apply -a pull placement osism apply -a upgrade placement ``` 6. Cinder ``` - osism apply -a pull cinder osism apply -a upgrade cinder ``` 7. Neutron ``` - osism apply -a pull neutron osism apply -a upgrade neutron ``` 8. Nova ``` - osism apply -a pull nova osism apply -a upgrade nova ``` 9. Octavia ``` - osism apply -a pull octavia osism apply -a upgrade octavia ``` @@ -97,6 +103,5 @@ of the APIs. This downtime is usually less than 1 minute. 10. Horizon ``` - osism apply -a pull horizon osism apply -a upgrade horizon ``` From 32a8da49439218c5e8f246170bad6babfdb38ed3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 27 Jun 2024 15:25:33 +0200 Subject: [PATCH 18/28] typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index fada30f2e1..956f65a60d 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -39,7 +39,7 @@ advise which ip adress should be used to reach the monitor instances. The `client.admin` keyring is placed in the file `environments/infrastructure/files/ceph/ceph.client.admin.keyring`. -## Systemctl Parameters, Swappiness and Friends +## SysCtl Parameters, Swappiness and Friends The swappiness is set via the `os_tuning_params` dictionary. The dictionary can only be completely overwritten via an entry in the file `environments/ceph/configuration.yml`. From ecc430192dd2ea4780805588c537b9d4055e4047 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 27 Jun 2024 15:27:20 +0200 Subject: [PATCH 19/28] remove TODO MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/guides/upgrade-guide/index.md b/docs/guides/upgrade-guide/index.md index 631f8c334e..f0e1a09014 100644 --- a/docs/guides/upgrade-guide/index.md +++ b/docs/guides/upgrade-guide/index.md @@ -25,8 +25,6 @@ adjustments are necessary. Read the release notes even if you are only updating ::: -TODO: validate order - * Step 1: [Upgrade the Manager](./manager) * Step 2: [Upgrade Logging & Monitoring](./logging-monitoring) * Step 3: [Upgrade the Infrastructure](./infrastructure) From ab79ce15270ecf8f0e4a2a0b3a14e9da8ce337de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 27 Jun 2024 15:51:22 +0200 Subject: [PATCH 20/28] add some explanantion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/index.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/guides/upgrade-guide/index.md b/docs/guides/upgrade-guide/index.md index f0e1a09014..fe6daf289c 100644 --- a/docs/guides/upgrade-guide/index.md +++ b/docs/guides/upgrade-guide/index.md @@ -5,16 +5,36 @@ sidebar_position: 20 # Upgrade Guide -## General hints +## A dedicated test environment -TODO: Testumgebung, Inkubation, Runbook, Personal, ... +Using a dedicated test environment to upgrade the complex technical system landscape +consististing of Kubernetes, Ceph and Openstack is benenfical for some reasons: + +1. **Developing and Testing Upgrade Procedures**: + - **Validation of Runbooks**: Allows you to develop, validate and refine upgrade procedures documented in runbooks, ensuring accuracy and completeness. + - **Procedure Testing**: Ensures that each step of the upgrade process is thoroughly tested specific for your conditions, reducing the risk of errors during the actual upgrade. + +2. **Testing New Technical Changes**: + - **Risk-Free Testing**: Provides a safe space to implement and test new technical changes without impacting the production environment. + - **Issue Identification**: Helps identify and resolve potential issues with new features or configurations before they go live. + +3. **Practicing Critical Steps for Training**: + - **Staff Training**: Enables IT staff to practice critical steps and gain hands-on experience with the upgrade process, enhancing their preparedness. + - **Engineer Familiarization**: Allows new engineers to become familiar with the system itself, features and workflows, reducing the learning curve. + +4. **Developing New Concepts**: + - **Concept Testing**: Facilitates the development and testing of new concepts, ensuring they are viable and effective before implementation in production. + - **Innovation Space**: Provides a controlled environment to experiment with innovative ideas and configurations without disrupting existing services. + +Overall, a dedicated test environment ensures a smoother, safer upgrade and operation process, minimizes risks, and enhances staff training and innovation. + +## General hints for the Upgrade documentation In the examples, the pull of images (if supported by a role) is always run first. While this is optional, it is recommended to speed up the execution of the upgrade action in the second step. This significantly reduces the times required for the restart from a service. - ## The order of the upgrade steps From daa79127afe113abd214de041b7714bac22729ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Thu, 27 Jun 2024 16:05:29 +0200 Subject: [PATCH 21/28] remove todo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/configuration-guide/ceph.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/guides/configuration-guide/ceph.md b/docs/guides/configuration-guide/ceph.md index 956f65a60d..22c23c3590 100644 --- a/docs/guides/configuration-guide/ceph.md +++ b/docs/guides/configuration-guide/ceph.md @@ -166,6 +166,10 @@ pools are to be created is `ceph.rbd`, then the parameters would be stored in | `openstack_pool_default_pg_num` | 64 | | `openstack_pool_default_min_size` | 0 | +It is important to set up the [placement groups](https://docs.ceph.com/en/nautilus/rados/operations/placement-groups/) appropriately so that +Ceph does not fall short of its potential in terms of performance, for example. +Dabei können Werkzeuge wie [PG Calc ](https://docs.ceph.com/en/latest/rados/operations/pgcalc/) helfen. + ## OSD devices For more advanced OSD layout requirements leave out the `devices` key @@ -174,7 +178,6 @@ and instead use `lvm_volumes`. Details for this can be found on the In order to aid in creating the `lvm_volumes` config entries and provision the LVM devices for them, OSISM has the two playbooks `ceph-configure-lvm-volumes` and `ceph-create-lvm-devices` available. -TODO: add reference to https://docs.ceph.com/en/latest/rados/operations/pgcalc/ and PG autocaler dryrun ### Configure the device layout From 09996004c6302729500200067cfdcefcd3d164c7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 13:00:30 +0200 Subject: [PATCH 22/28] add upgrade documentation for kubernetes, from release letter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/index.md | 13 +++++++------ docs/guides/upgrade-guide/kubernetes.md | 11 +++++++++++ 2 files changed, 18 insertions(+), 6 deletions(-) create mode 100644 docs/guides/upgrade-guide/kubernetes.md diff --git a/docs/guides/upgrade-guide/index.md b/docs/guides/upgrade-guide/index.md index fe6daf289c..8448db97f8 100644 --- a/docs/guides/upgrade-guide/index.md +++ b/docs/guides/upgrade-guide/index.md @@ -46,9 +46,10 @@ adjustments are necessary. Read the release notes even if you are only updating ::: * Step 1: [Upgrade the Manager](./manager) -* Step 2: [Upgrade Logging & Monitoring](./logging-monitoring) -* Step 3: [Upgrade the Infrastructure](./infrastructure) -* Step 4: [Upgrade the Network](./network) -* Step 5: [Upgrade Docker](./docker) -* Step 6: [Upgrade Openstack](./openstack) -* Step 7: [Upgrade the Ceph](./ceph) +* Step 2: [Upgrade Kubernetes](./kubernetes) +* Step 3: [Upgrade Logging & Monitoring](./logging-monitoring) +* Step 4: [Upgrade the Infrastructure](./infrastructure) +* Step 5: [Upgrade the Network](./network) +* Step 6: [Upgrade Docker](./docker) +* Step 7: [Upgrade Openstack](./openstack) +* Step 8: [Upgrade the Ceph](./ceph) diff --git a/docs/guides/upgrade-guide/kubernetes.md b/docs/guides/upgrade-guide/kubernetes.md new file mode 100644 index 0000000000..ca49ba447a --- /dev/null +++ b/docs/guides/upgrade-guide/kubernetes.md @@ -0,0 +1,11 @@ +--- +sidebar_label: Kubernetes +sidebar_position: 20 +--- + +# Kubernetes + +``` +osism apply k3s-upgrade +``` + From 08817ca0a3f8c0a1332b0161e5bd8d686ce433bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 13:06:36 +0200 Subject: [PATCH 23/28] add hint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/kubernetes.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/guides/upgrade-guide/kubernetes.md b/docs/guides/upgrade-guide/kubernetes.md index ca49ba447a..d8bef1d75c 100644 --- a/docs/guides/upgrade-guide/kubernetes.md +++ b/docs/guides/upgrade-guide/kubernetes.md @@ -5,6 +5,9 @@ sidebar_position: 20 # Kubernetes +This is an optional procedure. +This step is only neccessary if you have enabled the k3s cluster with `enable_osism_kubernetes: yes`. + ``` osism apply k3s-upgrade ``` From bd6a86ad73359acd633fe0fadcb9e49df94247a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 13:12:24 +0200 Subject: [PATCH 24/28] update version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/manager.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/upgrade-guide/manager.mdx b/docs/guides/upgrade-guide/manager.mdx index cb5faa2352..b0923fc796 100644 --- a/docs/guides/upgrade-guide/manager.mdx +++ b/docs/guides/upgrade-guide/manager.mdx @@ -23,7 +23,7 @@ In the example, OSISM release 7.0.5 is used. 1.1. Set the new OSISM version in the configuration repository. ``` - MANAGER_VERSION=7.0.5 + MANAGER_VERSION="7.1.2" sed -i -e "s/manager_version: .*/manager_version: ${MANAGER_VERSION:?}/g" environments/manager/configuration.yml ``` From 39a3e088eb586a88981adcd172459f91b0933882 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 13:12:55 +0200 Subject: [PATCH 25/28] update version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/manager.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/upgrade-guide/manager.mdx b/docs/guides/upgrade-guide/manager.mdx index b0923fc796..cf3ad2ce1c 100644 --- a/docs/guides/upgrade-guide/manager.mdx +++ b/docs/guides/upgrade-guide/manager.mdx @@ -11,12 +11,12 @@ import TabItem from '@theme/TabItem'; :::warning Always read the [release notes](https://osism.tech/docs/release-notes/) first to learn what has changed and what -adjustments are necessary. Read the release notes even if you are only updating from e.g. 7.0.2 to 7.0.5. +adjustments are necessary. Read the release notes even if you are only updating from e.g. 7.0.2 to 7.1.2. ::: The update of a manager service with a stable release of OSISM is described here. -In the example, OSISM release 7.0.5 is used. +In the example, OSISM release 7.1.2 is used. 1. Change the OSISM release in the configuration repository. From d9d683f664d8034e43437bd7ec3998a6c92674ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 13:33:57 +0200 Subject: [PATCH 26/28] Improve a bit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/manager.mdx | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/guides/upgrade-guide/manager.mdx b/docs/guides/upgrade-guide/manager.mdx index cf3ad2ce1c..947d8f022a 100644 --- a/docs/guides/upgrade-guide/manager.mdx +++ b/docs/guides/upgrade-guide/manager.mdx @@ -40,6 +40,7 @@ In the example, OSISM release 7.1.2 is used. make sync ``` + If Gilt is not installed via the `requirements.txt` of the manager environment it is important to use a version smaller v2. The v2 of Gilt is not yet usable. @@ -69,7 +70,12 @@ In the example, OSISM release 7.1.2 is used. git push ``` -2. Update the configuration repository on the manager node. +2. Login to the manager using the "dragon" user + ``` + ssh dragon@ + ``` + +3. Update the configuration repository on the manager node. ``` git pull @@ -77,7 +83,7 @@ In the example, OSISM release 7.1.2 is used. osism apply configuration ``` -3. Update the manager service on the manager node. +4. Update the manager service on the manager node. ``` osism update manager @@ -88,26 +94,27 @@ In the example, OSISM release 7.1.2 is used. * If `osism update manager` does not work yet, use `osism-update-manager` instead. -4. Refresh the facts cache. +5. Refresh the facts cache. ``` osism apply facts ``` -5. If Traefik is used on the manager node (`traefik_enable: true` in `environments/infrastructure/configuration.yml`) - then Traefik should also be upgraded. +6. Finally, the Ansible vault password must be made known again. ``` - osism apply traefik + osism set vault password ``` -6. Finally, the Ansible vault password must be made known again. +7. If Traefik is used on the manager node (`traefik_enable: true` in `environments/infrastructure/configuration.yml`) + then Traefik should also be upgraded. ``` - osism set vault password + osism apply traefik ``` -7. Optional: Tag the upgrade in Git + +8. Optional: Tag the upgrade in Git ``` - git tag ${MANAGER_VERSION:?} + git tag "$(grep "manager_version:" environments/manager/configuration.yml|awk '{print $2}')" git push --tags ``` From d7f5447a4948c297b12b7c981c73e0312b1b2b4f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 14:10:00 +0200 Subject: [PATCH 27/28] Move K8s Upgrade to dedicated file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/infrastructure.md | 23 +++++++-------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/docs/guides/upgrade-guide/infrastructure.md b/docs/guides/upgrade-guide/infrastructure.md index 25ace0c7a1..34bf5bf601 100644 --- a/docs/guides/upgrade-guide/infrastructure.md +++ b/docs/guides/upgrade-guide/infrastructure.md @@ -5,16 +5,7 @@ sidebar_position: 30 # Infrastructure -1. **Optional:** Kubernetes - - This is only necessary if the internal Kubernetes cluster has also been deployed. - This can be checked by executing `kubectl get nodes` on the manager node. - - ``` - osism apply k3s-upgrade - ``` - -2. **Optional:** Pull containers +1. **Optional:** Pull containers ``` osism apply -a pull common @@ -26,7 +17,7 @@ sidebar_position: 30 osism apply -a pull mariadb ``` -3. Cron, Fluentd & Kolla Toolbox +2. Cron, Fluentd & Kolla Toolbox The common role of Kolla is used to manage the services `cron`, `fluentd` and `kolla-toolbox`. @@ -39,31 +30,31 @@ sidebar_position: 30 osism apply -a upgrade common ``` -4. Loadbalancer +3. Loadbalancer ``` osism apply -a upgrade loadbalancer ``` -5. Redis +4. Redis ``` osism apply -a upgrade redis ``` -6. Memcached +5. Memcached ``` osism apply -a upgrade memcached ``` -7. RabbitMQ +6. RabbitMQ ``` osism apply -a upgrade rabbitmq ``` -8. MariaDB +7. MariaDB ``` osism apply -a upgrade mariadb From cdaceb42f5eae7710c70d5ea9c3799c8817dfcc9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc=20Sch=C3=B6chlin?= Date: Fri, 6 Sep 2024 16:35:52 +0200 Subject: [PATCH 28/28] improve order MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marc Schöchlin --- docs/guides/upgrade-guide/ceph.md | 2 +- docs/guides/upgrade-guide/docker.md | 22 +++---------------- docs/guides/upgrade-guide/index.md | 15 +++++++------ docs/guides/upgrade-guide/infrastructure.md | 2 +- docs/guides/upgrade-guide/kubernetes.md | 2 +- .../upgrade-guide/logging-monitoring.md | 2 +- docs/guides/upgrade-guide/network.md | 2 +- docs/guides/upgrade-guide/openstack.md | 2 +- 8 files changed, 17 insertions(+), 32 deletions(-) diff --git a/docs/guides/upgrade-guide/ceph.md b/docs/guides/upgrade-guide/ceph.md index dd5236366e..9e06a952f9 100644 --- a/docs/guides/upgrade-guide/ceph.md +++ b/docs/guides/upgrade-guide/ceph.md @@ -1,6 +1,6 @@ --- sidebar_label: Ceph -sidebar_position: 20 +sidebar_position: 30 --- # Ceph diff --git a/docs/guides/upgrade-guide/docker.md b/docs/guides/upgrade-guide/docker.md index d2ea040114..533efb17a4 100644 --- a/docs/guides/upgrade-guide/docker.md +++ b/docs/guides/upgrade-guide/docker.md @@ -1,6 +1,6 @@ --- sidebar_label: Docker -sidebar_position: 20 +sidebar_position: 40 --- # Docker @@ -18,28 +18,12 @@ All installable versions can be displayed with `apt-cache madison docker-ce`. $ apt-cache madison docker-ce docker-ce | 5:24.0.6-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:24.0.5-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:24.0.4-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:24.0.3-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:24.0.2-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:24.0.1-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:24.0.0-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:23.0.6-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:23.0.5-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:23.0.4-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:23.0.3-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:23.0.2-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages + ... docker-ce | 5:23.0.1-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:23.0.0-1~ubuntu.22.04~jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:20.10.24~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:20.10.23~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.22~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.21~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.20~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.19~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.18~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.17~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.16~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages - docker-ce | 5:20.10.15~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages + ... docker-ce | 5:20.10.14~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages docker-ce | 5:20.10.13~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages ``` diff --git a/docs/guides/upgrade-guide/index.md b/docs/guides/upgrade-guide/index.md index 8448db97f8..e3e4f0bdb2 100644 --- a/docs/guides/upgrade-guide/index.md +++ b/docs/guides/upgrade-guide/index.md @@ -46,10 +46,11 @@ adjustments are necessary. Read the release notes even if you are only updating ::: * Step 1: [Upgrade the Manager](./manager) -* Step 2: [Upgrade Kubernetes](./kubernetes) -* Step 3: [Upgrade Logging & Monitoring](./logging-monitoring) -* Step 4: [Upgrade the Infrastructure](./infrastructure) -* Step 5: [Upgrade the Network](./network) -* Step 6: [Upgrade Docker](./docker) -* Step 7: [Upgrade Openstack](./openstack) -* Step 8: [Upgrade the Ceph](./ceph) +* Step 2: [Upgrade the Network](./network) +* Step 3: [Upgrade the Ceph](./ceph) +* Step 4: [Upgrade Docker](./docker) +* Step 5: [Upgrade Kubernetes](./kubernetes) +* Step 6: [Upgrade Logging & Monitoring](./logging-monitoring) +* Step 7: [Upgrade the Infrastructure](./infrastructure) +* Step 8: [Upgrade Openstack](./openstack) + diff --git a/docs/guides/upgrade-guide/infrastructure.md b/docs/guides/upgrade-guide/infrastructure.md index 34bf5bf601..164377c644 100644 --- a/docs/guides/upgrade-guide/infrastructure.md +++ b/docs/guides/upgrade-guide/infrastructure.md @@ -1,6 +1,6 @@ --- sidebar_label: Infrastructure -sidebar_position: 30 +sidebar_position: 70 --- # Infrastructure diff --git a/docs/guides/upgrade-guide/kubernetes.md b/docs/guides/upgrade-guide/kubernetes.md index d8bef1d75c..5b45da2e93 100644 --- a/docs/guides/upgrade-guide/kubernetes.md +++ b/docs/guides/upgrade-guide/kubernetes.md @@ -1,6 +1,6 @@ --- sidebar_label: Kubernetes -sidebar_position: 20 +sidebar_position: 50 --- # Kubernetes diff --git a/docs/guides/upgrade-guide/logging-monitoring.md b/docs/guides/upgrade-guide/logging-monitoring.md index bd6c5713a1..fd0250b197 100644 --- a/docs/guides/upgrade-guide/logging-monitoring.md +++ b/docs/guides/upgrade-guide/logging-monitoring.md @@ -1,6 +1,6 @@ --- sidebar_label: Logging & Monitoring -sidebar_position: 40 +sidebar_position: 60 --- # Logging & Monitoring diff --git a/docs/guides/upgrade-guide/network.md b/docs/guides/upgrade-guide/network.md index 83fcf56bb1..e0ca6ff315 100644 --- a/docs/guides/upgrade-guide/network.md +++ b/docs/guides/upgrade-guide/network.md @@ -1,6 +1,6 @@ --- sidebar_label: Network -sidebar_position: 15 +sidebar_position: 20 --- # Network diff --git a/docs/guides/upgrade-guide/openstack.md b/docs/guides/upgrade-guide/openstack.md index ebe888e4e5..1570eb9268 100644 --- a/docs/guides/upgrade-guide/openstack.md +++ b/docs/guides/upgrade-guide/openstack.md @@ -1,6 +1,6 @@ --- sidebar_label: OpenStack -sidebar_position: 40 +sidebar_position: 80 --- # OpenStack