Nova services adoption (no extra cell, no ffu)

Note about remapping cell DB names from OSP cells naming scheme to the NG scheme with the superconductor layout. Add a step to rename default cell as cell1, and to delete stale Nova services records from cell1 DB during initial databases import, to properly transition it into a superconductor layout later on. Adjust minor gaps in the Placement adoption docs Address the switch for service overrides spec instead of externalEndpoints, where it is missing on the path to Nova adotpion. Remove Nova Metadata secret creation workarounds from the EDPM adotopion docs and test suits. Provide workaround for renaming 'default' cell's DB during adoption. Add test suits for Nova CP services adoption. Add missing nova and libvirt services for the edpm adoption tests. Verify no dataplane disruptions during the adoption process. Verify Nova services still control pre-created VM workload after adotpion is done. Update and fix the composition of the Nova related services pre-checks to execute it before stopping Nova/DB services on the source env. Stop Nova services in stop_openstack_services instead of edpm_adoption (that was too late to do that). Get services topology specific configuration in pull_openstack_configuration. Add missing role for that as well. Signed-off-by: Bohdan Dobrelia <[email protected]>
openstack-k8s-operators · Nov 6, 2023 · f2a7a19 · f2a7a19
1 parent aeb0add
commit f2a7a19
Show file tree

Hide file tree

Showing 18 changed files with 504 additions and 119 deletions.
diff --git a/docs/README.md b/docs/README.md
@@ -29,6 +29,8 @@ Perform the actions from the sub-documents in the following order:
 
 * [Placement adoption](openstack/placement_adoption.md)
 
+* [Nova adoption](openstack/nova_adoption.md)
+
 * [Cinder adoption](openstack/cinder_adoption.md)
 
 * [Horizon adoption](openstack/horizon_adoption.md)

diff --git a/docs/openstack/edpm_adoption.md b/docs/openstack/edpm_adoption.md
@@ -95,55 +95,10 @@ EOF
   $(cat ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa | base64 | sed 's/^/        /')
   EOF
   ```
-* Create the Nova Metadata secret (Workaround while nova isn't adopted yet):
-
-  ```bash
-  oc apply -f - <<EOF
-  apiVersion: v1
-  kind: Secret
-  metadata:
-      name: nova-metadata-neutron-config
-  data:
-      05-nova-metadata.conf: |
-  $(echo "[DEFAULT]\nnova_metadata_host = 1.2.3.4\nnova_metadata_port = 8775\nnova_metadata_protocol = http\nmetadata_proxy_shared_secret = 1234567842\n" | base64 | sed 's/^/        /')
-  EOF
-  ```
-
-* Stop the nova services.
-
-```bash
-
-# Update the services list to be stopped
-
-ServicesToStop=("tripleo_nova_api_cron.service"
-                "tripleo_nova_api.service"
-                "tripleo_nova_compute.service"
-                "tripleo_nova_conductor.service"
-                "tripleo_nova_libvirt.target"
-                "tripleo_nova_metadata.service"
-                "tripleo_nova_migration_target.service"
-                "tripleo_nova_scheduler.service"
-                "tripleo_nova_virtlogd_wrapper.service"
-                "tripleo_nova_virtnodedevd.service"
-                "tripleo_nova_virtproxyd.service"
-                "tripleo_nova_virtqemud.service"
-                "tripleo_nova_virtsecretd.service"
-                "tripleo_nova_virtstoraged.service"
-                "tripleo_nova_vnc_proxy.service")
-
-echo "Stopping nova services"
-
-for service in ${ServicesToStop[*]}; do
-    echo "Stopping the $service in each controller node"
-    $CONTROLLER1_SSH sudo systemctl stop $service
-    $CONTROLLER2_SSH sudo systemctl stop $service
-    $CONTROLLER3_SSH sudo systemctl stop $service
-done
-```
 
 * Deploy OpenStackDataPlaneNodeSet:
 
-  ```
+  ```yaml
   oc apply -f - <<EOF
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneNodeSet
@@ -160,6 +115,8 @@ done
       - install-os
       - configure-os
       - run-os
+      - libvirt
+      - nova
       - ovn
     env:
       - name: ANSIBLE_CALLBACKS_ENABLED
@@ -276,7 +233,7 @@ done
 
 * Deploy OpenStackDataPlaneDeployment:
 
-  ```
+  ```yaml
   oc apply -f - <<EOF
   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
@@ -302,6 +259,7 @@ done
     ```
 
 * Wait for the dataplane node set to reach the Ready status:
+
     ```
     oc wait --for condition=Ready osdpns/openstack --timeout=30m
     ```
diff --git a/docs/openstack/mariadb_copy.md b/docs/openstack/mariadb_copy.md
@@ -4,6 +4,11 @@ This document describes how to move the databases from the original
 OpenStack deployment to the MariaDB instances in the OpenShift
 cluster.
 
+> **NOTE** This example scenario describes a simple single-cell setup. Real
+> multi-stack topology recommended for production use results in different
+> cells DBs layout, and should be using different naming schemes (not covered
+> here this time).
+
 ## Prerequisites
 
 * Make sure the previous Adoption steps have been performed successfully.
@@ -13,7 +18,7 @@ cluster.
   * Podified MariaDB and RabbitMQ are running. No other podified
     control plane services are running.
 
-  * OpenStack services have been stopped
+  * OpenStack services have been [stopped](stop_openstack_services.md)
 
   * There must be network routability between:
 
@@ -34,7 +39,7 @@ cluster.
 Define the shell variables used in the steps below. The values are
 just illustrative, use values that are correct for your environment:
 
-```
+```bash
 MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
 
 PODIFIED_MARIADB_IP=$(oc get svc --selector "cr=mariadb-openstack" -ojsonpath='{.items[0].spec.clusterIP}')
@@ -57,21 +62,21 @@ COLLATION=utf8_general_ci
 
 * Test connection to the original DB (show databases):
 
-  ```
+  ```bash
   podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
       mysql -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" -e 'SHOW databases;'
   ```
 
 * Run mysqlcheck on the original DB to look for things that are not OK:
 
-  ```
+  ```bash
   podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
       mysqlcheck --all-databases -h $SOURCE_MARIADB_IP -u root "-p$SOURCE_DB_ROOT_PASSWORD" | grep -v OK
   ```
 
 * Test connection to podified DBs (show databases):
 
-  ```
+  ```bash
   oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
       mysql -h "$PODIFIED_MARIADB_IP" -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;'
   oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
@@ -80,17 +85,25 @@ COLLATION=utf8_general_ci
 
 ## Procedure - data copy
 
+> **NOTE**: We'll need to transition Nova services imported later on into a
+> superconductor architecture. For that, delete the old service records in
+> cells DBs, starting from the cell1. New records will be registered with
+> different hostnames provided by the Nova service operator. All Nova
+> services, except the compute agent, have no internal state, and its service
+> records can be safely deleted. Also we need to rename the former `default` cell
+> as `cell1`.
+
 * Create a temporary folder to store DB dumps and make sure it's the
   working directory for the following steps:
 
-  ```
+  ```bash
   mkdir ~/adoption-db
   cd ~/adoption-db
   ```
 
 * Create a dump of the original databases:
 
-  ```
+  ```bash
   podman run -i --rm --userns=keep-id -u $UID -v $PWD:$PWD:z,rw -w $PWD $MARIADB_IMAGE bash <<EOF
 
   # Note we do not want to dump the information and performance schema tables so we filter them
@@ -106,7 +119,7 @@ COLLATION=utf8_general_ci
 
 * Restore the databases from .sql files into the podified MariaDB:
 
-  ```
+  ```bash
   # db schemas to rename on import
   declare -A db_name_map
   db_name_map["nova"]="nova_cell1"
@@ -148,13 +161,17 @@ COLLATION=utf8_general_ci
       oc run ${container_name} --image ${MARIADB_IMAGE} -i --rm --restart=Never -- \
           mysql -h "${db_server}" -uroot "-p${db_password}" "${db_name}" < "${db_file}"
   done
+  oc exec -it mariadb-openstack -- mysql --user=root --password=${db_server_password_map["default"]} -e \
+      "update nova_api.cell_mappings set name='cell1' where name='default';"
+  oc exec -it mariadb-openstack-cell1 -- mysql --user=root --password=${db_server_password_map["default"]} -e \
+      "delete from nova_cell1.services where host not like '%nova-cell1-%' and services.binary != 'nova-compute';"
   ```
 
 ## Post-checks
 
 * Check that the databases were imported correctly:
 
-  ```
+  ```bash
   oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
   mysql -h "${PODIFIED_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
       | grep keystone

diff --git a/docs/openstack/nova_adoption.md b/docs/openstack/nova_adoption.md
@@ -0,0 +1,153 @@
+# Nova adoption
+
+> **NOTE** This example scenario describes a simple single-cell setup. Real
+> multi-stack topology recommended for production use results in different
+> cells DBs layout, and should be using different naming schemes (not covered
+> here this time).
+
+## Prerequisites
+
+* Previous Adoption steps completed. Notably,
+  * the [service databases](mariadb_copy.md)
+    must already be imported into the podified MariaDB;
+  * the [Keystone service](keystone_adoption.md) needs to be imported;
+  * the [Placement service](placement_adoption.md) needs to be imported;
+  * the [Glance service](glance_adoption.md) needs to be imported;
+  * the [OVN DB services](ovn_adoption.md) need to be imported;
+  * the [Neutron service](neutron_adoption.md) needs to be imported;
+  * Required services specific topology [configuration collected](pull_openstack_configuration.md#get-services-topology-specific-configuration);
+  * OpenStack services have been [stopped](stop_openstack_services.md)
+
+## Variables
+
+Define the shell variables and aliases used in the steps below. The values are
+just illustrative, use values that are correct for your environment:
+
+```bash
+alias openstack="oc exec -t openstackclient -- openstack"
+```
+
+## Procedure - Nova adoption
+
+> **NOTE**: We assume Nova Metadata deployed on the top level and not on each
+> cell level, so this example imports it the same way. If the source deployment
+> has a per cell metadata deployment, adjust the given below patch as needed.
+> Metadata service cannot be run in `cell0`.
+
+* Patch OpenStackControlPlane to deploy Nova:
+
+  ```yaml
+  oc patch openstackcontrolplane openstack -n openstack --type=merge --patch '
+  spec:
+    nova:
+      enabled: true
+      apiOverride:
+        route: {}
+      template:
+        secret: osp-secret
+        apiServiceTemplate:
+          override:
+            service:
+              internal:
+                metadata:
+                  annotations:
+                    metallb.universe.tf/address-pool: internalapi
+                    metallb.universe.tf/allow-shared-ip: internalapi
+                    metallb.universe.tf/loadBalancerIPs: 172.17.0.80
+                spec:
+                  type: LoadBalancer
+          customServiceConfig: |
+            [workarounds]
+            disable_compute_service_check_for_ffu=true
+        metadataServiceTemplate:
+          enabled: true # deploy single nova metadata on the top level
+          override:
+            service:
+              metadata:
+                annotations:
+                  metallb.universe.tf/address-pool: internalapi
+                  metallb.universe.tf/allow-shared-ip: internalapi
+                  metallb.universe.tf/loadBalancerIPs: 172.17.0.80
+              spec:
+                type: LoadBalancer
+          customServiceConfig: |
+            [workarounds]
+            disable_compute_service_check_for_ffu=true
+        schedulerServiceTemplate:
+          customServiceConfig: |
+            [workarounds]
+            disable_compute_service_check_for_ffu=true
+        cellTemplates:
+          cell0:
+            conductorServiceTemplate:
+              customServiceConfig: |
+                [workarounds]
+                disable_compute_service_check_for_ffu=true
+          cell1:
+            metadataServiceTemplate:
+              enabled: false # enable here to run it in a cell instead
+              override:
+                  service:
+                    metadata:
+                      annotations:
+                        metallb.universe.tf/address-pool: internalapi
+                        metallb.universe.tf/allow-shared-ip: internalapi
+                        metallb.universe.tf/loadBalancerIPs: 172.17.0.80
+                    spec:
+                      type: LoadBalancer
+              customServiceConfig: |
+                [workarounds]
+                disable_compute_service_check_for_ffu=true
+            conductorServiceTemplate:
+              customServiceConfig: |
+                [workarounds]
+                disable_compute_service_check_for_ffu=true
+  '
+  ```
+
+  The local Conductor services will be started for each cell, while the superconductor runs in `cell0`.
+  Note that ``disable_compute_service_check_for_ffu`` is mandatory for all imported Nova services, until
+  the [external dataplane imported](edpm_adoption.md), and until Nova Compute services fast-forward upgraded.
+
+## Post-checks
+
+* Check that Nova endpoints are defined and pointing to the
+  podified FQDNs and that Nova API responds.
+
+  ```bash
+  openstack endpoint list | grep nova
+  openstack server list
+  ```
+
+Compare the following outputs with the topology specific configuration
+[collected earlier](pull_openstack_configuration.md#get-services-topology-specific-configuration):
+
+* Query the superconductor for cell1 existance:
+
+  ```bash
+  oc rsh nova-cell0-conductor-0 nova-manage cell_v2 list_cells | grep -F '| cell1 |'
+  ```
+
+  The expected changes to happen:
+  * cell1's `nova` DB and user name become `nova_cell1`.
+  * Default cell is renamed to `cell1` (in a multi-cell setup, it should become indexed as the last cell instead).
+  * RabbitMQ transport URL no longer uses `guest`.
+
+* Verify no Nova compute dataplane disruptions during the adoption process:
+
+  ```bash
+  $CONTROLLER_SSH sudo podman exec -it libvirt_virtqemud virsh list --all | grep 'instance-00000001   running'
+  ```
+
+  * Verify if Nova services control the existing VM instance:
+
+  ```bash
+  openstack server list | grep -qF '| test | ACTIVE |' && openstack server stop test
+  openstack server list | grep -qF '| test | SHUTOFF |'
+  $CONTROLLER_SSH sudo podman exec -it libvirt_virtqemud virsh list --all | grep 'instance-00000001   shut off'
+
+  openstack server list | grep -qF '| test | SHUTOFF |' && openstack server start test
+  openstack server list | grep -F '| test | ACTIVE |'
+  $CONTROLLER_SSH sudo podman exec -it libvirt_virtqemud virsh list --all | grep 'instance-00000001   running'
+  ```
+  Note that in this guide, the same host acts as a controller, and also as a compute.
diff --git a/docs/openstack/placement_adoption.md b/docs/openstack/placement_adoption.md
@@ -2,8 +2,12 @@
 
 ## Prerequisites
 
-* Previous Adoption steps completed. Notably, the service databases
-  must already be imported into the podified MariaDB.
+* Previous Adoption steps completed. Notably,
+  * the [service databases](mariadb_copy.md)
+    must already be imported into the podified MariaDB.
+  * the [Keystone service](keystone_adoption.md) needs to be imported.
+  * the Memcached operator needs to be deployed (nothing to import for it from
+    the source environment).
 
 ## Variables