Nova: Refactor mariadb pre-checks, add post-checks

Get services topology specific configuration in
pull_openstack_configuration. Add missing role for that as well.

Split maridb source vs podified env vars shell headers to use it
in corresponding places.

Update and fix the composition of the services pre-check list to
execute it before stopping services.

Update and fix the composition of the list of the services to be
stopped (cannot pull data from stopped services).

Verify no dataplane disruptions during the FFU/adoption process.

Verify Nova services still control pre-created VM workload after
FFU/adotpion is done.

Signed-off-by: Bohdan Dobrelia <[email protected]>

docs/openstack/edpm_adoption.md

@@ -11,6 +11,9 @@ The values are just illustrative, use values that are correct for your environme
 
 ```bash
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+CONTROLLER_SSH="ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]"
+
+alias openstack="oc exec -t openstackclient -- openstack"
 ```
 
 ## Pre-checks
@@ -453,3 +456,21 @@ later on.
     oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
     ```
 
+* Verify no Nova compute dataplane disruptions during the adoption/upgrade process:
+
+  ```bash
+  $CONTROLLER_SSH sudo podman exec -it libvirt_virtqemud virsh list --all | grep 'instance-00000001   running'
+  ```
+
+* Verify if Nova services control the existing VM instance:
+
+  ```bash
+  openstack server list | grep -qF '| test | ACTIVE |' && openstack server stop test
+  openstack server list | grep -qF '| test | SHUTOFF |'
+  $CONTROLLER_SSH sudo podman exec -it libvirt_virtqemud virsh list --all | grep 'instance-00000001   shut off'
+
+  openstack server list | grep -qF '| test | SHUTOFF |' && openstack server start test
+  openstack server list | grep -F '| test | ACTIVE |'
+  $CONTROLLER_SSH sudo podman exec -it libvirt_virtqemud virsh list --all | grep 'instance-00000001   running'
+  ```
+  Note that in this guide, the same host acts as a controller, and also as a compute.

docs/openstack/mariadb_copy.md

@@ -18,6 +18,8 @@ cluster.
   * Podified MariaDB and RabbitMQ are running. No other podified
     control plane services are running.
 
+  * Required services specific topology [configuration collected](pull_openstack_configuration.md#get-services-topology-specific-configuration)
+
   * OpenStack services have been [stopped](stop_openstack_services.md)
 
   * There must be network routability between:
@@ -40,38 +42,28 @@ Define the shell variables used in the steps below. The values are
 just illustrative, use values that are correct for your environment:
 
 ```bash
-MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
-
 PODIFIED_MARIADB_IP=$(oc get svc --selector "cr=mariadb-openstack" -ojsonpath='{.items[0].spec.clusterIP}')
 PODIFIED_CELL1_MARIADB_IP=$(oc get svc --selector "cr=mariadb-openstack-cell1" -ojsonpath='{.items[0].spec.clusterIP}')
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
 
-# Replace with your environment's MariaDB IP:
-SOURCE_MARIADB_IP=192.168.122.100
-SOURCE_DB_ROOT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
-
 # The CHARACTER_SET and collation should match the source DB
 # if the do not then it will break foreign key relationships
 # for any tables that are created in the future as part of db sync
 CHARACTER_SET=utf8
 COLLATION=utf8_general_ci
 
+MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
+# Replace with your environment's MariaDB IP:
+SOURCE_MARIADB_IP=192.168.122.100
+SOURCE_DB_ROOT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
 ```
 
 ## Pre-checks
 
-* Test connection to the original DB (show databases):
+* Get the count of not-OK source databases:
 
   ```bash
-  podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
-      mysql -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" -e 'SHOW databases;'
-  ```
-
-* Run mysqlcheck on the original DB to look for things that are not OK:
-
-  ```bash
-  podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
-      mysqlcheck --all-databases -h $SOURCE_MARIADB_IP -u root "-p$SOURCE_DB_ROOT_PASSWORD" | grep -v OK
+  test -z "$PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK"
   ```
 
 * Test connection to podified DBs (show databases):
@@ -169,20 +161,52 @@ COLLATION=utf8_general_ci
 
 ## Post-checks
 
+Compare the following outputs with the topology specific configuration
+[collected earlier](pull_openstack_configuration.md#get-services-topology-specific-configuration):
+
 * Check that the databases were imported correctly:
 
   ```bash
   oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
   mysql -h "${PODIFIED_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
-      | grep keystone
+    | grep -q keystone
+
   # ensure neutron db is renamed from ovs_neutron
   oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
   mysql -h "${PODIFIED_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
-      | grep neutron
+    | grep -q neutron
+  echo $PULL_OPENSTACK_CONFIGURATION_DATABASES | grep ' ovs_neutron '
+
   # ensure nova cell1 db is extracted to a separate db server and renamed from nova to nova_cell1
   oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
   mysql -h "${PODIFIED_CELL1_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
-      | grep nova_cell1
+    | grep -q nova_cell1
+
+  # ensure default cell renamed to cell1, and the cell UUIDs retained intact
+  NOVADB_MAPPED_CELLS=$(oc rsh mariadb-openstack mysql -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" \
+    nova_api -e 'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')
+  uuidf='\S{8,}-\S{4,}-\S{4,}-\S{4,}-\S{12,}'
+  left_behind=$(comm -23 \
+    <(echo $PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS | grep -oE " $uuidf \S+") \
+    <(echo $NOVADB_MAPPED_CELLS | tr -s "| " " " | grep -oE " $uuidf \S+"))
+  changed=$(comm -13 \
+    <(echo $PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS | grep -oE " $uuidf \S+") \
+    <(echo $NOVADB_MAPPED_CELLS | tr -s "| " " " | grep -oE " $uuidf \S+"))
+  test $(grep -Ec ' \S+$' <<<$left_behind) -eq 1
+  default=$(grep -E ' default$' <<<$left_behind)
+  test $(grep -Ec ' \S+$' <<<$changed) -eq 1
+  grep -qE " $(awk '{print $1}' <<<$default) cell1$" <<<$changed
+
+  # ensure the registered Nova compute service name has not changed
+  oc rsh mariadb-openstack-cell1 mysql -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" nova_cell1 \
+    -e "select host from services where services.binary='nova-compute';" \
+    | grep -qF '| standalone.localdomain |'
+  echo $PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES | grep -qF ' standalone.localdomain'
+
+  # ensure the VM instance cell ID has not changed
+  oc rsh mariadb-openstack mysql -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" nova_api -e \
+    "select cell_id from instance_mappings;" | grep -qF -m1 ' 2 |'
+  echo $PULL_OPENSTACK_CONFIGURATION_NOVADB_INSTANCES_CELL_IDS | grep -q -m1 ' 2$'
   ```
 
 * During the pre/post checks the pod `mariadb-client` might have returned a pod security warning

docs/openstack/nova_adoption.md

@@ -141,3 +141,21 @@ alias openstack="oc exec -t openstackclient -- openstack"
   openstack endpoint list | grep nova
   openstack server list
   ```
+
+Compare the following outputs with the topology specific configuration
+[collected earlier](pull_openstack_configuration.md#get-services-topology-specific-configuration):
+
+* Query the superconductor for cell1 existance:
+
+  ```bash
+  oc rsh nova-cell0-conductor-0 nova-manage cell_v2 list_cells | grep -F '| cell1 |'
+  ```
+
+  The expected changes to happen:
+  * cell1's `nova` DB and user name become `nova_cell1`.
+  * Default cell is renamed to `cell1` (in a multi-cell setup, it should become indexed as the last cell instead).
+  * RabbitMQ transport URL no longer uses `guest`.
+
+> **NOTE** At this point, Nova control plane services have yet taken control over
+> existing Nova compute workloads. That would become possible to verify only after
+> [EDPM adoption](./edpm_adoption.md) completed.

docs/openstack/pull_openstack_configuration.md

@@ -8,7 +8,7 @@ missed or misconfigured.
 
 Make sure you have pull the os-diff repository and configure according to your
 environment:
-[Configure os-diff](planning.md#Configuration tooling)
+[Configure os-diff](planning.md#configuration-tooling)
 
 ## Pull configuration from a TripleO deployment
 
@@ -45,3 +45,61 @@ Once the ansible playbook has been run, you should have into your local director
     ▾ collect_tripleo_configs/
       ▾ glance/
 ```
+
+## Get services topology specific configuration
+
+Define the shell variables used in the steps below. The values are
+just illustrative, use values that are correct for your environment:
+
+```bash
+MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
+SOURCE_MARIADB_IP=192.168.122.100
+SOURCE_DB_ROOT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
+```
+
+Note the following outputs to compare it with post-adoption values later on:
+
+* Test connection to the original DB:
+
+  ```bash
+  PULL_OPENSTACK_CONFIGURATION_DATABASES=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
+      mysql -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" -e 'SHOW databases;')
+  ```
+  Note the `nova`, `nova_api`, `nova_cell0` databases residing in the same DB host.
+
+* Run mysqlcheck on the original DB to look for things that are not OK:
+
+  ```bash
+  PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
+      mysqlcheck --all-databases -h $SOURCE_MARIADB_IP -u root "-p$SOURCE_DB_ROOT_PASSWORD" | grep -v OK)
+  ```
+
+* Get Nova cells mappings from database:
+
+  ```bash
+  PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE mysql \
+      -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" nova_api -e \
+      'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')
+  ```
+
+* Get Nova instances cell_ids from database:
+
+  ```bash
+  PULL_OPENSTACK_CONFIGURATION_NOVADB_INSTANCES_CELL_IDS=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE mysql \
+      -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" nova_api -e \
+      "select cell_id from nova_api.instance_mappings;")
+  ```
+
+* Get the host names of the registered Nova compute services:
+
+  ```bash
+  PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE mysql \
+      -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" nova_api -e \
+      "select host from nova.services where services.binary='nova-compute';")
+  ```
+
+* Get the list of mapped Nova cells:
+
+  ```bash
+  PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS=$($CONTROLLER_SSH sudo podman exec -it nova_api nova-manage cell_v2 list_cells)
+  ```

docs/openstack/stop_openstack_services.md

@@ -58,6 +58,9 @@ openstack volume backup list --all-projects -c ID -c Status |grep -E '\| .+ing \
 openstack image list -c ID -c Status |grep -E '\| .+ing \|'
 ```
 
+Also collect the [topology configuration](pull_openstack_configuration.md#get-services-topology-specific-configuration),
+before stopping services required to gather it live. You will need it to compare it
+with the post-adoption values later on.
 
 ## Stopping control plane services
 

tests/roles/dataplane_adoption/tasks/nova_ffu.yaml

@@ -1,14 +1,13 @@
-- name: set podified MariaDB copy shell vars
-  no_log: "{{ use_no_log }}"
-  ansible.builtin.set_fact:
-    mariadb_copy_shell_vars: |
-      PODIFIED_DB_ROOT_PASSWORD="{{ podified_db_root_password }}"
+- name: get dst database service environment variables
+  ansible.builtin.include_role:
+    name: mariadb_copy
+    tasks_from: env_vars_dst.yaml
 
 - name: wait for cell1 Nova compute EDPM services version updated
   ansible.builtin.shell: |
     {{ shell_header }}
     {{ oc_header }}
-    {{ mariadb_copy_shell_vars }}
+    {{ mariadb_copy_shell_vars_dst }}
     oc rsh mariadb-openstack-cell1 mysql --user=root --password=${PODIFIED_DB_ROOT_PASSWORD} \
       -e "select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='nova-compute';"
   register: records_check_results
@@ -120,3 +119,7 @@
   retries: 10
   delay: 6
 
+- name: Adopted Nova FFU post-checks
+  ansible.builtin.include_role:
+    name: nova_adoption
+    tasks_from: verify.yaml

tests/roles/mariadb_copy/tasks/env_vars_dst.yaml

@@ -0,0 +1,27 @@
+- name: get podified MariaDB service cluster IP
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc get svc --selector "cr=mariadb-openstack" -ojsonpath='{.items[0].spec.clusterIP}'
+  register: podified_mariadb_ip_result
+
+- name: get podified cell1 MariaDB IP
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc get svc --selector "cr=mariadb-openstack-cell1" -ojsonpath='{.items[0].spec.clusterIP}'
+  register: podified_cell1_mariadb_ip_result
+
+- name: set MariaDB copy shell vars
+  no_log: "{{ use_no_log }}"
+  ansible.builtin.set_fact:
+    mariadb_copy_shell_vars_dst: |
+      PODIFIED_MARIADB_IP={{ podified_mariadb_ip_result.stdout }}
+      PODIFIED_CELL1_MARIADB_IP={{ podified_cell1_mariadb_ip_result.stdout }}
+      PODIFIED_DB_ROOT_PASSWORD="{{ podified_db_root_password }}"
+
+      # The CHARACTER_SET and collation should match the source DB
+      # if the do not then it will break foreign key relationships
+      # for any tables that are created in the future as part of db sync
+      CHARACTER_SET=utf8
+      COLLATION=utf8_general_ci

tests/roles/mariadb_copy/tasks/env_vars_src.yaml

@@ -0,0 +1,8 @@
+- name: set src MariaDB copy shell vars
+  no_log: "{{ use_no_log }}"
+  ansible.builtin.set_fact:
+    mariadb_copy_shell_vars_src: |
+      MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
+      # TODO: remove the default(external_...) when CI is transitioned to use 'source_...'
+      SOURCE_MARIADB_IP={{ source_mariadb_ip|default(external_mariadb_ip) }}
+      SOURCE_DB_ROOT_PASSWORD="{{ source_db_root_password|default(external_db_root_password) }}"

tests/roles/mariadb_copy/tasks/main.yaml

@@ -1,38 +1,19 @@
-- name: get podified MariaDB service cluster IP
-  ansible.builtin.shell: |
-    {{ shell_header }}
-    {{ oc_header }}
-    oc get svc --selector "cr=mariadb-openstack" -ojsonpath='{.items[0].spec.clusterIP}'
-  register: podified_mariadb_ip_result
+- name: get the source database service environment variables
+  ansible.builtin.include_tasks:
+    file: env_vars_src.yaml
 
-- name: get podified cell1 MariaDB IP
-  ansible.builtin.shell: |
-    {{ shell_header }}
-    {{ oc_header }}
-    oc get svc --selector "cr=mariadb-openstack-cell1" -ojsonpath='{.items[0].spec.clusterIP}'
-  register: podified_cell1_mariadb_ip_result
+- name: get the destination database service environment variables
+  ansible.builtin.include_tasks:
+    file: env_vars_dst.yaml
 
-- name: set MariaDB copy shell vars
+- name: Get the count of not-OK source databases
   no_log: "{{ use_no_log }}"
-  ansible.builtin.set_fact:
-    mariadb_copy_shell_vars: |
-      MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
-
-      PODIFIED_MARIADB_IP={{ podified_mariadb_ip_result.stdout }}
-      PODIFIED_CELL1_MARIADB_IP={{ podified_cell1_mariadb_ip_result.stdout }}
-      PODIFIED_DB_ROOT_PASSWORD="{{ podified_db_root_password }}"
-
-      # TODO: remove the default(external_...) when CI is transitioned to use 'source_...'
-      SOURCE_MARIADB_IP={{ source_mariadb_ip | default(external_mariadb_ip) }}
-      SOURCE_DB_ROOT_PASSWORD="{{ source_db_root_password | default(external_db_root_password) }}"
-
-      # The CHARACTER_SET and collation should match the source DB
-      # if the do not then it will break foreign key relationships
-      # for any tables that are created in the future as part of db sync
-      CHARACTER_SET=utf8
-      COLLATION=utf8_general_ci
+  environment:
+    PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK: "{{ pull_openstack_configuration_mysqlcheck_nok }}"
+  ansible.builtin.shell:
+    cmd: test -z "$PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK"
 
-- name: MariaDB copy pre checks
+- name: test connection to podified DBs (show databases)
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell:
     cmd: "{{ lookup('ansible.builtin.template', 'pre_checks.bash') }}"
@@ -49,5 +30,10 @@
 
 - name: MariaDB post-checks
   no_log: "{{ use_no_log }}"
+  environment:
+    PULL_OPENSTACK_CONFIGURATION_DATABASES: "{{ pull_openstack_configuration_databases }}"
+    PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS: "{{ pull_openstack_configuration_novadb_mapped_cells }}"
+    PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES: "{{ pull_openstack_configuration_nova_compute_hostnames }}"
+    PULL_OPENSTACK_CONFIGURATION_NOVADB_INSTANCES_CELL_IDS: "{{ pull_openstack_configuration_novadb_instances_cell_ids }}"
   ansible.builtin.shell:
     cmd: "{{ lookup('ansible.builtin.template', 'post_checks.bash') }}"

tests/roles/mariadb_copy/templates/dump_dbs.bash

@@ -1,7 +1,7 @@
 #!/bin/bash
 {{ shell_header }}
 {{ oc_header }}
-{{ mariadb_copy_shell_vars }}
+{{ mariadb_copy_shell_vars_src }}
 
 mkdir -p {{ mariadb_copy_tmp_dir }}
 cd {{ mariadb_copy_tmp_dir }}