Merge pull request #193 from bogdando/mariadb_nova_adoption_checks

Nova: workloads adoption, mariadb pre-/post-checks

docs_dev/assemblies/development_environment.adoc

@@ -7,11 +7,17 @@ Wallaby (or OSP 17.1) OpenStack in Standalone configuration.
 
 == Environment prep
 
+Get dataplane adoption repo:
+[,bash]
+----
+git clone https://github.com/openstack-k8s-operators/data-plane-adoption.git ~/data-plane-adoption
+----
+
 Get install_yamls:
 
 [,bash]
 ----
-git clone https://github.com/openstack-k8s-operators/install_yamls.git
+git clone https://github.com/openstack-k8s-operators/install_yamls.git ~/install_yamls
 ----
 
 Install tools for operator development:
@@ -44,7 +50,7 @@ a RedFish BMC emulator.
 
 [,bash]
 ----
-cd ..  # back to install_yamls
+cd ~/install_yamls
 make nmstate
 make namespace
 cd devsetup  # back to install_yamls/devsetup
@@ -112,7 +118,7 @@ To use Wallaby content instead, run the following:
 
 [,bash]
 ----
-cd install_yamls/devsetup
+cd ~/install_yamls/devsetup
 make standalone
 ----
 
@@ -261,12 +267,21 @@ ping -c 4 $(openstack server show baremetal-test -f json -c addresses | jq -r .a
 
 '''
 
+==== Virtual Machine Steps
+
 [,bash]
 ----
-export OS_CLOUD=standalone
-source ~/install_yamls/devsetup/scripts/edpm-deploy-instance.sh
+cd ~/data-plane-adoption
+bash tests/roles/development_environment/templates/pre_launch.bash
 ----
 
+'''
+
+==== Ceph Storage Steps
+
+Make sure a cinder-volume backend is properly configured, or skip below steps
+to create a test workload without volume attachments.
+
 Confirm the image UUID can be seen in Ceph's images pool.
 
 [,bash]

docs_user/modules/openstack-backend_services_deployment.adoc

@@ -9,8 +9,12 @@ podified OpenStack control plane services.
 
 == Prerequisites
 
-* The cloud which we want to adopt is up and running. It's on
+* The source cloud which we want to adopt is up and running. It's on
 OpenStack Wallaby release.
+* A VM instance named `test` is running on the source cloud and its
+floating IP is set into `FIP` env var. You can use a
+xref:../docs_dev/assemblies/development_environment.adoc#virtual-machine-steps[helper script]
+to create that test VM.
 * The `openstack-operator` is deployed, but `OpenStackControlPlane` is
 *not* deployed.
 +

docs_user/modules/openstack-edpm_adoption.adoc

@@ -15,12 +15,16 @@ ____
 == Variables
 
 Define the shell variables used in the Fast-forward upgrade steps below.
+Set `FIP` to the floating IP address of the `test` VM pre-created earlier on the source cloud.
 Define the map of compute node name, IP pairs.
 The values are just illustrative, use values that are correct for your environment:
 
 [,bash]
 ----
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+
+alias openstack="oc exec -t openstackclient -- openstack"
+FIP=192.168.122.20
 declare -A computes
 export computes=(
   ["standalone.localdomain"]="192.168.122.100"
@@ -428,7 +432,7 @@ ____
 +
 [,bash]
 ----
-  oc exec -it openstack-cell1-galera-0 -- mysql --user=root --password=${PODIFIED_DB_ROOT_PASSWORD} \
+  oc exec -it openstack-galera-0 -c galera -- mysql --user=root --password=${PODIFIED_DB_ROOT_PASSWORD} \
     -e "select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='nova-compute';"
 ----
 +
@@ -535,3 +539,21 @@ The above query should return an empty result as a completion criterion.
   oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations
   oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
 ----
+
+* Verify if Nova services con stop the existing test VM instance:
++
+[,bash]
+----
+${BASH_ALIASES[openstack]} server list | grep -qF '| test | ACTIVE |' && openstack server stop test
+${BASH_ALIASES[openstack]} server list | grep -qF '| test | SHUTOFF |'
+${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test | grep "it is in power state shutdown" || echo PASS
+----
+
+* Verify if Nova services can start the existing test VM instance:
++
+[,bash]
+----
+${BASH_ALIASES[openstack]} server list | grep -qF '| test | SHUTOFF |' && openstack server start test
+${BASH_ALIASES[openstack]} server list | grep -F '| test | ACTIVE |'
+${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test --fit-width -f json | jq -r '.state' | grep running
+----

docs_user/modules/openstack-mariadb_copy.adoc

@@ -17,6 +17,7 @@ ____
  ** The OpenStackControlPlane resource must be already created at this point.
  ** Podified MariaDB and RabbitMQ are running. No other podified
 control plane services are running.
+ ** Required services specific topology xref:openstack-pull_openstack_configuration.adoc#get-services-topology-specific-configuration[configuration collected]
  ** OpenStack services have been xref:stop_openstack_services.adoc[stopped]
  ** There must be network routability between:
   *** The adoption host and the original MariaDB.
@@ -34,49 +35,39 @@ just illustrative, use values that are correct for your environment:
 
 [,bash]
 ----
-MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
-
 PODIFIED_MARIADB_IP=$(oc get svc --selector "mariadb/name=openstack" -ojsonpath='{.items[0].spec.clusterIP}')
 PODIFIED_CELL1_MARIADB_IP=$(oc get svc --selector "mariadb/name=openstack-cell1" -ojsonpath='{.items[0].spec.clusterIP}')
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
 
-# Replace with your environment's MariaDB IP:
-SOURCE_MARIADB_IP=192.168.122.100
-SOURCE_DB_ROOT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
-
 # The CHARACTER_SET and collation should match the source DB
 # if the do not then it will break foreign key relationships
 # for any tables that are created in the future as part of db sync
 CHARACTER_SET=utf8
 COLLATION=utf8_general_ci
+
+MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
+# Replace with your environment's MariaDB IP:
+SOURCE_MARIADB_IP=192.168.122.100
+SOURCE_DB_ROOT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
 ----
 
 == Pre-checks
 
-* Test connection to the original DB (show databases):
+* Get the count of not-OK source databases:
 +
 [,bash]
 ----
-podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
-    mysql -h "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" -e 'SHOW databases;'
-----
-
-* Run mysqlcheck on the original DB to look for things that are not OK:
-+
-[,bash]
-----
-podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
-    mysqlcheck --all-databases -h $SOURCE_MARIADB_IP -u root "-p$SOURCE_DB_ROOT_PASSWORD" | grep -v OK
+test -z "$PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK"  || [ "$PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK" = " " ]
 ----
 
 * Test connection to podified DBs (show databases):
 +
 [,bash]
 ----
 oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
-    mysql -h "$PODIFIED_MARIADB_IP" -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;'
+    mysql -rsh "$PODIFIED_MARIADB_IP" -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;'
 oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
-    mysql -h "$PODIFIED_CELL1_MARIADB_IP" -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;'
+    mysql -rsh "$PODIFIED_CELL1_MARIADB_IP" -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;'
 ----
 
 == Procedure - data copy
@@ -163,29 +154,52 @@ EOF
     oc run ${container_name} --image ${MARIADB_IMAGE} -i --rm --restart=Never -- \
         mysql -h "${db_server}" -uroot "-p${db_password}" "${db_name}" < "${db_file}"
 done
-oc exec -it openstack-galera-0 -- mysql --user=root --password=${db_server_password_map["default"]} -e \
+oc exec -it openstack-galera-0 -c galera -- mysql --user=root --password=${db_server_password_map["default"]} -e \
     "update nova_api.cell_mappings set name='cell1' where name='default';"
-oc exec -it openstack-cell1-galera-0 -- mysql --user=root --password=${db_server_password_map["default"]} -e \
+oc exec -it openstack-cell1-galera-0 -c galera -- mysql --user=root --password=${db_server_password_map["default"]} -e \
     "delete from nova_cell1.services where host not like '%nova-cell1-%' and services.binary != 'nova-compute';"
 ----
 
 == Post-checks
 
+Compare the following outputs with the topology specific configuration
+xref:openstack-pull_openstack_configuration.adoc#get-services-topology-specific-configuration[collected earlier]:
+
 * Check that the databases were imported correctly:
 +
 [,bash]
 ----
-oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
-mysql -h "${PODIFIED_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
-    | grep keystone
+# use 'oc exec' and 'mysql -rs' to maintain formatting
+dbs=$(oc exec openstack-galera-0 -c galera -- mysql -rs -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;')
+echo $dbs | grep -Eq '\bkeystone\b'
+
 # ensure neutron db is renamed from ovs_neutron
-oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
-mysql -h "${PODIFIED_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
-    | grep neutron
+echo $dbs | grep -Eq '\bneutron\b'
+echo $PULL_OPENSTACK_CONFIGURATION_DATABASES | grep -Eq '\bovs_neutron\b'
+
 # ensure nova cell1 db is extracted to a separate db server and renamed from nova to nova_cell1
-oc run mariadb-client --image $MARIADB_IMAGE -i --rm --restart=Never -- \
-mysql -h "${PODIFIED_CELL1_MARIADB_IP}" -uroot "-p${PODIFIED_DB_ROOT_PASSWORD}" -e 'SHOW databases;' \
-    | grep nova_cell1
+c1dbs=$(oc exec openstack-cell1-galera-0 -c galera -- mysql -rs -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" -e 'SHOW databases;')
+echo $c1dbs | grep -Eq '\bnova_cell1\b'
+
+# ensure default cell renamed to cell1, and the cell UUIDs retained intact
+novadb_mapped_cells=$(oc exec openstack-galera-0 -c galera -- mysql -rs -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" \
+  nova_api -e 'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')
+uuidf='\S{8,}-\S{4,}-\S{4,}-\S{4,}-\S{12,}'
+left_behind=$(comm -23 \
+  <(echo $PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS | grep -oE " $uuidf \S+") \
+  <(echo $novadb_mapped_cells | tr -s "| " " " | grep -oE " $uuidf \S+"))
+changed=$(comm -13 \
+  <(echo $PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS | grep -oE " $uuidf \S+") \
+  <(echo $novadb_mapped_cells | tr -s "| " " " | grep -oE " $uuidf \S+"))
+test $(grep -Ec ' \S+$' <<<$left_behind) -eq 1
+default=$(grep -E ' default$' <<<$left_behind)
+test $(grep -Ec ' \S+$' <<<$changed) -eq 1
+grep -qE " $(awk '{print $1}' <<<$default) cell1$" <<<$changed
+
+# ensure the registered Nova compute service name has not changed
+novadb_svc_records=$(oc exec openstack-cell1-galera-0 -c galera -- mysql -rs -uroot "-p$PODIFIED_DB_ROOT_PASSWORD" \
+  nova_cell1 -e "select host from services where services.binary='nova-compute' order by host asc;")
+diff -Z <(echo $novadb_svc_records) <(echo $PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES)
 ----
 
 * During the pre/post checks the pod `mariadb-client` might have returned a pod security warning

docs_user/modules/openstack-nova_adoption.adoc

@@ -17,8 +17,8 @@ must already be imported into the podified MariaDB;
  ** the xref:glance_adoption.adoc[Glance service] needs to be imported;
  ** the xref:ovn_adoption.adoc[OVN DB services] need to be imported;
  ** the xref:neutron_adoption.adoc[Neutron service] needs to be imported;
- ** Required services specific topology link:pull_openstack_configuration.md#get-services-topology-specific-configuration[configuration collected];
- ** OpenStack services have been xref:stop_openstack_services.adoc[stopped]
+ ** Required services specific topology xref:openstack-pull_openstack_configuration.adoc#get-services-topology-specific-configuration[configuration collected];
+ ** OpenStack services have been xref:openstack-stop_openstack_services.adoc[stopped]
 
 == Variables
 
@@ -132,3 +132,26 @@ podified FQDNs and that Nova API responds.
 openstack endpoint list | grep nova
 openstack server list
 ----
+
+Compare the following outputs with the topology specific configuration
+xref:openstack-pull_openstack_configuration.adoc#get-services-topology-specific-configuration[collected earlier]:
+
+* Query the superconductor for cell1 existance and compare it to pre-adoption values:
++
+[,bash]
+----
+echo $PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS
+oc rsh nova-cell0-conductor-0 nova-manage cell_v2 list_cells | grep -F '| cell1 |'
+----
++
+The expected changes to happen:
+
+ ** cell1's `nova` DB and user name become `nova_cell1`.
+ ** Default cell is renamed to `cell1` (in a multi-cell setup, it should become indexed as the last cell instead).
+ ** RabbitMQ transport URL no longer uses `guest`.
+
+____
+*NOTE* At this point, Nova control plane services have yet taken control over
+existing Nova compute workloads. That would become possible to verify only after
+xref:./edpm_adoption.adoc[EDPM adoption] completed.
+____

docs_user/modules/openstack-pull_openstack_configuration.adoc

@@ -90,3 +90,65 @@ Once it's done, you should have into your local path a directory per services su
       ▾ glance/
       ▾ keystone/
 ----
+
+== Get services topology specific configuration
+
+Define the shell variables used in the steps below. The values are
+just illustrative, use values that are correct for your environment:
+
+[,bash]
+----
+MARIADB_IMAGE=quay.io/podified-antelope-centos9/openstack-mariadb:current-podified
+SOURCE_MARIADB_IP=192.168.122.100
+SOURCE_DB_ROOT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' MysqlRootPassword:' | awk -F ': ' '{ print $2; }')
+----
+
+Export shell variables for the following outputs to compare it with post-adoption values later on:
+
+* Test connection to the original DB:
++
+[,bash]
+----
+export PULL_OPENSTACK_CONFIGURATION_DATABASES=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
+    mysql -rsh "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" -e 'SHOW databases;')
+echo "$PULL_OPENSTACK_CONFIGURATION_DATABASES"
+----
++
+Note the `nova`, `nova_api`, `nova_cell0` databases residing in the same DB host.
+
+* Run mysqlcheck on the original DB to look for things that are not OK:
++
+[,bash]
+----
+export PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE \
+    mysqlcheck --all-databases -h $SOURCE_MARIADB_IP -u root "-p$SOURCE_DB_ROOT_PASSWORD" | grep -v OK)
+echo "$PULL_OPENSTACK_CONFIGURATION_MYSQLCHECK_NOK"
+----
+
+* Get Nova cells mappings from database:
++
+[,bash]
+----
+export PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE mysql \
+    -rsh "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" nova_api -e \
+    'select uuid,name,transport_url,database_connection,disabled from cell_mappings;')
+echo "$PULL_OPENSTACK_CONFIGURATION_NOVADB_MAPPED_CELLS"
+----
+
+* Get the host names of the registered Nova compute services:
++
+[,bash]
+----
+export PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES=$(podman run -i --rm --userns=keep-id -u $UID $MARIADB_IMAGE mysql \
+    -rsh "$SOURCE_MARIADB_IP" -uroot "-p$SOURCE_DB_ROOT_PASSWORD" nova_api -e \
+    "select host from nova.services where services.binary='nova-compute';")
+echo "$PULL_OPENSTACK_CONFIGURATION_NOVA_COMPUTE_HOSTNAMES"
+----
+
+* Get the list of mapped Nova cells:
++
+[,bash]
+----
+export PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS=$($CONTROLLER_SSH sudo podman exec -it nova_api nova-manage cell_v2 list_cells)
+echo "$PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS"
+----

docs_user/modules/openstack-stop_openstack_services.adoc

@@ -62,6 +62,10 @@ openstack share list --all-projects -c ID -c Status |grep -E '\| .+ing \|'| grep
 openstack image list -c ID -c Status |grep -E '\| .+ing \|'
 ----
 
+Also collect the xref:openstack-pull_openstack_configuration.adoc#get-services-topology-specific-configuration[topology configuration],
+before stopping services required to gather it live. You will need it to compare it
+with the post-adoption values later on.
+
 == Stopping control plane services
 
 We can stop OpenStack services at any moment, but we may leave things in an

mkdocs.yml

@@ -27,6 +27,7 @@ theme:
 nav:
   - Overview: README.md
   - OpenStack:
+    - contributing/development_environment.md
     - openstack/planning.md
     - openstack/backend_services_deployment.md
     - openstack/pull_openstack_configuration.md

tests/playbooks/test_minimal.yaml

@@ -24,7 +24,9 @@
     ansible.builtin.shell:
       executable: /bin/bash
   roles:
+    - development_environment
     - backend_services
+    - pull_openstack_configuration
     - stop_openstack_services
     - mariadb_copy
     - ovn_adoption

tests/playbooks/test_with_ceph.yaml

@@ -28,7 +28,9 @@
     ansible.builtin.shell:
       executable: /bin/bash
   roles:
+    - development_environment
     - backend_services
+    - pull_openstack_configuration
     - stop_openstack_services
     - mariadb_copy
     - ovn_adoption