Nova adoption ffu (no extra cell) #192

bogdando · 2023-11-06T15:22:26Z

Extracted nova services FFU step being done from #176

Update EDPM adoption docs and tests to execute Nova compute post-FFU.

For that, deploy an additional nova-compute-ffu EDPM service and
patch openstack control plane CR for nova services.
Because of the different lifecycle management tooling used for both
actions, orchestrate FFU w/o a lock-step between nova compute EDPM
and podified control plane services.

Jira: OSPRH-338

Depends-on: https://review.rdoproject.org/r/c/rdo-jobs/+/50917

softwarefactory-project-zuul · 2023-11-06T18:04:20Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/b03024da30604a15b1a498c1491770fc

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 1h 36m 39s

softwarefactory-project-zuul · 2023-11-08T18:40:29Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/473fd3e628504e53ba1dff4e0408cf1f

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 01m 27s

softwarefactory-project-zuul · 2023-11-13T17:35:52Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/375347d7b9ce40e48c615ac4c6d53b02

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 02m 23s

bogdando · 2023-11-14T17:22:44Z

this has been tested on my env, let's fix CI...

softwarefactory-project-zuul · 2023-11-14T19:22:52Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/a6888a92072c4e3595c31b7cb9da9a14

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 21m 12s

softwarefactory-project-zuul · 2023-11-15T14:43:20Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/02bd8c038bb94cefa57fee73bcd432a9

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 03m 56s

tests/roles/dataplane_adoption/tasks/nova_ffu.yaml

docs/openstack/edpm_adoption.md

tests/roles/dataplane_adoption/tasks/main.yaml

softwarefactory-project-zuul · 2023-11-17T20:06:11Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/836fbc4237bd4aadbaa4a202b402eb41

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 33m 34s

gibizer · 2023-11-20T10:55:13Z

The adopted compute did not reported a status as the compute version is still wallaby.

fatal: [localhost]: FAILED! => {"attempts": 20, "changed": true, "cmd": "set -euxo pipefail\n\n\nPODIFIED_DB_ROOT_PASSWORD=\"12345678\"\n\noc rsh mariadb-openstack-cell1 mysql --user=root --password=${PODIFIED_DB_ROOT_PASSWORD}  -e \"select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='nova-compute';\"\n", "delta": "0:00:00.251873", "end": "2023-11-17 14:51:43.574921", "msg": "", "rc": 0, "start": "2023-11-17 14:51:43.323048", "stderr": "+ PODIFIED_DB_ROOT_PASSWORD=12345678\n+ oc rsh mariadb-openstack-cell1 mysql --user=root --password=12345678 -e 'select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='\\''nova-compute'\\'';'", "stderr_lines": ["+ PODIFIED_DB_ROOT_PASSWORD=12345678", "+ oc rsh mariadb-openstack-cell1 mysql --user=root --password=12345678 -e 'select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='\\''nova-compute'\\'';'"], "stdout": "version\n56", "stdout_lines": ["version", "56"]}

https://review.rdoproject.org/zuul/build/df8d241f458747ea897046524cdf807e/log/controller/data-plane-adoption-tests-repo/data-plane-adoption/tests/logs/test_with_ceph_out_2023-11-17T14:15:42EST.log

I don't know where are the nova-compute logs in CI during the adoption testing. :/ So I only have indirect checks

The DataPlaneService/nova looks good to me: https://logserver.rdoproject.org/92/192/67fec82340574efbfb56d234e7bf3a680888939f/github-check/data-plane-adoption-github-rdo-centos-9-extracted-crc/df8d241/controller/ci-framework-data/logs/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-0812f031e363406238d47a3ba3cfb33412e9a2d143d2b4b9365c9796b80bb8aa/namespaces/openstack/crs/openstackdataplaneservices.dataplane.openstack.org/nova-compute-extraconfig.yaml

The execution log of the nova edpm role also look clean: https://logserver.rdoproject.org/92/192/67fec82340574efbfb56d234e7bf3a680888939f/github-check/data-plane-adoption-github-rdo-centos-9-extracted-crc/df8d241/controller/ci-framework-data/logs/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-0812f031e363406238d47a3ba3cfb33412e9a2d143d2b4b9365c9796b80bb8aa/namespaces/openstack/pods/nova.compute.extraconfig-openstack-tgdjr/logs/openstackansibleee.log

Without the compute logs it is hard to tell why the compute does not come up :/

gibizer · 2023-11-20T10:55:23Z

recheck
maybe it was just slow?

fao89 · 2023-11-20T11:00:31Z

@gibizer this is what we have for the logs so far:
https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/data_plane_adoption/collect_logs_crc.yaml#L6

gibizer · 2023-11-20T11:22:17Z

@gibizer this is what we have for the logs so far: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/data_plane_adoption/collect_logs_crc.yaml#L6

In case of an greenfield job ci-framework takes bunch of logs from the computes. It would be good to reuse that logic somehow here in the adoption jobs

softwarefactory-project-zuul · 2023-11-27T19:36:14Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/a67f04155be7487d83ab6966f60165db

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 38m 16s

gibizer · 2023-11-27T19:58:42Z

recheck PS2

gibizer · 2023-11-27T20:03:39Z

recheck

gibizer · 2023-11-27T20:04:50Z

recheck PS2

softwarefactory-project-zuul · 2023-11-27T20:59:05Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/19bff67041a54aec92db93b6edba241f

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 40m 51s

gibizer · 2023-11-27T22:08:20Z

recheck ps3

softwarefactory-project-zuul · 2023-11-27T22:51:30Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/1f6eba7722c64b7d938462bd36b211b9

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 41m 21s

gibizer · 2023-11-28T08:27:28Z

recheck ps5

softwarefactory-project-zuul · 2023-11-28T10:46:38Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/c5afcfcf9d6f4683883dc742d95000c2

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 16m 59s

bogdando · 2023-11-28T12:48:59Z

post-CI run logs extraction looks broken here https://github.com/openstack-k8s-operators/ci-framework/blob/main/roles/artifacts/tasks/edpm.yml#L44 @cjeanner fyi

bogdando · 2023-11-28T14:27:40Z

recheck https://review.rdoproject.org/r/c/rdo-jobs/+/50917

softwarefactory-project-zuul · 2023-11-28T14:27:41Z

This change depends on a change that failed to merge.

Change https://review.rdoproject.org/r/c/rdo-jobs/+/50917 is needed.

bogdando · 2023-11-28T14:28:03Z

recheck

softwarefactory-project-zuul · 2023-11-28T18:52:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/47c5b94f603744e782dc849df3b27702

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 05m 01s

bogdando · 2023-11-29T14:52:56Z

recheck

softwarefactory-project-zuul · 2023-11-29T18:08:22Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/ad32854f5dac47dda1d1c1f197a2485a

❌ data-plane-adoption-github-rdo-centos-9-extracted-crc FAILURE in 2h 26m 58s

bogdando · 2023-11-30T16:18:58Z

Failed due to known issues with isolnet configs (no rabbitmq connectivity) https://logserver.rdoproject.org/92/192/93526b893a7944fe8af2e158446cae8477dac1c6/github-check/data-plane-adoption-github-rdo-centos-9-extracted-crc/db7b697/standalone/containers/nova/nova-compute.log

gibizer · 2023-12-01T13:15:11Z

recheck we saw https://review.rdoproject.org/r/c/rdo-jobs/+/50917 make this PR pass CI.

Update EDPM adoption docs and tests to execute Nova compute post-FFU. For that, deploy an additional nova-compute-ffu EDPM service and patch openstack control plane CR for nova services. Because of the different lifecycle management tooling used for both actions, orchestrate FFU w/o a lock-step between nova compute EDPM and podified control plane services. Signed-off-by: Bohdan Dobrelia <[email protected]>

softwarefactory-project-zuul · 2023-12-01T16:23:25Z

This change depends on a change that failed to merge.

Change https://review.rdoproject.org/r/c/rdo-jobs/+/50917 is needed.

gibizer · 2023-12-01T16:23:46Z

recheck rebased the dependency

marios · 2023-12-04T11:59:04Z

tests/roles/dataplane_adoption/tasks/nova_ffu.yaml

+    {{ mariadb_copy_shell_vars }}
+    oc rsh mariadb-openstack-cell1 mysql --user=root --password=${PODIFIED_DB_ROOT_PASSWORD} \
+      -e "select a.version from nova_cell1.services a join nova_cell1.services b where a.version!=b.version and a.binary='nova-compute';"
+  register: records_check_results


new job is failing here https://logserver.rdoproject.org/53/50653/11/check/data-plane-adoption-OSP-17-to-extracted-crc/1a62ea3/controller/data-plane-adoption-tests-repo/data-plane-adoption/tests/logs/test_with_ceph_out_2023-12-04T05:20:41EST.log

https://logserver.rdoproject.org/53/50653/11/check/data-plane-adoption-OSP-17-to-extracted-crc/1a62ea3/controller/pod/nova-cell1-conductor-0-logs.txt

2023-12-04 10:32:38.433 1 DEBUG oslo_db.sqlalchemy.engines [None req-76342adf-c98e-4a74-91f9-71e7e86e87f2 - - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/engines.py:335�[00m 2023-12-04 10:32:38.434 1 ERROR nova.context [None req-76342adf-c98e-4a74-91f9-71e7e86e87f2 - - - - - -] Error gathering result from cell 292fe7d7-f10c-4546-876d-753875e67b77: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'nova_cell1'@'192.168.122.100' (using password: YES)")

The conductor log shows that the issue is earlier than the compute adoption. Show the new k8s control plane has a wrong / incomplete DB setup as the conductor cannot talk to its DB. Wondering how the db sync on that same DB was run successfully.

The Nova CR status is Ready so there was a succesfully db sync run.

I checked the logs and dumps but I don't see why the cell1 conductor cannot connect to the DB. Unfortunately all the passwords are masked in must gather so I cannot check those. @marios if you have a held node with this issue then I can check the creds there

@gibizer thanks for having a look sorry we were trying to get to a solution and missed your comments. in the end it was a dns issue resolved with https://github.com/openstack-k8s-operators/data-plane-adoption/pull/218/files

green run there if you want to poke at logs https://review.rdoproject.org/zuul/build/87df5976f8814ea9a319eea1caececb2/artifacts

The green result nova-compute logs looks good to me!

bogdando mentioned this pull request Nov 6, 2023

Nova services adoption (no extra cell) #176

Closed

bogdando force-pushed the nova_adoption_ffu branch from d591e29 to 453c6ad Compare November 6, 2023 16:25

bogdando mentioned this pull request Nov 6, 2023

Nova: workloads adoption, mariadb pre-/post-checks #193

Merged

bogdando force-pushed the nova_adoption_ffu branch 2 times, most recently from 1dcded2 to d6baec4 Compare November 8, 2023 16:37

bogdando force-pushed the nova_adoption_ffu branch 5 times, most recently from 3ba8a12 to 3b256aa Compare November 13, 2023 15:31

bogdando force-pushed the nova_adoption_ffu branch 2 times, most recently from 626671d to 01af10f Compare November 14, 2023 17:00

bogdando force-pushed the nova_adoption_ffu branch from 01af10f to eb31157 Compare November 15, 2023 12:37

bogdando requested review from jistr, fao89 and gibizer November 15, 2023 12:38

gibizer reviewed Nov 15, 2023

View reviewed changes

bogdando force-pushed the nova_adoption_ffu branch from eb31157 to 180fb86 Compare November 17, 2023 15:26

fao89 force-pushed the nova_adoption_ffu branch from 180fb86 to 67fec82 Compare November 17, 2023 17:30

bogdando force-pushed the nova_adoption_ffu branch from 8083b4a to 93526b8 Compare November 28, 2023 12:39

This comment was marked as outdated.

Sign in to view

jistr approved these changes Dec 1, 2023

View reviewed changes

bogdando force-pushed the nova_adoption_ffu branch from 93526b8 to 1ff183a Compare December 1, 2023 16:13

gibizer approved these changes Dec 1, 2023

View reviewed changes

cescgina approved these changes Dec 4, 2023

View reviewed changes

jistr merged commit a7dbd5c into openstack-k8s-operators:main Dec 4, 2023
2 checks passed

marios reviewed Dec 4, 2023

View reviewed changes

bogdando deleted the nova_adoption_ffu branch December 4, 2023 13:27

Nova adoption ffu (no extra cell) #192

Nova adoption ffu (no extra cell) #192

Conversation

bogdando commented Nov 6, 2023 • edited by gibizer Loading

softwarefactory-project-zuul bot commented Nov 6, 2023

softwarefactory-project-zuul bot commented Nov 8, 2023

softwarefactory-project-zuul bot commented Nov 13, 2023

bogdando commented Nov 14, 2023

softwarefactory-project-zuul bot commented Nov 14, 2023

softwarefactory-project-zuul bot commented Nov 15, 2023

softwarefactory-project-zuul bot commented Nov 17, 2023

gibizer commented Nov 20, 2023

gibizer commented Nov 20, 2023

fao89 commented Nov 20, 2023

gibizer commented Nov 20, 2023

softwarefactory-project-zuul bot commented Nov 27, 2023

gibizer commented Nov 27, 2023

gibizer commented Nov 27, 2023

gibizer commented Nov 27, 2023

softwarefactory-project-zuul bot commented Nov 27, 2023

gibizer commented Nov 27, 2023

softwarefactory-project-zuul bot commented Nov 27, 2023

gibizer commented Nov 28, 2023

softwarefactory-project-zuul bot commented Nov 28, 2023

bogdando commented Nov 28, 2023

bogdando commented Nov 28, 2023

softwarefactory-project-zuul bot commented Nov 28, 2023

bogdando commented Nov 28, 2023

softwarefactory-project-zuul bot commented Nov 28, 2023

This comment was marked as outdated.

bogdando commented Nov 29, 2023

softwarefactory-project-zuul bot commented Nov 29, 2023

bogdando commented Nov 30, 2023

gibizer commented Dec 1, 2023

softwarefactory-project-zuul bot commented Dec 1, 2023

gibizer commented Dec 1, 2023

marios Dec 4, 2023

Choose a reason for hiding this comment

gibizer Dec 4, 2023

Choose a reason for hiding this comment

gibizer Dec 4, 2023

Choose a reason for hiding this comment

gibizer Dec 4, 2023

Choose a reason for hiding this comment

marios Dec 6, 2023

Choose a reason for hiding this comment

gibizer Dec 6, 2023

Choose a reason for hiding this comment

bogdando commented Nov 6, 2023 •

edited by gibizer

Loading